The Beginnings of Audience Interaction

Below is an excerpt from the online portfolio of former Carnegie Mellon CS undergraduate Daniel Maynes-Aminzade. His work pioneered some of the technologies and techniques used in Audience Interaction-based projects in the ETC.

The original porfolio can be found at http://www.monzy.org/audience

The Cinematrix Interactive Entertainment System

At SIGGRAPH in 1991, Loren and Rachel Carpenter unveiled an interactive entertainment system that allowed members of a large audience to control an onscreen game using red and green reflective paddles. In the spirit of this approach, I developed a variety of techniques that enabled members of an audience to participate, either cooperatively or competitively, in shared entertainment experiences. These techniques allow audiences with hundreds of people to control onscreen activity by (1) leaning left and right in their seats, (2) batting a beach ball while its shadow is used as a pointing device, and (3) pointing laser pointers at the screen. All of these techniques can be implemented with inexpensive, off the shelf hardware.

Audience movement tracking allows audience members to control an onscreen game without the use of physical props. We position a camera at the front of the auditorium and point it at the audience. We then encourage the members of the audience to move in certain ways, and analyze the streaming video to control an onscreen game. For example, audience members can lean left or right in their chairs to steer a race car or move a paddle in the video game Pong. Here are videos of a large audience playing Pong (7 MB) and Pole Position (11 MB), a classic car racing game.

Another technique uses pixel differencing to determine the amount of motion in the crowd.  In this "chase" game, the left side of the audience controls the coyote and the right side controls the roadrunner.  The more that a side of the audience moves, the faster their character runs.  The small image inserts on either side of the screen show the two sides of the audience, and the bar graphs above them show the level of audience activity on each side.

The common crowd activity of batting a beach ball before a concert provided the inspiration for another interaction technique. We project a game on the front screen of a movie theater, and as the audience bats a beach ball into the air, the ball casts a shadow on the screen. We point a camera at the screen and use computer vision techniques to track the ball’s shadow. This tracking allows the audience to play interactive games using the shadow of the ball as a cursor.  Here is a video of the show (10 MB).

Missile Command using beach ball shadow tracking. The beach ball (top left) casts a shadow (bottom right) which acts as a cursor.

It is not uncommon for members of a large movie theater audience to shine laser pointers at the movie screen before the film begins. By pointing a camera at the screen and tracking the dots, we can create compelling interactive entertainment experiences. The red dots can be attached to a collaborative paint program or a game that encourages maze navigation or rapid flocking of laser points to particular locations.  At right is an example of a game in which the audience is encouraged to uncover a series of hidden images using their laser pointers. This video (13 MB) shows the game in action.

Whack-A-Hamster is another game that uses laser pointers as input devices.  The audience directs their laser points at the hamsters to "whack" them.  Here is a sample video (18 MB).

I have also used laser tracking to carry out live audience polls and trivia.  This is a video showing a poll (9MB), and this video (10 MB) shows a trivia question.  The audience members use their lasers to point at  choices, and the bar graphs continually update to show the audience's preferences until the time limit is reached.

Shadow tracking works well with many different objects.  The game pictured at left allows the audience to raise and lower a surface by "holding it up" with shadows.  When this game is shown in a large auditorium, the audience members in the front row are given pool noodles (long, cylindrical Styrofoam rods) that they hold in front of the screen to create shadows.  The top of each shadow is actually the control point on a spline, and the audience attempts to modify the surface so that the ball bounces into the target.  This video (11 MB) shows my former research advisor giving it a try.

Once the shadow tracking was in place, it was easy to replicate this popular game, shown at last year's Siggraph.  The participants can hold up the falling cascade of letters using their shadows.  Here is a sample video (23 MB).