Detection and Projection
Recognizing gestures is just part of the software's job. It also has to interface with applications so that the gestures you make translate into meaningful actions on the screen. With some applications, this is pretty straightforward. Flipping through a photo album may only rely on a few gestures to navigate pictures and zoom in or out of views. Each of those gestures may be fairly simple.
But other programs might require a greater variety of complex gestures. Let's say you've just come home with the newest version of "Extreme Table Tennis Pro Elite" and you're ready to test your skills against the toughest computer opponents to ever pick up a paddle. You pop your game into a console system that has a 3-D gesture component and pick up a real paddle of your own. What happens next?
The system analyzes the scene in front of it. It detects the presence of the paddle in your hand. As the game begins, you watch the screen and wait for your opponent to volley for serve. As the digital ball screams toward you, the 3-D gesture system determines where the ball would really go within the context of your physical space if it were an actual solid object.
You make your move, preparing a wicked return with crazy backspin. Now the 3-D system has to analyze your reaction, plot it against the flight path of the ball and determine if you made contact or if you completely whiffed it. Assuming your amazing table tennis skills haven't failed you, you hit the ball successfully. Now the system has to determine where the digital ball would go based upon your real physical movements. The software projects a flight path and the ball follows it.
Some games may not involve a physical prop. Your progress through the game will depend entirely upon the movements you make with your body. The system's job is to make sure the actions you take impact the progression of the game appropriately. And all of these actions have to be accounted for within the game itself. It's a big job! That's why some applications require you to move in a specific way to calibrate the system before you get started.