Initialization Options

Color data is available in the following two formats:

• RGB color provides 32-bit, linear X8R8G8B8-formatted color bitmaps, in sRGB color space. To work with RGB data, an application should specify a color or color_YUV image type when it opens the stream.
• YUV color provides 16-bit, gamma-corrected linear UYVY-formatted color bitmaps, where the gamma correction in YUV space is equivalent to sRGB gamma in RGB space. Because the YUV stream uses 16 bits per pixel, this format uses less memory to hold bitmap data and allocates less buffer memory when you open the stream. To work with YUV data, your application should specify the raw YUV image type when it opens the stream. YUV data is available only at the 640x480 resolution and only at 15 FPS. 

The sensor array uses a USB connection to pass data to the PC, and that connection provides a given amount of bandwidth. The Bayer color image data that the sensor returns at 1280x1024 is compressed and converted to RGB before transmission to the runtime. The runtime then decompresses the data before it passes the data to your application. The use of compression makes it possible to return color data at frame rates as high as 30 FPS, but the algorithm that is used leads to some loss of image fidelity.

The sensor array uses a USB connection to pass data to the PC, and that connection provides a given amount of bandwidth. The Bayer color image data that the sensor returns at 1280x1024 is compressed and converted to RGB before transmission to the runtime. The runtime then decompresses the data before it passes the data to your application. The use of compression makes it possible to return color data at frame rates as high as 30 FPS, but the algorithm that is used leads to some loss of image fidelity.

The depth data stream provides frames in which each pixel represents the Cartesian distance, in millimeters, from the camera plane to the nearest object at that particular x and y coordinate in the depth sensor's field of view. The following depth data streams are available:

• Frame size of 640×480 pixels
• Frame size of 320×240 pixels
• Frame size of 80×60 pixels 

The format of the depth data depends on whether the application specifies depth only or depth and player index at NUI initialization, as follows:

• For depth only, the low-order 12 bits (bits 0?11) of each pixel contain depth data and the remaining 4 bits are unused.
• For depth and player index, the low-order 3 bits (bits 0?2) of each pixel contain the player index and the remaining bits contain depth data. 

In the SDK , the Kinect system processes sensor data to identify two human figures in front of the sensor array and then creates the Player Segmentation map. This map is a bitmap in which the pixel values correspond to the player index of the person in the field of view who is closest to the camera, at that pixel position.

Although the player segmentation data is a separate logical stream, in practice the depth data and player segmentation data are merged into a single frame:

• The 13 high-order bits of each pixel represent the distance from the depth sensor to the closest object, in millimeters.
• The 3 low-order bits of each pixel represent the player index of the tracked player who is visible at the pixel's x and y coordinates. These bits are treated as an integer value and are not used as flags in a bit field. 

Application code gets the latest frame of image data by calling a frame retrieval method and passing a buffer. If the latest frame of data is ready, it is copied into the buffer. If your code requests frames of data faster than new frames are available, you can choose whether to wait for the next frame or to return immediately and try again later. The NUI Image Camera API never provides the same frame of data more than once.

Applications can use either of the following two usage models:

• Polling Model The polling model is the simplest option for reading data frames. First, the application code opens the image stream. It then requests a frame and specifies how long to wait for the next frame of data (between 0 and an infinite number of milliseconds). The request method returns when a new frame of data is ready or when the wait time expires, whichever comes first. Specifying an infinite wait causes the call for frame data to block and to wait as long as necessary for the next frame.
  
  When the request returns successfully, the new frame is ready for processing. If the time-out value is set to zero, the application code can poll for completion of a new frame while it performs other work on the same thread. A native C++ application calls NuiImageStreamOpen to open a color or depth stream and omits the optional event. Managed code calls ImageStream.Open. To poll for color and depth frames, native C++applications call NuiImageStreamGetNextFrame and managed code calls ImageStream.GetNextFrame.
  
  
• Event Model The event model supports the ability to integrate retrieval of a skeleton frame into an application engine with more flexibility and more accuracy.
  
  In this model, C++ application code passes an event handle to NuiImageStreamOpen. When a new frame of image data is ready, the event is signaled. Any waiting thread wakes and gets the frame of skeleton data by calling NuiImageGetNextFrame. During this time, the event is reset by the NUI Image Camera API.
  
  Managed code uses the event model by hooking a Runtime.DepthFrameReady or Runtime.ImageFrameReady event to an appropriate event handler. When a new frame of data is ready, the event is signaled and the handler runs and calls ImageStream.GetNextFrame to get the frame.

Application code gets the latest frame of skeleton data in the same way that it gets a frame of image data: by calling a frame retrieval method and passing a buffer. Applications can use either a polling model or an event model, in the same way as for image frames. An application must choose one model or the other; it cannot use both models simultaneously.

To use the polling model:

• A native C++ application calls NuiSkeletonGetNextFrame to retrieve a skeleton frame.
• Managed code calls SkeletonEngine.GetNextFrame.
  

• C++ application code passes an event handle to NuiSkeletonTrackingEnable.
  
  When a new frame of skeleton data is ready, the event is signaled. Any waiting thread wakes and gets the frame of skeleton data by calling NuiSkeletonGetNextFrame. During this time, the event is reset by the NUI Skeleton API.
  
  
• Managed code uses the event model by hooking a Runtime.SkeletonFrameReady event to an appropriate event handler. 

The skeletal tracking engine processes depth frame data to calculate the floor clipping plane, which is described in “Floor Determination,” later in this document. If the application indicates at initialization that it uses skeleton tracking, the skeletal tracking engine signals a skeleton frame each time it processes the depth data, whether or not a skeleton currently appears in the frame. Applications that use the floor clipping plane values can retrieve the skeleton frame. The returned skeleton frame includes the timestamp of the corresponding depth image so that applications can match skeleton data with depth image data.

The skeletal tracking engine provides full skeletal tracking for one or two players in the sensor's field of view. When a player is actively tracked, calls to get the next skeleton frame return complete skeleton data for the player. Passive tracking is provided automatically for up to four additional players in the sensor's field of view. When a player is being tracked passively, the skeleton frame contains only limited information about that player's position. By default, the first two skeletons that the skeletal tracking system finds are actively tracked, as shown in Figure 4.

Figure 4.  Figure 4. Active tracking for two players

For all returned skeletons, the following data is provided:

• The current tracking state of the associated skeleton:
• For skeletons that are passively tracked, this value indicates position-only tracking.
• For an actively tracked skeleton, the value indicates skeleton-tracked.
  
• A unique tracking ID that remains assigned to a single player as that player moves around the screen.

The tracking ID is guaranteed to remain consistently applied to the same player for as long as he or she remains in the field of view. A given tracking ID is guaranteed to remain at the same index in the skeleton data array for as long as the tracking ID is in use. If the tracking ID of the skeleton at a particular index in the array changes, one of two things has happened: Either the tracked player left the field of view and tracking started on another player in the field of view, or the tracked player left the field of view, then returned, and is now being tracked again.

• A position (of type Vector4) that indicates the center of mass for that player. This value is the only available positional value for passive players.
• For the actively tracked players, returned data also includes the current full skeletal data.
• For the passively tracked players, returned data includes only basic positional and identification data, and no skeletal data. 

Image frames of the depth map are 640x480, 320×240, or 80x60 pixels in size. Each pixel represents the Cartesian distance, in millimeters, from the camera plane to the nearest object at that particular x and y coordinate, as shown in Figure 5. A pixel value of 0 indicates that the sensor did not find any objects within its range at that location.

Figure 5.  Figure 5. Depth stream values

The x and y coordinates of the image frame do not represent physical units in the room, but rather pixels on the depth imaging sensor. The interpretation of the x and y coordinates depends on specifics of the optics and imaging sensor. For discussion purposes, this projected space is referred to as the depth image space.

Player skeleton positions are expressed in x, y, and z coordinates. Unlike the coordinate of depth image space, these three coordinates are expressed in meters. The x, y, and z axes are the body axes of the depth sensor. This is a right-handed coordinate system that places the sensor array at the origin point with the positive z axis extending in the direction in which the sensor array points. The positive y axis extends upward, and the positive x axis extends to the left (with respect to the sensor array), as shown in Figure 6. For discussion purposes, this expression of coordinates is referred to as the skeleton space.

Figure 6.  Figure 6. Skeleton-space coordinate system for the sensor array

Each skeleton frame also contains a floor clip plane vector, which contains the coefficients of an estimated floor plane equation. The skeleton tracking system updates this estimate for each frame and uses it as a clipping plane for removing the background and segmentation of players. The general plane equation is:

  
Ax + By + Cz + D = 0

where:

  A = vFloorClipPlane.x 
  B = vFloorClipPlane.y 
  C = vFloorClipPlane.z 
  D = vFloorClipPlane.w

The equation is normalized so that the physical interpretation of D is the height of the camera from the floor, in meters. It is worth noting that the floor might not always be visible. In this case, the floor clip plane is a zero vector. The floor clip plane can be found in the vFloorClipPlane member of the NUI_SKELETON_FRAME structure in the native interface and in the SkeletonFrame.FloorClipPlane field in the managed interface.

By default, the skeleton system always mirrors the user who is being tracked. For applications that use an avatar to represent the user, such mirroring could be desirable if the avatar is shown facing into the screen. However, if the avatar faces the user, mirroring would present the avatar as backwards. Depending on its requirements, an application can create a transformation matrix to mirror the skeleton and then apply the matrix to the points in the array that contains the skeleton positions for that skeleton. The application is responsible for choosing the proper plane for reflection.