by Montgomery Martin
Introduction to Motion Tracking using the Xbox 360 Kinect Camera
Over this three part tutorial, we will learn how to use the open-source sketchbook application Processing to act as a communication bridge between the Kinect hardware and Isadora. Part 1 serves as an introduction to the Kinect and what it can offer, along with some important points about particular versions of the camera
The Xbox 360 Kinect is a unique combination of camera, infrared motion sensor, and video game controller. Beyond video games, the device has been embraced by computer programmers, the maker movement, theatre, dance, and performance artists alike for its inexpensive plug-and-play “hackability”. Since its introduction, the Kinect has been deployed to generate audiovisual and lighting effects that move and shift with performer’s bodies, control interactive games or performance installations, and manipulate virtual avatars and puppets: all in real-time in response to human movement.
An Xbox 360 Kinect Camera. Note the extra cable with USB connector and power adapter.
The Kinect sensor can be made to interface with Isadora for a wide range of interactive functions and video effects. To do so requires using a third party application which can control and receive data from the Kinect camera for transmission into Isadora. During these tutorials, we will learn how to interface between the Kinect and Isadora using Processing, a free, open-source programming language and software sketchbook application created to teach artists, students, designers, and hobbyists how to code within the context of the visual arts. A popular paid alternative to using Processing is Delicode NI Mate. In a future tutorial we will cover some of the additional features and benefits to working with NI Mate.
Choosing the Right Hardware
To get started with Kinect tracking, You need a stand-alone Kinect camera and a special power adapter that includes the USB connector needed to connect the camera to your computer. You do not need an entire Xbox 360 console!
There are several versions of the Kinect camera available. Most notably, Microsoft recently released the Kinect 2 sensor for the Xbox One. However, the software tools for working with the Kinect 2 are not yet widely available online for free. The Kinect2 works on a different platform that is not compatible with many tools made for the original Kinect. However, there are a wide range of software solutions for working with the original Xbox 360 Kinect.
While the original Kinect camera is no longer in production, a scavenger hunt through the bargain bins in many video game shops in your area will usually turn up one or two units. Often such units are available for around $20 USD.
On the base of the Kinect sensor, you can identify the model number of the unit you are holding. Always verify the model number before making a purchase:
Kinect 1414— This is the original Kinect sensor, and has been verified and tested to work with these tools under Processing 2.2.1.
Kinect 1473 — This looks identical to the 1414, but it is an updated model. Several software programs do not work with this model of Kinect. But the software used in these tutorials do work with the model 1473.
Kinect for Windows— While this model looks identical to the Xbox 360 Kinect, many report that this model does not work with Mac OS X software tools.
Kinect 2 — The Kinect 2 does not work with the software outlined in this tutorial.
IMPORTANT! In addition to the Kinect sensor itself, you also need a power adapter. This power adapter is special because it not only provides power, but gives you the USB connector needed to attach the Kinect to your computer. In a search of video game shops in our city, not every Kinect had this crucial but separate piece, so double check before you buy one. A Google search for kinect 360 power adapter will lead you to many third party options.
Understanding the Kinect Sensor
The Xbox Kinect is a powerful tool, however the device makes some important assumptions.
Always bear in mind that the Xbox 360 Kinect is fundamentally a video game controller designed for living room use. The device’s most popular and successful applications in the video game world have primarily been dance games, or games in which players must make large, clear, expressive, violent gestures. While the Kinect does pick up some subtle movements such as breathing, these are exceptionally difficult to track effectively.
Because it’s intended to be used in a living room, the Kinect is designed around the assumption that it will be placed at the base of a television set: approximately 80 to 100 cm (32″ to 40″ inches) above the floor, placed face-to-face with the body it is tracking. Placing the Kinect on the floor may result in unreliable full-body motion tracking.
The Kinect’s field of view is approximately 43° vertically and 56° horizontally. This means that for a performer between 170 to 180 cm in height (or 5’6” to 6’) the optimal distance from the sensor for full body movement tracking is between 2.5 to 3.5 m (or 8’ to 12’).
A performer has as much horizontal movement as they do distance from the centre line of the sensor’s field of view. The result is a roughly trapezoidal tracking space. While the sensor can track performers closer or further beyond this range, this area is the “sweet spot” for full-body movement tracking.
The Kinect tracks human movement using infrared light. As such, it can “see” in total darkness. However, you should be aware that some theatrical lighting equipment does cast light in the infrared spectrum, and performers wearing highly reflective materials may interfere with accurate tracking. Plan carefully!
Multiple Bodies, Multiple Kinects
The Kinect is capable of tracking multiple human bodies.This tutorial and associated tools will facilitate full skeleton tracking of only a single human body, however.
If you wish to use multiple sensors to expand coverage or track multiple performers using Processing, each sensor needs to be connected to a separate computer. Be aware that multiple Xbox 360 Kinect cameras cause interference with each other, as the infrared light fields emitted by each device overlap and confuse the sensors.
What the Kinect can Track
The Xbox Kinect provides positional data on the X, Y, Z coordinates of the following body parts:
- The head.
- The neck.
- The torso.
- Right and left hips.
- Right and left knee.
- Right and left foot, at approximately the ankle.
- Right and left shoulder.
- Right and left elbows.
- Right and left hands, at approximately the wrist.
In total, a single performer produces 45 ‘channels’ of information.
As of writing, the Kinect does not track most fine hand movements or the rotation of body parts.
The Kinect tracks human bodies only. It does not reliably track non-human bodies or objects.
Finally, by using Syphon (Mac OS) or Spout (Windows), Isadora can receive images from the infrared and RGB cameras on the Kinect. You may find that this video image alone is extremely useful for tracking using the Eyes++ actor, especially if you wish to track objects or smaller parts of the body, or even place the sensor in a non-standard position such as above or below the performers.