Natural interaction with virtual objects using vision-based six DOF sphere tracking

Abstract

A common task in computer entertainment is the ability to interact with virtual 3D objects. Interacting with these objects using standard computer input devices such as a mouse and keyboard can often be a difficult task. For this reason, Tangible User Interfaces (TUIs) were developed to allow more natural interaction with complex virtual objects by manipulating physical objects in a familiar way. Applying the movements of a physical object to control the movement of a virtual object is often done by embedding devices in the physical object, or by passively tracking the object. We present a new TUI system that includes a passive optical tracking method to determine the six degree-of-freedom (DOF) pose of a sphere in a real-time video stream, and then apply the pose to a virtual object. Our tracking system is based on standard computer vision techniques and applications of 3D geometry. The pose of the sphere is accurately resolved under partial occlusions, allowing the object to be manipulated by hand without a tracking failure. We demonstrate the use of our TUI system to control virtual 3D objects in an interactive way, proving to be a useful tool for computer entertainment applications.