Pointing Device Based on Estimation of Trajectory and Shape of a Human Hand in a Monocular Image Sequence

Abstract

Pointing devices are essential components of graphical user interfaces, and the mouse in particular is widely used because of its intuitive and easy operation. Since it must be directly touched by the user, however, the mouse is restricted in the locations where it can be used. A pointing device consists of a pointing mechanism and a switching mechanism, so the use of a noncontact device to carry out these actions should remove the restriction concerning location. In this study, we investigate the construction of a pointing device that does not impart a feeling of restraint or awkwardness, which estimates the user’s hand shape and position from images captured by a monocular camera, a noncontact device. In this system, the captured image is transformed from a Cartesian coordinate system to a log-polar system to reduce image data and computational cost, and achieve real-time operation without using special hardware other than a regular camera. Higher order local autocorrelation features of the log-polar coordinate space were used to achieve robustness against background change and hand rotation. In addition to direct pointing, the ability to recognize gestures from the hand’s motion trajectory was incorporated to achieve more comfortable user-computer interaction. In experiments using a system consisting of a regular computer and digital video camera, tracking of the hand and estimation of symbolic signs from extracted frames was stable at a practical average speed of 30 ms per frame.