Hand pose estimation and motion recognition using egocentric RGB-D video

Abstract

Manipulation performed by humans contains a lot of information that helps robots to learn how to handle objects. Since hand poses and motions are related to manipulated objects, extracting this information is one of the important tasks for robotics community. This paper presents a framework to recognize human manipulations including hand motions, hand poses, and shapes of manipulated objects using egocentric RGB-D videos. Our framework is straightforward but powerful through the efficient use of depth information and egocentric vision. We estimate hand poses with an example-based method through the limited appearances of the hand in egocentric vision. First, from a sensed point cloud, our framework distinguishes hands, manipulated objects and an environment using skin color detection and limitation on the range of the moving hand. Next, we estimate a hand pose by aligning the extracted hand point cloud with a pre-recorded database of hand point clouds of different poses. The position and orientation of the head-mounted sensor are estimated to acquire the hand motion in the world coordinate system. Then, the type of hand motion is classified using Dynamic Programming matching between a series of velocity vectors of estimated and a database of wrist trajectories. Finally, we experiment the effectiveness of our framework for hand motion recognition to validate our work.