Vision-Based Hand Detection for Registration of Virtual Objects in Augmented Reality

Abstract

This paper presents a markerless Augmented Reality (AR) framework which utilizes outstretched hand for registration of virtual objects in the real environment. Hand detection methods used in this work are purely based on computer vision algorithms to detect bare hand without assistance from markers or any other devices such as mechanical devices and magnetic devices. We use a stereo camera to capture video images, so that the depth information of the hand can be constructed. Skin color segmentation is employed to segment the hand region from the images. Instead of fiducial markers, the center of the palm and fingertips are tracked in real time. Incorporating the depth information computed, the 3D positions of these hand features are then used to estimate the 6DOF camera pose with respect to the hand, which in turn allows the virtual objects to be augmented onto the palm accurately. This method increases the ease of manipulation of AR objects. Users can inspect and manipulate the AR objects in an intuitive way by using their bare hands.