A Soft Proposal Segmentation Network (SPS-Net) for Hand Segmentation on Depth Videos

Abstract

Hand segmentation is an important prerequisite for acquiring accurate 3D hand poses on depth images, as it can significantly reduce the complicity of hand pose estimation. However, it has been ignored or treated as a trivial problem for a long time, since most of the hand pose estimation works suppose the hand part is given or can be easily segmented by a depth threshold, which is not practical in many realistic scenarios (e.g., side/egocentric view). In order to perform the robust hand segmentation in various scenarios, we propose a Soft Proposal Segmentation Network (SPS-Net). A key difference from existing hand segmentation methods is that the SPS-Net can utilize the temporal information on depth videos. As a benefit, significant performance gains over the existing methods can be obtained, as demonstrated on two popular public datasets with diverse camera viewpoints and background. Moreover, integrating SPS-Net with a simple 3D hand pose estimator, we achieved the leading result on the 3D Hand Pose Tracking Task of the Hand2017 Challenge - the only world-wide open challenge of its kind.