Hand segmentation under different viewpoints by combination of Mask R-CNN with tracking

Abstract

This paper presents a new method for hand segmentation from images and video. The method based mainly on an advanced technique for instance segmentation (Mask RCNN) which has been shown very efficient in segmentation task on COCO dataset. However, Mask R-CNN has some limitations. It works on still images, so cannot explore temporal information of the object of interest such as dynamic hand gestures. Second Mask R-CNN usually fails to detect object suffered from motion blur at low resolution as hand. Our proposed method improves Mask R-CNN by integrating a Mean Shift tracker that tracks hands in consecutive frames and removes false alarms. We have also trained another model of Mask R-CNN on cropped regions extended from hand centers to obtain a better accuracy of segmentation. We have evaluated both methods on a self-constructed multi-view dataset of hand gestures and show how robust these methods are to view point changes. Experimental results showed that our method achieved better performance than the original Mask R-CNN under different viewpoints.