MultiPoseSeg: Feedback Knowledge Transfer for Multi-Person Pose Estimation and Instance Segmentation

Abstract

Multi-person pose estimation and instance segmentation suffer large performance loss when images are with an increasing number of people and their uncontrolled complex appearance. Yet, existing models cannot efficiently leverage unbalanced training images, i.e., few of them are with multi-person, and most are with single-person, making them ineffective for challenging multi-person scenarios. To tackle multi-person cases with a limited portion of them, we propose MultiPoseSeg, a data preparation and feedback knowledge transfer system designed for multi-person pose estimation and instance segmentation. First, MultiPoseSeg categorically performs random data reduction to reduce the single-person bias from the train dataset. Second, MultiPoseSeg employs the knowledge transfer from ancestor models to converge the model learning with a limited amount of data and time. This way, our model learns and train on human pose and instance segmentation to advance the training and testing accuracy. Finally, MultiPoseSeg proposes keypoint maps to identify the keypoint coordinates for soft and hard keypoints and segmentation maps to assign centroid to each human instance, which helps to cluster the pixels to a particular instance. We have evaluated MultiPoseSeg using COCO and OCHuman challenging datasets and demonstrated MultiPoseSeg outperforms state-of-the-art bottom-up models in terms of both accuracy and runtime performance, achieving 0.728 mAP for pose and 0.445 mAP for segmentation on COCO dataset. All the unbiased data and code has been made available at: https://github.com/RaiseLab/MultiPoseSeg