Multi-task learning for object keypoints detection and classification

Abstract

Object keypoints detection and classification are both central research topics in computer vision. Due to their wide range potential applications in the real world, substantial efforts have been taken to advance their performance. However, these two related tasks are mainly treated separately in previous works. We argue that keypoints detection and classification can be complementary tasks and beneficial to each other. Knowing the category of a object is able to reduce the searching space of keypoints detection models and facilitate more precise localization. On the other hand, having the knowledge of object keypoints can make classification models pay more attention on areas that are more associated with the object, which will inevitably promote classification accuracy. Embracing this observation, we propose to model keypoints detection and classification in a multi-task learning framework. Specifically, a multi-task deep network is designed and trained to conduct both tasks, where we devise the model structure delicately to carry out sufficient training of both tasks. Extensive experiments are set up on the AIFASHION DATASET and Human3.6M DATASET to validate our proposal, we show that our algorithm outperforms separate models trained individually on each task.