Exploring privileged information from simple actions for complex action recognition

Abstract

Complex action recognition is an important yet challenging problem in computer vision. Sufficient labeled training data are required for learning a robust model. However, labeling complex actions is often time-consuming and expensive. Considering that each complex action is composed of a sequence of simple actions, we propose a new learning framework for complex action recognition by using a sequence of existing simple actions. A matrix is first designed as the probability matrix by manual annotation, which encodes the occurrence of simple actions in complex actions. As the probability matrix depends on label information, it is only available for the training data and is regarded as privileged information. A new framework is proposed, which is called latent task learning with privileged information (LTL-PI). An efficient algorithm is also presented for solving the proposed LTL-PI formulation, which obtains the optimal sparse weight parameters. To validate the proposed LTL-PI algorithm, extensive experiments are carried out on two challenging complex action datasets: the Olympic Sports dataset and the UCF50 dataset. Experimental results show that LTL-PI is a promising development that improves the performance of complex action recognition, and the designed privileged information can offer promising enhancement.