Structured RNN for human interaction

Abstract

Understanding human activities has been an important research area in computer vision. Generally, the authors can model the human interactions as a temporal sequence with the transition in relationships of humans and objects. Besides, many studies have proved the effectiveness of long short‐term memory (LSTM) on long‐term temporal dependency problems. Here, the authors proposed a novel structured recurrent neural network (S‐RNN) to model spatio‐temporal relationships between human subjects and objects in daily human interactions. The authors represent the evolution of different components and the relationships between them over time by several subnets. Then, the hidden representations of those relations are fused and fed into the later layers to obtain the final hidden representation. The final prediction is carried out by the single‐layer perceptron. The experimental results of different tasks on the CAD‐120, SBU‐Kinect‐Interaction, multi‐modal and multi‐view and interactive, and NTU RGB+D data sets showed advantages of the proposed method compared with the state‐of‐art methods.