Multilevel LSTM for Action Recognition Based on Skeleton Sequence

Abstract

Skeleton-based human action recognition has a broad range of applications for human-computer interaction and intelligent monitoring. Human actions can be represented by the trajectory of the skeleton joints, which contain key information about actions. Long-term short-term memory (LSTM) networks exhibit outstanding performance in 3D human action recognition because they are capable of modeling dynamics and dependencies in sequential data. In this paper, we propose a skeleton-based multilevel LSTM network for action recognition. First, the data for each joint and its parent joint is used as input to a fine-grained subnet. Then the features of the upper body joint are merged into the upper body subnet, the features of the lower body are merged into the lower body subnet. Finally the features of the upper body subnets and lower body subnets are structured and fused to achieve higher recognition accuracy. Experimental results on the public data set NTU RGB+D demonstrate the effectiveness of the proposed network.