Skeleton-based Action Recognition with Lie Group and Deep Neural Networks

Abstract

Skeleton-based action recognition has always been an important research topic of computer vision since the skeleton data is more robust to illumination and rotation. Traditional action recognition methods mainly rely on manual features. Among those methods, the skeleton feature representation modeled on Lie group can effectively describe the three-dimensional geometric relationship between joints. In recent years, deep learning methods such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) and Long Short-Term Memory Networks (LSTM) have also achieved good performance in action recognition. In order to obtain more spatio-temporal information, we combine manual features and deep learning methods to propose a deep neural network LS-LieNet with CNN and Bi-direction LSTM (Bi-LSTM) based on the LieNet [1] network. First, the LS-LieNet network inputs the extracted Lie group representation of skeleton into a special CNN network which is designed for Lie group. Second, the transformed Lie algebra features are fed into the Bi-LSTM network before the fully connected layer of CNN. Then, the predicted labels and scores of the two network softmax layers are merged to effectively recognize the action. The experiment results on the standard 3D human action dataset show that the proposed LS-LieNet can efficiently improve the accuracy of action recognition.