Action Recognition Algorithm of Spatio–Temporal Differential LSTM Based on Feature Enhancement

Kai Hu,Fei Zheng,Yiwu Ding,Junlan Jin,Liguo Weng

doi:10.3390/app11177876

Abstract

The Long Short-Term Memory (LSTM) network is a classic action recognition method because of its ability to extract time information. Researchers proposed many hybrid algorithms based on LSTM for human action recognition. In this paper, an improved Spatio–Temporal Differential Long Short-Term Memory (ST-D LSTM) network is proposed, an enhanced input differential feature module and a spatial memory state differential module are added to the network. Furthermore, a transmission mode of ST-D LSTM is proposed; this mode enables ST-D LSTM units to transmit the spatial memory state horizontally. Finally, these improvements are added into classical Long-term Recurrent Convolutional Networks (LRCN) to test the new network’s performance. Experimental results show that ST-D LSTM can effectively improve the accuracy of LRCN.

Highlights

Human action recognition involves many fields, such as computer vision, image processing, deep learning, etc
The basic Long Short-Term Memory (LSTM) unit, Spatio–Temporal LSTM (ST-LSTM) unit and ST-D LSTM unit were used in the stacking part of the LSTM, and the common connection mode; the zigzag connection mode and the differential connection mode corresponding to each unit were selected
The ST-D LSTM can satisfy the requirements of rapidity, accuracy, and stability

Summary

Introduction

Human action recognition involves many fields, such as computer vision, image processing, deep learning, etc. It is widely used in human–computer interaction [1], video surveillance [2], intelligent transportation, sports analysis, smart home, etc. Its research methods are divided into two categories: one is based on manual feature extraction [3–7], and the other is based on deep learning. There are noises [8] in the datasets, such as illumination, similar actions (like jogging and running), dynamic backgrounds, etc. These noises make manually extracted features ineffective in classification, so its related research is limited. The calculation speed of the iDT algorithm is very slow and it can not meet real-time requirements

Methods

Results

Conclusion