Multi-views reinforced LSTM for video-based action recognition

Abstract

Recently, the long short-term memory network (LSTM) and attention mechanism have greatly boosted the research of video-based action recognition. For this task, feature extraction especially temporal feature extraction is essential. However, most studies focus on improving the temporal feature extraction ability of the model, ignoring the lack of temporal information in the input. To alleviate the issue above, we propose multi-views reinforced LSTM (MR-LSTM). First, we propose an innovative feature extractor named multi-views temporal feature extractor (MTFE) to extract multi-views temporal features from RGB frames in different views. Secondly, we propose multi-views reinforced attention (MRA) mechanism, which utilizes multi-views features to enrich the temporal information in the input of LSTM. MTFE and MRA mechanisms alleviate the lack of temporal information in the input of LSTM. Equipped with the modules above, LSTM can extract more discriminative temporal features. Finally, we propose non-fair fusion strategy to obtain more discriminative fusion features that are beneficial for classification. The ablation experiment demonstrates the effectiveness of all proposed modules. In comprehensive experiments on UCF101 and HMDB51 datasets, our architecture performs competitively against state-of-the-art methods.