Deep Learning-based Action Recognition Research Articles

Designing an Athletic Training Assistance System under Deep Learning Guided by Kinesiology Theory

Abstract The combined application of computer artificial intelligence technology and sports promotes the positive development of the sports industry. The combined application of computer artificial intelligence technology and sports promotes the positive development of the sports industry. After completing the model construction, the structure of the external limb sports training auxiliary machinery is designed, and the system structure of the sports training auxiliary machinery is constructed by combining the needs of sports training. The optimization of external limb sports training auxiliary machinery was achieved by performing and solving mathematical modeling of a single limb. The performance of the action recognition model and the sports auxiliary machinery is examined through experiments, in which the action recognition model based on deep learning in this paper has the best effect, which is 95.75%, and the loss values of the model after convergence are all below 0.2. In the empirical testing of the performance of the sports assistive machine, the obtained target trajectory tracking error is within ±3mm and ±2mm, respectively, and the performance of the deep learning-based action recognition model and the external limb sports training assistive machine is excellent, which can effectively help the sports training. The purpose of this study is to provide design direction and practical significance for the design of a sports training assistance system.

A Survey on 3D Skeleton-Based Action Recognition Using Learning Method.

Three-dimensional skeleton-based action recognition (3D SAR) has gained important attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or red-green-blue (RGB) data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3-dimensional (3D) skeleton data as a valuable modality. Subsequently, we provide a comprehensive introduction to mainstream action recognition techniques based on 4 fundamental deep architectures, i.e., recurrent neural networks, convolutional neural networks, graph convolutional network, and Transformers. All methods with the corresponding architectures are then presented in a data-driven manner with detailed discussion. Finally, we offer insights into the current largest 3D skeleton dataset, NTU-RGB+D, and its new edition, NTU-RGB+D 120, along with an overview of several top-performing algorithms on these datasets. To the best of our knowledge, this research represents the first comprehensive discussion of deep learning-based action recognition using 3D skeleton data.

A Highly Compressed Accelerator With Temporal Optical Flow Feature Fusion and Tensorized LSTM for Video Action Recognition on Terminal Device

Deep learning based action recognition has become ubiquitous in the video analysis area; however, large neural networks require enormous computations to achieve high performance, which hinder them from mobile applications that are tightly constrained by hardware resources. In this work, we introduce a highly compact and fast neural network based Action Recognition Accelerator named ARA on the terminal device. We build an LSTM based spatio-temporal action recognition model with extracted time-series features from RGB frames and flow features from optical flow fields. Then the LSTM based spatio-temporal model is deeply compressed with tensor decomposition to further reduce redundant parameters and lessen computation overhead. Based on the datasets UCF-11, UCF-101, and HMDB51, our proposed method achieves 95.87%, 94.08%, and 75.71% classification accuracy, being comparable with other state-of-the-art methods. In particular, our proposed method significantly compresses the parameter of the LSTM model 215× on the UCF-101 dataset. The proposed system can also achieve a fast running speed of 157.7 FPS on GPU. Furthermore, we validate the performance of the proposed system on an ARM-based terminal device; the results show it only has 0.017s latency and 4.73W power consumption.

The application of body symbols in contemporary art from the perspective of Da Vinci’s theory

Abstract Motion recognition methods can distinguish body symbols and play an important role in real-time safety monitoring of pedestrians. In order to solve the problem of lack of data for deep learning-based action recognition models, a human motion synthesis method with hidden state initialization is proposed. Commonly used human motion synthesis methods utilize Recurrent Neural Network (RNN) to automatically generate subsequent motion sequences by taking several frames of motion data as input. In previous work, the initial hidden state of the RNN is usually set to zero or randomly initialized hidden state, resulting in a jump between the end frame of motion input and the first frame of motion synthesis, which affects the quality of the generated motion. To solve this problem, a method for estimating the initial hidden state estimation is proposed, which takes the initial hidden state as the independent variable, uses the objective function of the neural network as the optimization objective, and uses the gradient descent method to optimize the solution to obtain a suitable initial hidden state. The proposed motion model with initial hidden state estimation reduces the prediction error in the first frame by 63.51% and 6.90%, respectively, and the total error in 10 frames by 50.00% and the total errors of 10 frames were reduced by 50.00% and 4.89%, respectively. The experimental results show that the proposed motion model with initial hidden state estimation has better motion synthesis quality and motion prediction accuracy than the method without initial hidden state estimation; the proposed method improves the quality of motion synthesis by accurately estimating the first frame hidden state of the RNN human motion model, which can provide reliable data support for motion recognition models in real-time security monitoring.

Special Issue on Deep Learning-Based Action Recognition

Human action recognition (HAR) has gained popularity because of its various applications, such as human–object interaction [...]

Identifying the key frames: An attention-aware sampling method for action recognition

Deep learning based methods have achieved remarkable progress in action recognition. Existing works mainly focus on designing novel deep architectures to learn video representations for action recognition. Most existing methods treat sampled frames equally and average all the frame-level predictions to generate video-level predictions at the testing stage. However, within a video, discriminative actions may occur sparsely in a few frames whereas most other frames are irrelevant to the ground truth which may even lead to wrong results. As a result, we think that the strategy of selecting relevant frames would be a further important key to enhance the existing deep learning based action recognition. In this paper, we propose an attention-aware sampling method for action recognition, which aims to discard the irrelevant and misleading frames and preserve the most discriminative frames. We formulate the process of mining key frames from videos as a Markov decision process and train the attention agent through deep reinforcement learning without extra labels. The agent takes features and predictions from the baseline model as inputs and generates importance scores for all frames. Moreover, our approach is extensible, which can be applied to different existing deep learning based action recognition models. We achieve very competitive action recognition performance on two widely used action recognition datasets.

An Action Recognition Technology for Badminton Players Using Deep Learning

With the fast development of sports in recent years, the number of people participating in various sports has increased day by day. Because of the advantages of fewer constraints on the field and ease of learning, badminton became one of the most popular sports among them. Numerous works have been done specifically for the recognition of action of badminton players to improve and popularize it, but the traditional badminton player’s badminton action recognition algorithm employs the method of manually constructing a topology map to model the action sequence contained in multiple video frames. Besides, it learns each video frame in a targeted manner to reflect data changes, which is prone to high computational cost, low network generalization, and catastrophic forgetting. In response to the above problems, this paper proposes a deep learning-based action recognition technology for badminton players, which re-encodes the human hitting action sequence data with multirelational characteristics into relational triples, and learns by decoupling based on long short-term memory network. At the same time, this paper designs and completes a set of badminton action recognition schemes based on acceleration and angular velocity signals. Experimental results show that the proposed method achieves 63%, 84%, and 92%, respectively, recognition accuracy on multiple benchmark datasets, which improves the accuracy of human hitting action recognition. As a result, the evaluation will be useful in future work to improve the structure of current deep learning models for higher results in badminton action recognition.

STA-TSN: Spatial-Temporal Attention Temporal Segment Network for action recognition in video.

Most deep learning-based action recognition models focus only on short-term motions, so the model often causes misjudgments of actions that are combined by multiple processes, such as long jump, high jump, etc. The proposal of Temporal Segment Networks (TSN) enables the network to capture long-term information in the video, but ignores that some unrelated frames or areas in the video can also cause great interference to action recognition. To solve this problem, a soft attention mechanism is introduced in TSN and a Spatial-Temporal Attention Temporal Segment Networks (STA-TSN), which retains the ability to capture long-term information and enables the network to adaptively focus on key features in space and time, is proposed. First, a multi-scale spatial focus feature enhancement strategy is proposed to fuse original convolution features with multi-scale spatial focus features obtained through a soft attention mechanism with spatial pyramid pooling. Second, a deep learning-based key frames exploration module, which utilizes a soft attention mechanism based on Long-Short Term Memory (LSTM) to adaptively learn temporal attention weights, is designed. Third, a temporal-attention regularization is developed to guide our STA-TSN to better realize the exploration of key frames. Finally, the experimental results show that our proposed STA-TSN outperforms TSN in the four public datasets UCF101, HMDB51, JHMDB and THUMOS14, as well as achieves state-of-the-art results.

Transfer Learning-enabled Action Recognition for Human-robot Collaborative Assembly

Human-robot collaboration (HRC) is critical to today’s tendency towards high-flexible assembly in manufacturing. Human action recognition, as one of the core prerequisites for HRC, enables industrial robots to understand human intentions and to execute planning adaptively. However, existing deep learning-based action recognition methods rely heavily on a huge amount of annotation data, which may not be effective or realistic in practice. Therefore, a transfer learning-enabled action recognition approach is proposed in this research to facilitate robot reactive control in HRC assembly. Meanwhile, a decision-making mechanism for robotic planning is introduced as well. Lastly, the proposed approach is evaluated in an aircraft bracket assembly scenario to reveal its significance.

Few-shot activity recognition with cross-modal memory network

Deep learning based action recognition methods require large amount of labelled training data. However, labelling large-scale video data is time consuming and tedious. In this paper, we consider a more challenging few-shot action recognition problem where the training samples are few and rare. To solve this problem, memory network has been designed to use an external memory to remember the experience learned in training and then apply it to few-shot prediction during testing. However, existing memory-based methods just update the visual information with fixed label embeddings in the memory, which cannot adapt well to novel activities during testing. To alleviate the issue, we propose a novel end-to-end cross-modal memory network for few-shot activity recognition. Specifically, the proposed memory architecture stores the dynamic visual and textual semantics for some high-level attributes related to human activities. And the learned memory can provide effective multi-modal information for new activity recognition in the testing stage. Extensive experimental results on two video datasets, including HMDB51 and UCF101, indicate that our method could achieve significant improvements over other previous methods.

Batch Entropy Supervised Convolutional Neural Networks for Feature Extraction and Harmonizing for Action Recognition

Deep learning-based action recognition in videos has obtained much attention because of achieving remarkable performance in diverse applications. However, due to the heterogeneous background and noisy spatio-temporal cues, extracting highly discriminative features is still quite challenging. To deal with this problem, numerous methods have been published based on the attention mechanism and skeleton modality. Instead of focusing on data pre-processing, we shed light on the feature map and concentrate on extracting highly discriminative features. First, we introduce Batch-wise Entropy Supervised Stream (BESS) to extend feature discrimination similar to the uncertainty of the corresponding batch. Secondly, to obtain a more generalized model, we propose a Stream to Harmonize the feature discrimination by Augmenting both Features (HAFS) of ResNext101 and BESS. These two streams are hallucinated by the distillation and feature fusion technique effectively into HAFS. We introduce a new metric to assess the characteristics of the feature map. This metric depicts the relationship between the feature discrimination and recognition accuracy. Finally, we comprehensively evaluate our approach on two benchmark datasets, HMDB51 and UCF101. Experimental results demonstrate that, extending and then harmonizing the feature discrimination is one of the effective ways of generating highly discriminative features. Experimental outcomes indicate the superiority of our proposed technique over the existing state-of-the-art methods.

ConvNets-based action recognition from skeleton motion maps

With the advance of deep learning, deep learning based action recognition is an important research topic in computer vision. The skeleton sequence is often encoded into an image to better use Convolutional Neural Networks (ConvNets) such as Joint Trajectory Maps (JTM). However, this encoding method cannot effectively capture long temporal information. In order to solve this problem, This paper presents an effective method to encode spatial-temporal information into color texture images from skeleton sequences, referred to as Temporal Pyramid Skeleton Motion Maps (TPSMMs), and Convolutional Neural Networks (ConvNets) are applied to capture the discriminative features from TPSMMs for human action recognition. The TPSMMs not only capture short temporal information, but also embed the long dynamic information over the period of an action. The proposed method has been verified and achieved the state-of-the-art results on the widely used UTD-MHAD, MSRC-12 Kinect Gesture and SYSU-3D datasets.

Attention-Aware Sampling via Deep Reinforcement Learning for Action Recognition

Deep learning based methods have achieved remarkable progress in action recognition. Existing works mainly focus on designing novel deep architectures to achieve video representations learning for action recognition. Most methods treat sampled frames equally and average all the frame-level predictions at the testing stage. However, within a video, discriminative actions may occur sparsely in a few frames and most other frames are irrelevant to the ground truth and may even lead to a wrong prediction. As a result, we think that the strategy of selecting relevant frames would be a further important key to enhance the existing deep learning based action recognition. In this paper, we propose an attentionaware sampling method for action recognition, which aims to discard the irrelevant and misleading frames and preserve the most discriminative frames. We formulate the process of mining key frames from videos as a Markov decision process and train the attention agent through deep reinforcement learning without extra labels. The agent takes features and predictions from the baseline model as input and generates importance scores for all frames. Moreover, our approach is extensible, which can be applied to different existing deep learning based action recognition models. We achieve very competitive action recognition performance on two widely used action recognition datasets.

Robust Multi-Feature Learning for Skeleton-Based Action Recognition

Skeleton-based action recognition has advanced significantly in the past decade. Among deep learning-based action recognition methods, one of the most commonly used structures is a two-stream network. This type of network extracts high-level spatial and temporal features from skeleton coordinates and optical flows, respectively. However, other features, such as the structure of the skeleton or the relations of specific joint pairs, are sometimes ignored, even though using these features can also improve action recognition performance. To robustly learn more low-level skeleton features, this paper introduces an efficient fully convolutional network to process multiple input features. The network has multiple streams, each of which has the same encoder-decoder structure. A temporal convolutional network and a co-occurrence convolutional network encode the local and global features, and a convolutional classifier decodes high-level features to classify the action. Moreover, a novel fusion strategy is proposed to combine independent feature learning and dependent feature relating. Detailed ablation studies are performed to confirm the network's robustness to all feature inputs. If more features are combined and the number of streams increases, performance can be further improved. The proposed network is evaluated on three skeleton datasets: NTU-RGB + D, Kinetics, and UTKinect. The experimental results show its effectiveness and performance superiority over state-of-the-art methods.

Action Recognition Using a Far-Infrared Sensor Array by Convolutional RNN

This paper proposes a Deep Learning-based action recognition method from an extremely low-resolution FIR image sequence. The method recognizes daily actions by humans (e.g. walking, sitting down, standing up, and so on) and abnormal actions (e.g. falling down) without privacy concerns. While privacy concerns can be ignored, it is difficult to compute feature points and to obtain a clear edge of the human body from an extremely low-resolution FIR image. To address these problems, this paper proposes a Deep Learning-based action recognition method whose inputs are the FIR images and their frame differences cropped by the gravity center of human regions.

3D-based Deep Convolutional Neural Network for action recognition with depth sequences

Traditional algorithms to design hand-crafted features for action recognition have been a hot research area in the last decade. Compared to RGB video, depth sequence is more insensitive to lighting changes and more discriminative due to its capability to catch geometric information of object. Unlike many existing methods for action recognition which depend on well-designed features, this paper studies deep learning-based action recognition using depth sequences and the corresponding skeleton joint information. Firstly, we construct a 3D-based Deep Convolutional Neural Network (3D2CNN) to directly learn spatio-temporal features from raw depth sequences, then compute a joint based feature vector named JointVector for each sequence by taking into account the simple position and angle information between skeleton joints. Finally, support vector machine (SVM) classification results from 3D2CNN learned features and JointVector are fused to take action recognition. Experimental results demonstrate that our method can learn feature representation which is time-invariant and viewpoint-invariant from depth sequences. The proposed method achieves comparable results to the state-of-the-art methods on the UTKinect-Action3D dataset and achieves superior performance in comparison to baseline methods on the MSR-Action3D dataset. We further investigate the generalization of the trained model by transferring the learned features from one dataset (MSR-Action3D) to another dataset (UTKinect-Action3D) without retraining and obtain very promising classification accuracy.