Human Action Recognition In Videos Research Articles

Multifeature fusion action recognition based on key frames

SummaryAs an important technology in computer vision, video‐based human action recognition has a great commercial value, which has attracted extensive attention in the field of computer vision and pattern recognition in both academia and industry. To date, there are a wide variety of applications of human action recognition, such as surveillance, robotics, health care, video searching, and human–computer interaction. However, there are many challenges involved in human action recognition in videos, such as cluttered backgrounds, occlusions, viewpoint variation, execution rate, and camera motion. However, data redundancy and single feature were largely limited the accuracy of human action recognition. In this article, adopting the key frame extraction and multifeature fusion techniques, a novel action recognition method was proposed, which can improve the recognition accuracy. The main works are as follows: 1) in order to solve the problem of data redundancy, a key frame extraction method based on node contribution weighting is proposed to extract video key frames; 2) different kinds of information flows are extracted from the obtained key frame sequences, and different convolutional neural networks are used to obtain corresponding classification results and merge, so as to better complement the information in different flows. Lastly, the experimental results show that our method improves the accuracy of action recognition.

Human Action Recognition in Videos using Convolution Long Short-Term Memory Network with Spatio-Temporal Networks

Two-stream convolutional networks plays an essential role as a powerful feature extractor in human action recognition in videos. Recent studies have shown the importance of two-stream Convolutional Neural Networks (CNN) to recognize human action recognition. Recurrent Neural Networks (RNN) has achieved the best performance in video activity recognition combining CNN. Encouraged by CNN's results with RNN, we present a two-stream network with two CNNs and Convolution Long-Short Term Memory (CLSTM). First, we extricate Spatio-temporal features using two CNNs using pre-trained ImageNet models. Second, the results of two CNNs from step one are combined and fed as input to the CLSTM to get the overall classification score. We also explored the various fusion function performance that combines two CNNs and the effects of feature mapping at different layers. And, conclude the best fusion function along with layer number. To avoid the problem of overfitting, we adopt the data augmentation techniques. Our proposed model demonstrates a substantial improvement compared to the current two-stream methods on the benchmark datasets with 70.4% on HMDB-51 and 95.4% on UCF-101 using the pre-trained ImageNet model. Doi: 10.28991/esj-2021-01254 Full Text: PDF

Analysis of Deep Neural Networks for Human Activity Recognition in Videos—A Systematic Literature Review

From the past few decades, Human activity recognition (HAR) is one of the vital research areas in computer vision in which much research is ongoing. The researcher’s focus is shifting towards this area due to its vast range of real-life applications to assist in daily living. Therefore, it is necessary to validate its performance on standard benchmark datasets and state-of-the-art systems before applying it in real-life applications. The primary objective of this Systematic Literature Review (SLR) is to collect existing research on video-based human activity recognition, summarize, and analyze the state-of-the-art deep learning architectures regarding various methodologies, challenges, and issues. The top five scientific databases (such as ACM, IEEE, ScienceDirect, SpringerLink, and Taylor & Francis) are accessed to accompany this systematic study by summarizing 70 different research articles on human activity recognition after critical review. Human activity recognition in videos is a challenging problem due to its diverse and complex nature. For accurate video classification, extraction of both spatial and temporal features from video sequences is essential. Therefore, this SLR focuses on reviewing the recent advancements in stratified self-deriving feature-based deep learning architectures. Furthermore, it explores various deep learning techniques available for HAR, challenges researchers to face to build a robust model, and state-of-the-art datasets used for evaluation. This SLR intends to provide a baseline for video-based human activity recognition research while emphasizing several challenges regarding human activity recognition accuracy in video sequences using deep neural architectures.

Efficient Feature Mapping in Classifying Proportional Data

In image classification, traditional kernels or feature mapping functions of Support Vector Machine(SVM) use discriminative features without considering the true nature of the data. Our work in this paper is motivated by the need to consider intrinsic distribution of L1 normalized histograms and develop a flexible feature mapping technique by combining histogram based features and distribution based density features. The proposed mapping technique contains prior knowledge about the the data which provides a flexible representation and thus increases the discriminative power of the classifier. Such flexibility is achieved due to the explanatory capabilities of Dirichlet, generalized Dirichlet and Beta-Liouville distributions to model proportional data. In addition to that, we present a general framework to estimate the parameters of these distributions by taking maximum likelihood (MLE) approach. Experimental results show that the proposed technique increases the effectiveness of SVM kernels for different computer vision tasks such as natural scene recognition, satellite image classification and human action recognition in videos.

Human Action Recognition in Videos using a Robust CNN LSTM Approach

Action recognition in videos is currently a topic of interest in the area of computer vision, due to potential applications such as: multimedia indexing, surveillance in public spaces, among others. In this paper we propose (1) Implement a CNN–LSTM architecture. First, a pre-trained VGG16 convolutional neural network extracts the features of the input video. Then, an LSTM classifies the video in a particular class. (2) Study how the number of LSTM units affects the performance of the system. To carry out the training and test phases, we used the KTH, UCF-11 and HMDB-51 datasets. (3) Evaluate the performance of our system using accuracy as evaluation metric. We obtain 93%, 91% and 47% accuracy respectively for each dataset.

Pose-Guided Inflated 3D ConvNet for action recognition in videos

Human action recognition in videos is still an important while challenging task. Existing methods based on RGB image or optical flow are easily affected by clutters and ambiguous backgrounds. In this paper, we propose a novel Pose-Guided Inflated 3D ConvNet framework (PI3D) to address this issue. First, we design a spatial–temporal pose module, which provides essential clues for the Inflated 3D ConvNet (I3D). The pose module consists of pose estimation and pose-based action recognition. Second, for multi-person estimation task, the introduced pose estimation network can determine the action most relevant to the action category. Third, we propose a hierarchical pose-based network to learn the spatial–temporal features of human pose. Moreover, the pose-based network and I3D network are fused at the last convolutional layer without loss of performance. Finally, the experimental results on four data sets (HMDB-51, SYSU 3D, JHMDB and Sub-JHMDB) demonstrate that the proposed PI3D framework outperforms the existing methods on human action recognition. This work also shows that posture cues significantly improve the performance of I3D.

A fast human action recognition network based on spatio-temporal features

Artificial intelligence models are widely used in the field of human activity recognition, and human action recognition is an important aspect of human activity recognition. The core of human action recognition is to understand the temporal relationship between video frames. Almost all state-of-the-art methods of human action recognition in videos use optical flow. However, traditional local optical flow estimation methods areexpensive and not trained end-to-end. In this paper, we propose a fast network for human action recognition. Our purpose is to improve the efficiency of optical flow feature extraction and explore the fusion method of spatio-temporal features. For spatio-temporal features, our method combines spatial features and temporal features into fusion features. In addition, we propose CNN with OFF instead of the VGG16 network, which is used to process optical flow features to obtain abundant features. Our model only needs RGB inputs to get the state-of-the-art accuracy of 91.5% on UCF-101, 67.9% on HMDB51, 83.3% on MSR Daily Activity3D, and 91.25% on Florence 3D action, respectively. Compared with most state-of-the-art video action recognition models, our proposed model can effectively improve the accuracy of human action recognition.

Distinct Two-Stream Convolutional Networks for Human Action Recognition in Videos Using Segment-Based Temporal Modeling

The Two-stream convolution neural network (CNN) has proven a great success in action recognition in videos. The main idea is to train the two CNNs in order to learn spatial and temporal features separately, and two scores are combined to obtain final scores. In the literature, we observed that most of the methods use similar CNNs for two streams. In this paper, we design a two-stream CNN architecture with different CNNs for the two streams to learn spatial and temporal features. Temporal Segment Networks (TSN) is applied in order to retrieve long-range temporal features, and to differentiate the similar type of sub-action in videos. Data augmentation techniques are employed to prevent over-fitting. Advanced cross-modal pre-training is discussed and introduced to the proposed architecture in order to enhance the accuracy of action recognition. The proposed two-stream model is evaluated on two challenging action recognition datasets: HMDB-51 and UCF-101. The findings of the proposed architecture shows the significant performance increase and it outperforms the existing methods.

Video classification by fusing two-stream image template classification and pretrained network

A motion energy image (MEI) is a spatial template that collapses regions of motion into a single image in which more moving pixels are brighter than others. The forward single-step history image (fSHI) is a spatiotemporal template that shows the presence and direction of motion. Each video can be described using these templates. Recently, the popularity of deep learning architectures for human activity recognition encourages us to explore the effectiveness of combining them with these templates. Hence, three new methods are introduced to convert the problem of human activity recognition in video into image templates classification. In method 1, each video is split into N groups of consecutive frames, and the MEI is computed for each group. Transfer learning with the fine-tuning technique is used for classifying these templates. fSHIs or spatiotemporal templates are used for classification in method 2, similar to method 1. Fusing the two streams of these templates is introduced as method 3. Among these methods, method 3 outperforms the others and is called the proposed method. It achieves recognition accuracies of 92.60% and 93.40% for the UCF Sport and UCF-11 action datasets, respectively. Also, it is compared with state-of-the-art approaches, and the results show that the proposed method has the best performance.

Emerging intelligent big data analytics for cloud and edge computing

Emerging intelligent big data analytics for cloud and edge computing

Human action recognition in videos based on spatiotemporal features and bag-of-poses

Currently, there is a large number of methods that use 2D poses to represent and recognize human action in videos. Most of these methods use information computed from raw 2D poses based on the straight line segments that form the body parts in a 2D pose model in order to extract features (e.g., angles and trajectories). In our work, we propose a new method of representing 2D poses. Instead of directly using the straight line segments, firstly, the 2D pose is converted to the parameter space in which each segment is mapped to a point. Then, from the parameter space, spatiotemporal features are extracted and encoded using a Bag-of-Poses approach, then used for human action recognition in the video. Experiments on two well-known public datasets, Weizmann and KTH, showed that the proposed method using 2D poses encoded in parameter space can improve the recognition rates, obtaining competitive accuracy rates compared to state-of-the-art methods.

Deep metric learning for open-set human action recognition in videos

Human action recognition (HAR) is a topic widely studied in computer vision and pattern recognition. Despite the success of recent models for this issue, most of them approach HAR from the closed-set perspective. The closed-set recognition works under the assumption that all classes are known a priori and they appear during the training and test phase. Unlike most previous works, we approach HAR from the open-set perspective, that is, previously unknown classes are considered in the model. Additionally, feature extraction for HAR in the context of open set is still underexplored in the recent literature, since one needs to represent known classes with a low intra-class variance to reject unknown examples. To achieve this task, we propose a deep metric learning model named triplet inflated 3D convolutional neural network (TI3D), which builds upon the well-known I3D model. TI3D is a representation learning model that takes as input video sequences and outputs 256-dimensional representations. We perform extensive experiments and statistical comparisons on the UCF-101 dataset using a 30-fold cross-validation procedure in 25 different scenarios with varying degrees of openness and a varying number of training and test classes. Results reveal that the proposed TI3D achieves better performance than non-metric learning models in terms of $$F_1$$ score and Youdens index, indicating a promising approach for open-set video action recognition.

Fine grained sport action recognition with Twin spatio-temporal convolutional neural networks

Human action recognition in video is one of the key problems in visual data interpretation. Despite intensive research, the recognition of actions with low inter-class variability remains a challenge. This paper presents a new Twin Spatio-Temporal Convolutional Neural Network (TSTCNN) for this purpose. When applied to table tennis, it is possible to detect and recognize 20 table tennis strokes. The model has been trained on a specific dataset, so called TTStroke-21, recorded in natural conditions at the Faculty of Sports of the University of Bordeaux. Our model takes as inputs an RGB image sequence and its computed Optical Flow. The proposed Twin architecture is a two stream network both comprising 3 spatio-temporal convolutional layers, followed by a fully connected layer where data are fused. Our method reaches an accuracy of 91.4% against 43.1% for our baseline, a Two-Stream Inflated 3D ConvNet (I3D).

Data-level information enhancement: Motion-patch-based Siamese Convolutional Neural Networks for human activity recognition in videos

Data augmentation is critical for deep learning-based human activity recognition (HAR) systems. However, conventional data augmentation methods, such as random-cropping, may generate bad samples that are unrelated to a particular activity (e.g. the background patches without saliency motion information). As a result, the random-cropping based data augmentation may affect negatively the overall performance of HAR systems. Humans, in turn, tend to pay more attention to motion information when recognizing activities. In this work, we attempt to enhance the motion information in HAR systems and mitigate the influence of bad samples through a Siamese architecture, termed as Motion-patch-based Siamese Convolutional Neural Network (MSCNN). The term motion patch is defined as a specific square region that includes critical motion information in the video. We propose a simple yet effective method for selecting those regions. To evaluate the proposed MSCNN, we conduct a number of experiments on the popular datasets UCF-101 and HMDB-51. The mathematical model and experimental results show that the proposed architecture is capable of enhancing the motion information and achieves comparable performance.

Human Action Recognition Based on Multiple Features and Modified Deep Learning Model

In order to improve the accuracy of human action recognition in video and the computational efficiency of large data sets, an action recognition algorithm based on multiple features and modified deep learning model is proposed. First, the deep network pre-training process is used to learn and optimize the RBM parameters, and the deep belief nets (DBN) model is constructed through deep learning. Then, human 13 joint points and critical points of optical flow are automatically extracted by DBN model. Second, these more abstract and more effective human motion features are combined to represent human actions. Ultimately, the entire DBN network structure is fine-tuned by support vector machine (SVM) algorithm to classify human actions. We demonstrate that human 13 joint points and critical points of optical flow are two very effective human action characterizations, our proposed approach greatly reduces the required samples, and shortens the training time of the samples, can efficiently process large data sets and can effectively recognize novel actions. We performed experiments on the KTH data set, Weizmann data set, the ballet data set and UCF101 data set to evaluate the proposed method, the experiment results show that the average recognition accuracy is over 98%, which validates its effectiveness, and show that our results are stable, reliable, and significantly better than the results of two state-of-the-art approaches on four different data sets. So, it lays a good theoretical foundation for practical applications.

A two-fold transformation model for human action recognition using decisive pose

Human action recognition in videos is a tough task due to the complex background, geometrical transformation and an enormous volume of data. Hence, to address these issues, an effective algorithm is developed, which can identify human action in videos using a single decisive pose. To achieve the task, a decisive pose is extracted using optical flow, and further, feature extraction is done via a two-fold transformation of wavelet. The two-fold transformation is done via Gabor Wavelet Transform (GWT) and Ridgelet Transform (RT). The GWT produces a feature vector by calculating first-order statistics values of different scale and orientations of an input pose, which have robustness against translation, scaling and rotation. The orientation-dependent shape characteristics of human action are computed using RT. The fusion of these features gives a robust unified algorithm. The effectiveness of the algorithm is measured on four publicly datasets i.e. KTH, Weizmann, Ballet Movement, and UT Interaction and accuracy reported on these datasets are 96.66%, 96%, 92.75% and 100%, respectively. The comparison of accuracies with similar state-of-the-arts shows superior performance.

Attention‐based spatial–temporal hierarchical ConvLSTM network for action recognition in videos

Human action recognition in videos is an important research topic in computer vision due to its wide applications. Actions naturally contain both spatial and temporal information. The key to action recognition is to model the spatial and temporal structures of actions. In this study, the authors propose an attention‐based spatial–temporal hierarchical convolutional long short‐term memory (ST‐HConvLSTM) network to model the structures of actions in the spatial and temporal domains. The ST‐HConvLSTM consists of two parts: a spatial–temporal attention module and a novel LSTM‐like architecture named hierarchical ConvLSTM (HConvLSTM). The HConvLSTM can model the spatial and temporal structures of actions. The spatial–temporal attention module can figure out which part of the video is more discriminative for action recognition and makes the HConvLSTM focus on it. In addition, a weighted fusion strategy is proposed to fuse the appearance information and motion information of the video. The proposed ST‐HConvLSTM is evaluated on UCF101, HMDB51 and Kinetics datasets. Experimental results show that the authors’ proposed ST‐HConvLSTM achieves state‐of‐the‐art performance compared with other recent LSTM‐like architectures and attention‐based methods.

Human action recognition based on action relevance weighted encoding

Human action recognition in videos is one of the most important and active topics in computer vision, and building more discriminative video representation is crucial for action classification. In this paper, we propose a new video representation which not only captures temporal dynamics in a video but also takes account of the importance of different trajectories and video frames in action recognition. More specifically, first, we propose a new trajectory descriptor based on traditionally used local descriptors including histograms of oriented gradients, histograms of optical flow and motion boundary histogram, and it takes the correlation between trajectories and target action into consideration. Secondly, a new method to compute saliency map based on optical flow is introduced to highlight regions of foreground motion. Thirdly, to identify discriminating trajectories and frames in a video, trajectory action relevance and frame action relevance are defined and used as weights during encoding. Finally, two video representations which describe a video at different levels of abstraction are proposed, they are complementary to each other and combined to generate more expressive and discriminative representation. Experimental results on KTH, JHMDB and HMDB51 demonstrate the effectiveness and excellent performance of the proposed approach.

Deep network for human action recognition using Weber motion

Effective motion estimation is one of the prime steps for any human action recognition (HAR) algorithm. Optical flow (OF) and motion history image (MHI) are two well-known methods for motion estimation in videos. OF has several advantages over MHI. But the major drawback with OF is that it is computationally very expensive as compared to the MHI. Therefore, in this paper, a new motion estimation technique named as Weber Motion History Image (WMHI) is proposed. Here, an extremely fast algorithm is proposed for HAR using WMHI, pose information, and convolutional neural network (CNN). In spite of being fast and less space consuming, the algorithm outperforms the existing pose based CNN results on five benchmark datasets namely JHMDB [1], sub-JHMDB [1], MPII [2] and HMDB51 [3] and UCF101 [4]. The work mainly focuses on a new efficient algorithm which can be implemented for real-time HAR in videos. For real-time implementation, the two basic criteria on which an algorithm can be analyzed are space and time complexity. The proposed algorithm is faster as compared to the existing OF based HAR systems. In terms of space complexity, the feature size of the proposed algorithm is almost 50% of the existing OF based algorithm. The recognition results still outperform the existing result by a significant margin.

Vector space based augmented structural kinematic feature descriptor for human activity recognition in videos

A vector space based augmented structural kinematic (VSASK) feature descriptor is proposed for human activity recognition. An action descriptor is...