Action Recognition In Video Sequences Research Articles

Temporal Fusion of Time-Distributed VGG-16 and LSTM for Precise Action Recognition in Video Sequences

Autonomous systems play a vital role in creating an ambient environment, but for comprehensive human-robot interaction, the system must interpret and identify specific human behaviours. The demand for recognizing human activity from videos has significantly increased across diverse domains, such as medicine, intelligent video surveillance, human-robot interaction, and ambient-assisted environments. Deep learning has emerged as a pioneering technique for human activity recognition from videos due to its ability to learn features from input videos automatically. This research proposes a novel approach for recognizing activities in videos by leveraging the combined power of the VGG-16 convolutional neural network (CNN) and the Long Short-Term Memory (LSTM). To assess the efficiency of our suggested methodology, experiments are conducted on the widely used benchmark dataset UCF-50. The experimental results demonstrate that our VGG-16 and LSTM-based model outperforms state-of-the-art methods for activity recognition, achieving higher accuracy and robustness across various activity categories.

Survey on artificial intelligence-based human action recognition in video sequences

In the artificial intelligence domain, human action recognition (HAR) has evolved as one of the major active research topics as a reason of diverse applications, namely video surveillance. The extensive variation types amidst routine human activities make the recognition procedure much more intricate. As a reason for an increment in the cameras’ usage, automated systems are essential to categorize activities, such as these utilizing computationally intelligent methodologies, namely machine learning and deep learning (DL). This paper’s organization covers all the characteristics of the HAR’s general framework. This work talks about the public dataset’s characteristics utilized that are aimed at HAR. After that, it summarizes and classifies the recently published research enhancement under the common framework. The merits and demerits of dimensionality reduction, action representation, and as well as action analysis methodologies are offered. After that, the outcomes exhibit the DL methodologies’ performance via graphs. This article gives a clear vision of what we can explore in the activity recognition area. The work talks about HAR’s challenges and its future directions.

LgNet: A Local-Global Network for Action Recognition and Beyond

This work addresses the task of action recognition in video sequences. In real world applications, this task is quite challenging due to the complex background of video content, the similarities between different types of actions, the dependence on a large amount of annotated data, and so on. Most of the existing methods fail to distinguish similar actions with the same static appearance and motion pattern. We attempt to address this issue from the perspective of a local-global view, considering videos as combinations of a set of action units (local semantic information) and their relations along temporal dimension (global relation information). To achieve this end, we propose a novel Local-global Networks (LgNet) to enhance recognition of similar action. Besides, we propose an end-to-end training method to decrease the reliance on annotated data. It combines self-supervised learning and supervised learning, which not only enables the model to learn video representations from a large number unannotated data but also avoids subsequent finetuning. The proposed training method can be flexibly equipped to a wide array of vision tasks. Experiments on several benchmark datasets show that our proposed model and training method achieve state-of-the-art performance.

A composite framework of deep multiple view human joints feature extraction and selection strategy with hybrid adaptive sunflower optimization‐whale optimization algorithm for human action recognition in video sequences

AbstractIn computer vision and pattern recognition field, video‐based human action recognition (HAR) is the most predominant research area. Object recognition is needed to recognize the subjects regarding video contents, which allows reactive enquiry in a large number of camera contents, mainly in security based platforms where there is a prevalent growth of closed circuit television cameras. Generally, object detectors that have high performance are trained on a large collection of public benchmarks. Identifying human activities from unconstrained videos is the primary challenging task. Further, the feature extraction and feature selection from these unconstrained videos is also considered as a challenging issue. For that, in this article a new composite framework of HAR model is constructed by introducing an efficient feature extraction and selection strategy. The proposed feature extraction model extracts multiple view features, human joints features based on the domain knowledge of the action and fuses them with deep high level features extracted by an improved fully resolution convolutional neural networks. Also, it optimizes the feature selection strategy using the hybrid whale optimization algorithm and adaptive sun flower optimization that maximizes the feature entropy, correlation. It minimizes the error rate for improving the recognition accuracy of the proposed composite framework. The proposed model is validated on four different datasets, namely, Olympics sports, Virat Release 2.0, HMDB51, and UCF 50 sports action dataset to prove its effectiveness. The simulation results show that the proposed composite framework outperforms all the existing human recognition model in terms of classification accuracy and detection rate.

Human Action Recognition in Video Sequence using Logistic Regression by Features Fusion Approach based on CNN Features

Human Action Recognition in Video Sequence using Logistic Regression by Features Fusion Approach based on CNN Features

Action representation and recognition through temporal co-occurrence of flow fields and convolutional neural networks

Many applications require action recognition skills, from human-machine interaction to intelligent video surveillance. Action recognition in video sequences cannot be based on simply processing raw color images or optical flow fields. Color images provide appearance information of moving objects, but lack motion features. They are also very sensitive to variations due to clothing and camera pose that badly affect the action recognition accuracy. In turn, raw optical flow measures instantaneous motion, not the overall dynamics of actions, and is sensitive to noise. More robust and meaningful motion features and classifiers are thus required for action recognition to be reliable. This paper proposes a new action recognition technique based on a deep convolutional neural network (CNN) fed with Histograms of Optical Flow Co-Occurrence (HOF-CO) motion features. HOF-CO is a robust motion representation previously proposed by the authors to encode the relative frequency of pairs of optical flow directions computed at each image pixel. Experimental results show that this approach outperforms state-of-the-art action recognition methods on three different public datasets KTH, UCF-11 Youtube and HOLLYWOOD2.

Timed-image based deep learning for action recognition in video sequences

The paper addresses two issues relative to machine learning on 2D + X data volumes, where 2D refers to image observation and X denotes a variable that can be associated with time, depth, wavelength, etc. The first issue addressed is conditioning these structured volumes for compatibility with respect to convolutional neural networks operating on 2D image file formats. The second issue is associated with sensitive action detection in the “2D + Time” case (video clips and image time series). For the data conditioning issue, the paper first highlights that referring 2D spatial convolution to its 1D Hilbert based instance is highly accurate for information compressibility upon tight frames of convolutional networks. As a consequence of this compressibility, the paper proposes converting the 2D + X data volume into a single meta-image file format, prior to machine learning frameworks. This conversion is such that any 2D frame of the 2D + X data is reshaped as a 1D array indexed by a Hilbert space-filling curve and the third variable X of the initial file format becomes the second variable in the meta-image format. For the sensitive action recognition issue, the paper provides: (i) a 3 category video database involving non-violent, moderate and extreme violence actions; (ii) the conversion of this database into a timed meta-image database from the 2D + Time to 2D conditioning stage described above and (iii) outstanding 2-level and 3-level violence classification results from deep convolutional neural networks operating on meta-image databases.

2-D Skeleton-Based Action Recognition via Two-Branch Stacked LSTM-RNNs

Action recognition in video sequences is an interesting field for many computer vision applications, including behavior analysis, event recognition, and video surveillance. In this article, a method based on 2D skeleton and two-branch stacked Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) cells is proposed. Unlike 3D skeletons, usually generated by RGB-D cameras, the 2D skeletons adopted in this article are reconstructed starting from RGB video streams, therefore allowing the use of the proposed approach in both indoor and outdoor environments. Moreover, any case of missing skeletal data is managed by exploiting 3D-Convolutional Neural Networks (3D-CNNs). Comparative experiments with several key works on KTH and Weizmann datasets show that the method described in this paper outperforms the current state-of-the-art. Additional experiments on UCF Sports and IXMAS datasets demonstrate the effectiveness of our method in the presence of noisy data and perspective changes, respectively. Further investigations on UCF Sports, HMDB51, UCF101, and Kinetics400 highlight how the combination between the proposed two-branch stacked LSTM and the 3D-CNN-based network can manage missing skeleton information, greatly improving the overall accuracy. Moreover, additional tests on KTH and UCF Sports datasets also show the robustness of our approach in the presence of partial body occlusions. Finally, comparisons on UT-Kinect and NTU-RGB+D datasets show that the accuracy of the proposed method is fully comparable to that of works based on 3D skeletons.

Open Education Resource for School Children with Down Syndrome

Action recognition in video sequences is a challenging problem of computer vision due to the similarity of visual contents, changes in the viewpoint for the same actions, camera motion with action performer, scale and pose of an actor, and different illumination conditions. Also, there is no designated action recognition model for hazy videos. This model proposes a novel unified and unique model for action recognition in haze built with Convolutional Neural Network (CNN) and deep bidirectional LSTM (DB-LSTM) network. First, every frame of the hazy video is feed into the AOD-Net (All-in-One Dehazing Network). Next, deep features are extracted from every sampled dehazed frame by using VGG-16, which helps reduce the redundancy and complexity. Later, the sequential and temporal information among frame features is learnt using DB-LSTM network, where multiple layers are stacked together in both the forward and backward passes of DB-LSTM to increase its depth. The proposed unified method is capable of learning long term sequences and can process lengthy videos (even hazy videos) in real time by analyzing features for a certain time interval. Experimental results on both synthesized and natural video datasets show decent results on par with other state of the art methods in action recognition using the proposed method on the benchmark data set UCF-101. This helps the Down Syndrome Students to recognize an action faster.

Classification of Action Based Video using Heterogeneous Feature Extraction and SVM

Action recognition (AR) plays a fundamental role in computer vision and video analysis. We are witnessing an astronomical increase of video data on the web and it is difficult to recognize the action in video due to different view point of camera. For AR in video sequence, it depends upon appearance in frame and optical flow in frames of video. In video spatial and temporal components of video frames features play integral role for better classification of action in videos. In the proposed system, RGB frames and optical flow frames are used for AR with the help of Convolutional Neural Network (CNN) pre-trained model Alex-Net extract features from fc7 layer. Support vector machine (SVM) classifier is used for the classification of AR in videos. For classification purpose, HMDB51 dataset have been used which includes 51 Classes of human action. The dataset is divided into 51 action categories. Using SVM classifier, extracted features are used for classification and achieved best result 95.6% accuracy as compared to other techniques of the state-of- art.

Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features

Recurrent neural network (RNN) and long short-term memory (LSTM) have achieved great success in processing sequential multimedia data and yielded the state-of-the-art results in speech recognition, digital signal processing, video processing, and text data analysis. In this paper, we propose a novel action recognition method by processing the video data using convolutional neural network (CNN) and deep bidirectional LSTM (DB-LSTM) network. First, deep features are extracted from every sixth frame of the videos, which helps reduce the redundancy and complexity. Next, the sequential information among frame features is learnt using DB-LSTM network, where multiple layers are stacked together in both forward pass and backward pass of DB-LSTM to increase its depth. The proposed method is capable of learning long term sequences and can process lengthy videos by analyzing features for a certain time interval. Experimental results show significant improvements in action recognition using the proposed method on three benchmark data sets including UCF-101, YouTube 11 Actions, and HMDB51 compared with the state-of-the-art action recognition methods.

Region-sequence based six-stream CNN features for general and fine-grained human action recognition in videos

This paper addresses the problems of both general and also fine-grained human action recognition in video sequences. Compared with general human actions, fine-grained action information is more difficult to detect and occupies relatively small-scale image regions. Our work seeks to improve fine-grained action discrimination, while also retaining the ability to perform general action recognition. Our method first estimates human pose and human parts positions in video sequences by extending our recent work on human pose tracking, and crops different scaled patches to obtain richer action information in a variety of different scales of appearance and motion cues. We then utilize a Convolutional Neural Network (CNN) to process each such image patch. Instead of using the output one dimension feature from the full-connection layer, we utilize the outputs of the pooling layer of CNN structure, which contains more spatial information. Then the high dimension of the pooling features is reduced by encoding, to generate the final human action descriptors for classification. Our method reduces feature dimension while also effectively combining appearance and motion information in a unified framework. We have carried out empirical experiments using two publicly available human action datasets, comparing the human action recognition result of our algorithm against six recent state-of-the-art methods from the literature. The results suggest comparatively strong performance of our method.

An efficient and sparse approach for large scale human action recognition in videos

This paper focuses on human action recognition in video sequences. A method based on optical flow estimation is presented, where critical points of this flow field are extracted. Multi-scale trajectories are generated from those points and are characterized in the frequency domain. Finally, a sequence is described by fusing this frequency information with motion orientation and shape information. This method has been tested on video datasets with recognition rates among the highest in the state of the art. Contrary to recent dense sampling strategies, the proposed method only requires critical points of motion flow field, thus permitting a lower computational cost and a better sequence description. A cross-dataset generalization is performed to illustrate the robustness of the method to recognition dataset biases. Results, comparisons and prospects on complex action recognition datasets are finally discussed.

3-D Projected PCA Based DMM Feature Fusing with SMO-SVM for Human Action Recognition

Action recognition in video sequence is a very important and challenging problem yet. This paper presents an efficient feature extraction method for human action recognition for depth video sequence. For the video sequence acquired by depth sensor, all 3-D projections (xy, yz and zx) are calculated for each depth frame. For each projection view, the difference between each alternative frames have been considered to form the Depth Motion Map (DMM). Principle Component Analysis technique is applied to decrease the facet of DMM-feature. Sequential minimal optimization (SMO) is pre-owned to train the Support Vector Machine (SVM). The proposed approach is evaluated on MSR Action-3D data set and compared with the existing approaches. The empirical results convey that proposed approach achieves good results than the existing approaches.

Fusing $${\mathcal {R}}$$ R Features and Local Features with Context-Aware Kernels for Action Recognition

The performance of action recognition in video sequences depends significantly on the representation of actions and the similarity measurement between the representations. In this paper, we combine two kinds of features extracted from the spatio-temporal interest points with context-aware kernels for action recognition. For the action representation, local cuboid features extracted around interest points are very popular using a Bag of Visual Words (BOVW) model. Such representations, however, ignore potentially valuable information about the global spatio-temporal distribution of interest points. We propose a new global feature to capture the detailed geometrical distribution of interest points. It is calculated by using the 3D $${\mathcal {R}}$$R transform which is defined as an extended 3D discrete Radon transform, followed by the application of a two-directional two-dimensional principal component analysis. For the similarity measurement, we model a video set as an optimized probabilistic hypergraph and propose a context-aware kernel to measure high order relationships among videos. The context-aware kernel is more robust to the noise and outliers in the data than the traditional context-free kernel which just considers the pairwise relationships between videos. The hyperedges of the hypergraph are constructed based on a learnt Mahalanobis distance metric. Any disturbing information from other classes is excluded from each hyperedge. Finally, a multiple kernel learning algorithm is designed by integrating the $$l_{2}$$l2 norm regularization into a linear SVM classifier to fuse the $${\mathcal {R}}$$R feature and the BOVW representation for action recognition. Experimental results on several datasets demonstrate the effectiveness of the proposed approach for action recognition.

On-line deep learning method for action recognition

In this paper an unsupervised on-line deep learning algorithm for action recognition in video sequences is proposed. Deep learning models capable of deriving spatio-temporal data have been proposed in the past with remarkable results, yet, they are mostly restricted to building features from a short window length. The model presented here, on the other hand, considers the entire sample sequence and extracts the description in a frame-by-frame manner. Each computational node of the proposed paradigm forms clusters and computes point representatives, respectively. Subsequently, a first-order transition matrix stores and continuously updates the successive transitions among the clusters. Both the spatial and temporal information are concurrently treated by the Viterbi Algorithm, which maximizes a criterion based upon (a) the temporal transitions and (b) the similarity of the respective input sequence with the cluster representatives. The derived Viterbi path is the node's output, whereas the concatenation of nine vicinal such paths constitute the input to the corresponding upper level node. The engagement of ART and the Viterbi Algorithm in a Deep learning architecture, here, for the first time, leads to a substantially different approach for action recognition. Compared with other deep learning methodologies, in most cases, it is shown to outperform them, in terms of classification accuracy.

ACTION RECOGNITION USING FEATURE TRANSFORM DESCRIPTOR FROM MINED DENSE SPATIO TEMPORAL

Action recognition in video sequence has been a major challenging research area for number of years. Apriori algorithm and SIFT descriptor based approach for action recognition is proposed in this paper. Here, two phases can be carried out for accurate and updating of action recognition. In the first phase, the input should be the video sequence. For preprocessing the sequences frame can be formatted by background modeling for every successive frame. After modeling the background, the corners are detected for every frame and compound features are extracted. Data mining is performed by using Apriori algorithm as well as with the help of compound features extraction the action can be segregated in this video sequence. In the second phase, the same process is performed as well as analysis for new input frame and new pattern are updated for perfect recognition process. Due to Apriori algorithm, the processing delay is reduced and accuracy is improved.