Approach For Action Recognition Research Articles

Multimodal Score Fusion with Sparse Low-rank Bilinear Pooling for Egocentric Hand Action Recognition

With the advent of egocentric cameras, there are new challenges where traditional computer vision is not sufficient to handle this kind of video. Moreover, egocentric cameras often offer multiple modalities that need to be modeled jointly to exploit complimentary information. In this article, we propose a sparse low-rank bilinear score pooling approach for egocentric hand action recognition from RGB-D videos. It consists of five blocks: a baseline CNN to encode RGB and depth information for producing classification probabilities; a novel bilinear score pooling block to generate a score matrix; a sparse low-rank matrix recovery block to reduce redundant features, which is common in bilinear pooling; a one-layer CNN for frame-level classification; and an RNN for video-level classification. We proposed to fuse classification probabilities instead of traditional CNN features from RGB and depth modality, involving an effective yet simple sparse low-rank bilinear score pooling to produce a fused RGB-D score matrix. To demonstrate the efficacy of our method, we perform extensive experiments over two large-scale hand action datasets, namely, THU-READ and FPHA, and two smaller datasets, GUN-71 and HAD. We observe that the proposed method outperforms state-of-the-art methods and achieves accuracies of 78.55% and 96.87% over the THU-READ dataset in cross-subject and cross-group settings, respectively. Further, we achieved accuracies of 91.59% and 43.87% over the FPHA and Gun-71 datasets, respectively.

From CNNs to Transformers in Multimodal Human Action Recognition: A Survey

Due to its widespread applications, human action recognition is one of the most widely studied research problems in Computer Vision. Recent studies have shown that addressing it using multimodal data leads to superior performance as compared to relying on a single data modality. During the adoption of deep learning for visual modelling in the last decade, action recognition approaches have mainly relied on Convolutional Neural Networks (CNNs). However, the recent rise of Transformers in visual modelling is now also causing a paradigm shift for the action recognition task. This survey captures this transition while focusing on Multimodal Human Action Recognition (MHAR). Unique to the induction of multimodal computational models is the process of ‘fusing’ the features of the individual data modalities. Hence, we specifically focus on the fusion design aspects of the MHAR approaches. We analyze the classic and emerging techniques in this regard, while also highlighting the popular trends in the adaption of CNN and Transformer building blocks for the overall problem. In particular, we emphasize on recent design choices that have led to more efficient MHAR models. Unlike existing reviews, which discuss Human Action Recognition from a broad perspective, this survey is specifically aimed at pushing the boundaries of MHAR research by identifying promising architectural and fusion design choices to train practicable models. We also provide an outlook of the multimodal datasets from their scale and evaluation viewpoint. Finally, building on the reviewed literature, we discuss the challenges and future avenues for MHAR.

Athletes' Action Recognition and Posture Estimation Algorithm Based on Image Processing Technology

A posture estimation algorithm based on image processing technology involves analyzing images or video frames to determine the position and orientation of human bodies or specific body parts. This process typically begins with detecting key points or landmarks on the body, such as joints or anatomical features. This paper presents a novel approach for athletes' action recognition and posture estimation using image processing technology, specifically focusing on the Contour Mapping Multivariant Recognition and Estimation (CMMRE) algorithm. The primary objective is to develop a robust system capable of accurately identifying human actions and estimating postures from visual data, with potential applications in sports analysis, surveillance, and human-computer interaction. The CMMRE algorithm employs advanced techniques for contour mapping, feature extraction, and prediction, leveraging deep learning methodologies to analyze and interpret visual information effectively. Through a series of experiments and simulations, the algorithm's performance is evaluated, showcasing its ability to achieve high accuracy in action recognition tasks across various scenarios. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement. The algorithm achieved an average accuracy of 95% in recognizing walking and running actions, and 80% in identifying jumping actions. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement.

A Multi-Modal Egocentric Activity Recognition Approach towards Video Domain Generalization.

Egocentric activity recognition is a prominent computer vision task that is based on the use of wearable cameras. Since egocentric videos are captured through the perspective of the person wearing the camera, her/his body motions severely complicate the video content, imposing several challenges. In this work we propose a novel approach for domain-generalized egocentric human activity recognition. Typical approaches use a large amount of training data, aiming to cover all possible variants of each action. Moreover, several recent approaches have attempted to handle discrepancies between domains with a variety of costly and mostly unsupervised domain adaptation methods. In our approach we show that through simple manipulation of available source domain data and with minor involvement from the target domain, we are able to produce robust models, able to adequately predict human activity in egocentric video sequences. To this end, we introduce a novel three-stream deep neural network architecture combining elements of vision transformers and residual neural networks which are trained using multi-modal data. We evaluate the proposed approach using a challenging, egocentric video dataset and demonstrate its superiority over recent, state-of-the-art research works.

Improved skeleton-based activity recognition using convolutional block attention module

Inferring human activities from the skeletons extracted from activity photos or videos is a fundamental yet important issue in the research community of computer vision. Current skeleton-based activity recognition methods have accomplished this task by extracting features in either spatial or temporal perspective, or combining both views; however, the extracted features lack further after-processing like feature augmentation. Aiming at the above shortcomings, this article presents a new human activity recognition approach based on human skeleton data, named Convolutional Block Attention Module-based Spatial Temporal Convolutional Network (CBAM-STGCN). To enhance the discriminative validity of the generated features, we leverage the attention mechanism to weight feature units within each dimension by introducing two attention modules designed in terms of different aspects into the model architecture. Specifically, the original feature map of an activity sample is adaptively refined by the convolutional block attention module, where two separate attention maps obtained with regard to spatial and channel dimensions are adopted to generate the corresponding attended feature maps. Evaluation results on the Kinetics benchmark dataset showed that the proposed model has improved 1.76 % and 2.43 % according to top-1 and top-5 metrics, respectively, compared to the conventional baseline model. Further ablation experiments proved that both channel and spatial attention modules contribute to the model performance.

Enhancing temporal action localization in an end-to-end network through estimation error incorporation

Temporal action localization presents a significant challenge in computer vision, as the development of an efficient method for this task remains elusive. The objective is to identify human activities within untrimmed videos, determining when and which actions occur in each video. While using trimmed videos could potentially resolve the localization problem and enhance classification accuracy, it is impractical for real-world applications as the trimming process itself requires human intervention. This highlights the importance of temporal localization. Due to the availability of several successful approaches for action recognition in trimmed video, conventional multi-stage methods for untrimmed video, commonly employ a network to generate activity proposals, followed by a separate network for classification. These disjoint networks are optimized individually and thus usually vary from the global optimum, leading to less precise candidate action proposals. To address this challenge, we propose a novel end-to-end neural network that utilizes error estimation for precise action localization and recognition in untrimmed videos. The proposed method performs the localization and classification of action instances simultaneously, thereby optimizing the corresponding networks concurrently. To increase the precision of the action proposal boundaries, the Regression module is innovatively utilized as part of the proposed end-to-end network, along with the Evaluation and Classification modules. This module estimates the potential error in proposal time boundaries and enhances the result accuracy. We have conducted experiments on THUMOS 14 and ActivityNet-1.3, which are considered the most challenging datasets for temporal action localization. The novel, yet fairly simple, proposed network achieves remarkable performance improvement compared to the other state-of-the-art methods. This improvement, which is more pronounced in the cases of high temporal intersection with ground truth, is accomplished without requiring extra data or complicated architecture. By incorporating error estimation, we achieved improvement in mean Average Precision (mAP). The proposed approach particularly shines for the localization of challenging activities in the complex and diverse dataset ActivityNet-1.3. For instance, for the “drinking coffee” activity, the mean Average Precision (mAP) was enhanced fivefold compared to the best-reported results.

STAR-3D: A Holistic Approach for Human Activity Recognition in the Classroom Environment

The video camera is essential for reliable activity monitoring, and a robust analysis helps in efficient interpretation. The systematic assessment of classroom activity through videos can help understand engagement levels from the perspective of both students and teachers. This practice can also help in robot-assistive classroom monitoring in the context of human–robot interaction. Therefore, we propose a novel algorithm for student–teacher activity recognition using 3D CNN (STAR-3D). The experiment is carried out using India’s indigenously developed supercomputer PARAM Shivay by the Centre for Development of Advanced Computing (C-DAC), Pune, India, under the National Supercomputing Mission (NSM), with a peak performance of 837 TeraFlops. The EduNet dataset (registered under the trademark of the DRSTATM dataset), a self-developed video dataset for classroom activities with 20 action classes, is used to train the model. Due to the unavailability of similar datasets containing both students’ and teachers’ actions, training, testing, and validation are only carried out on the EduNet dataset with 83.5% accuracy. To the best of our knowledge, this is the first attempt to develop an end-to-end algorithm that recognises both the students’ and teachers’ activities in the classroom environment, and it mainly focuses on school levels (K-12). In addition, a comparison with other approaches in the same domain shows our work’s novelty. This novel algorithm will also influence the researcher in exploring research on the “Convergence of High-Performance Computing and Artificial Intelligence”. We also present future research directions to integrate the STAR-3D algorithm with robots for classroom monitoring.

Perceiving Actions via Temporal Video Frame Pairs

Video action recognition aims to classify the action category in given videos. In general, semantic-relevant video frame pairs reflect significant action patterns such as object appearance variation and abstract temporal concepts like speed, rhythm, etc. However, existing action recognition approaches tend to holistically extract spatiotemporal features. Though effective, there is still a risk of neglecting the crucial action features occurring across frames with a long-term temporal span. Motivated by this, in this paper, we propose to perceive actions via frame pairs directly and devise a novel Nest Structure with frame pairs as basic units. Specifically, we decompose a video sequence into all possible frame pairs and hierarchically organize them according to temporal frequency and order, thus transforming the original video sequence into a Nest Structure. Through naturally decomposing actions, the proposed structure can flexibly adapt to diverse action variations such as speed or rhythm changes. Next, we devise a Temporal Pair Analysis module (TPA) to extract discriminative action patterns based on the proposed Nest Structure. The designed TPA module consists of a pair calculation part to calculate the pair features and a pair fusion part to hierarchically fuse the pair features for recognizing actions. The proposed TPA can be flexibly integrated into existing backbones, serving as a side branch to capture various action patterns from multi-level features. Extensive experiments show that the proposed TPA module can achieve consistent improvements over several typical backbones, reaching or updating CNN-based SOTA results on several challenging action recognition benchmarks.

A Self-Supervised Human Activity Recognition Approach via Body Sensor Networks in Smart City

In smart cities, pervasive sensing and wearable computing techniques are increasingly being employed to monitor and recognize human activities through body sensor networks, which have been widely used in urban safety, healthcare, and manufacturing. However, most researchers regard human activity recognition (HAR) as a high-cost research task requiring a large amount of labeled data, which is often unrealistic in real-world applications. To address this problem, we propose a self-supervised learning framework for HAR (SS-HAR). SS-HAR initially takes the unlabeled data generated by data augmentation as the input of the network, mines the supervised information of the unlabeled data under the effect of self-supervised learning, and uses the obtained backbone network as a feature extractor to extract activity features for subsequent classification. After that, we use part of the labeled data as the training set and extract the activity features using the backbone network for training and fitting the classifier. Then we utilize the rest of the data to verify the feasibility and effectiveness of the proposed self-supervised learning method. We have conducted multiple experiments on three publicly available datasets and one self-collected basketball activity dataset SZU_HAD_Basketball. The experimental results show that the SS-HAR method is able to achieve higher classification accuracy and stability than supervised and semi-supervised methods. Specifically, on the UCI dataset, SS-HAR achieves better classification performance compared to other approaches which improve the classification accuracy by 1% over the supervised method and by 5%-6% over the semi-supervised method respectively.

A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks

A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks

SpATr: MoCap 3D human action recognition based on spiral auto-encoder and transformer network

Recent technological advancements have significantly expanded the potential of human action recognition through harnessing the power of 3D data. This data provides a richer understanding of actions, including depth information that enables more accurate analysis of spatial and temporal characteristics. In this context, We study the challenge of 3D human action recognition. Unlike prior methods, that rely on sampling 2D depth images, skeleton points, or point clouds, often leading to substantial memory requirements and the ability to handle only short sequences, we introduce a novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network), specifically designed for fixed-topology mesh sequences. The SpATr model disentangles space and time in the mesh sequences. A lightweight auto-encoder, based on spiral convolutions, is employed to extract spatial geometrical features from each 3D mesh. These convolutions are lightweight and specifically designed for fix-topology mesh data. Subsequently, a temporal transformer, based on self-attention, captures the temporal context within the feature sequence. The self-attention mechanism enables long-range dependencies capturing and parallel processing, enabling scalability for long sequences. The proposed method is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub, from the Archive of Motion Capture As Surface Shapes (AMASS). Our results analysis demonstrates the competitive performance of our SpATr model in 3D human action recognition while maintaining efficient memory usage. The code and the training results are publicly available at https://github.com/h-bouzid/spatr.

A novel physical activity recognition approach using deep ensemble optimized transformers and reinforcement learning

In recent years, human physical activity recognition has increasingly attracted attention from different research fields such as healthcare, computer-human interaction, lifestyle monitoring, and athletics. Deep learning models have been extensively employed in developing physical activity recognition systems. To improve these models, their hyperparameters need to be initialized with optimal values. However, tuning these hyperparameters manually is time-consuming and may lead to inaccurate results. Moreover, the application of these models to different data resources and the integration of their results into the overall data processing pipeline are challenging issues in physical activity recognition systems. In this paper, we propose a novel ensemble method for physical activity recognition based on a deep transformer-based time-series classification model that uses heart rate, speed, and distance time-series data to recognize physical activities. In particular, we develop a modified arithmetic optimization algorithm to automatically adjust the optimal values of the classification models’ hyperparameters. Moreover, a reinforcement learning-based ensemble approach is proposed to optimally integrate the results of the classification models obtained using heart rate, speed, and distance time-series data and, subsequently, recognize the physical activities. Experiments performed on a real-world dataset demonstrated that the proposed method achieves promising efficiency in comparison to other state-of-the-art models. More specifically, the proposed method increases the performance compared to the second-best performer by around 3.44 %, 9.45 %, 5.43 %, 2.54 %, and 7.53 % based on accuracy, precision, recall, specificity, and F1-score evaluation metrics, respectively.

Hierarchical compositional representations for few-shot action recognition

Recently action recognition has received more and more attention for its comprehensive and practical applications in intelligent surveillance and human–computer interaction. However, few-shot action recognition has not been well explored and remains challenging because of data scarcity. In this paper, we propose a novel hierarchical compositional representations (HCR) learning approach for few-shot action recognition. Specifically, we divide a complicated action into several sub-actions by carefully designed hierarchical clustering and further decompose the sub-actions into more fine-grained spatially attentional sub-actions (SAS-actions). Although there exist large differences between base classes and novel classes, they can share similar patterns in sub-actions or SAS-actions. Furthermore, we adopt the Earth Mover’s Distance in the transportation problem to measure the similarity between video samples in terms of sub-action representations. It computes the optimal matching flows between sub-actions as distance metric, which is favorable for comparing fine-grained patterns. Extensive experiments show our method achieves the state-of-the-art results on HMDB51, UCF101 and Kinetics datasets.

A New Time-Series Classification Approach for Human Activity Recognition with Data Augmentation

A New Time-Series Classification Approach for Human Activity Recognition with Data Augmentation

MAR: Masked Autoencoders for Efficient Action Recognition

Standard approaches for video action recognition usually operate on full input videos, which is inefficient due to the widespread spatio-temporal redundancy in videos. The recent progress in masked video modelling, specifically VideoMAE, has shown the ability of vanilla Vision Transformers (ViT) to complement spatio-temporal contexts using limited visual content. Inspired by this, we propose Masked Action Recognition (MAR), which reduces redundant computation by discarding a proportion of patches and operating only on a portion of the videos. MAR includes two essential components: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cell running masking</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bridging classifier</i> . Specifically, to enable the ViT to perceive the details beyond the visible patches, cell running masking is used to preserve the spatio-temporal correlations in videos. This ensures that the patches at the same spatial location can be observed in turn for easy reconstructions. Additionally, we notice that, although the partially observed features can reconstruct semantically explicit invisible patches, they fail to achieve accurate classification. To address this issue, we propose a bridging classifier that can help fill the semantic gap between the ViT encoded features used for reconstruction and the specialized features used for classification. Our proposed MAR can reduce the computational cost of ViT by 53%. Extensive experiments have demonstrated that MAR consistently outperforms existing ViT models by a notable margin. Notably, we found that a ViT-Large model fine-tuned by MAR achieves comparable performance to a ViT-Huge model fine-tuned by standard training methods on both Kinetics-400 and Something-Something v2 datasets. Moreover, the computation overhead of our ViT-Large model is only 14.5% of that of the ViT-Huge model. Codes have been made available <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/alibaba-mmai-research/Masked-Action-Recognition</uri> .

Transductive Learning With Prior Knowledge for Generalized Zero-Shot Action Recognition

It is challenging to achieve generalized zero-shot action recognition. Different from the conventional zero-shot tasks which assume that the instances of the source classes are absent in the test set, the generalized zero-shot task studies the case that the test set contains both the source and the target classes. Due to the gap between visual feature and semantic embedding as well as the inherent bias of the learned classifier towards the source classes, the existing generalized zero-shot action recognition approaches are still far less effective than traditional zero-shot action recognition approaches. Facing these challenges, a novel transductive learning with prior knowledge (TLPK) model is proposed for generalized zero-shot action recognition. First, TLPK learns the prior knowledge which assists in bridging the gap between visual features and semantic embeddings, and preliminarily reduces the bias caused by the visual-semantic gap. Then, a transductive learning method that employs unlabeled target data is designed to overcome the bias problem in an effective manner. To achieve this, a target semantic-available approach and a target semantic-free approach are devised to utilize the target semantics in two different ways, where the target semantic-free approach exploits prior knowledge to produce well-performed semantic embeddings. By exploring the usage of the aforementioned prior-knowledge learning and transductive learning strategies, TLPK significantly bridges the visual-semantic gap and alleviates the bias between the source and the target classes. The experiments on the benchmark datasets of HMDB51 and UCF101 demonstrate the effectiveness of the proposed model compared to the state-of-the-art methods. The source code of this work can be found in https://mic.tongji.edu.cn.

A Novel Prediction Method of Transfer-Assisted Action Oriented to Individual Differences for the Excretion Care Robot.

The excretion care robot's (ECR) accurate recognition of transfer-assisted actions is crucial during its usage. However, transfer action recognition is a challenging task, especially since the differentiation of actions seriously affects its recognition speed, robustness, and generalization ability. We propose a novel approach for transfer action recognition assisted by a bidirectional long- and short-term memory (Bi-LSTM) network combined with a multi-head attention mechanism. Firstly, we utilize posture sensors to detect human movements and establish a lightweight three-dimensional (3D) model of the lower limbs. In particular, we adopt a discrete extended Kalman filter (DEKF) to improve the accuracy and foresight of pose solving. Then, we construct an action prediction model that incorporates a fused Bi-LSTM with Multi-head attention (MHA Bi-LSTM). The MHA extracts key information related to differentiated movements from different dimensions and assigns varying weights. Utilizing the Bi-LSTM network effectively combines past and future information to enhance the prediction results of differentiated actions. Finally, comparisons were made by three subjects in the proposed method and with two other time series based neural network models. The reliability of the MHA Bi-LSTM method was verified. These experimental results show that the introduced MHA Bi-LSTM model has a higher accuracy in predicting posture sensor-based excretory care actions. Our method provides a promising approach for handling transfer-assisted action individual differentiation in excretion care tasks.

Application of big data technology in traditional dance video movement recognition research

Abstract This paper proposes a multi-feature fusion approach for action recognition under big data technology with the goal of improving traditional dance video action recognition. By analyzing the basic method of dance action, the extraction process of dance action features is analyzed using both single-layer and hierarchical methods. Multi-feature fusion action recognition is chosen as the main method for action recognition. The image and audio features of the dance video are combined to improve the accuracy of recognizing dance actions. Use the optical flow algorithm to construct a histogram of the optical flow direction. The method’s feasibility is explored by applying the multi-feature fusion recognition method to traditional dance movement recognition. The results show that in traditional dance movement recognition performance, the performance of the method of multi-feature fusion recognition is improved by 7.6% compared to other traditional methods. The multi-feature fusion recognition method has more than 50% accuracy in recognizing different traditional dance movements and similar movements in terms of movement recognition accuracy. To a certain degree, this study enhances the efficiency of traditional dance movement recognition and conserves human and financial resources in dance movement recognition.

Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

In this article, a hierarchical method for action recognition based on temporal and spatial features is proposed. In current HAR methods, camera movement, sensor movement, sudden scene changes, and scene movement can increase motion feature errors and decrease accuracy. Another important aspect to take into account in a HAR method is the required computational cost. The proposed method provides a preprocessing step to address these challenges. As a preprocessing step, the method uses optical flow to detect camera movements and shots in input video image sequences. In the temporal processing block, the optical flow technique is combined with the absolute value of frame differences to obtain a time saliency map. The detection of shots, cancellation of camera movement, and the building of a time saliency map minimise movement detection errors. The time saliency map is then passed to the spatial processing block to segment the moving persons and/or objects in the scene. Because the search region for spatial processing is limited based on the temporal processing results, the computations in the spatial domain are drastically reduced. In the spatial processing block, the scene foreground is extracted in three steps: silhouette extraction, active contour segmentation, and colour segmentation. Key points are selected at the borders of the segmented foreground. The last used features are the intensity and angle of the optical flow of detected key points. Using key point features for action detection reduces the computational cost of the classification step and the required training time. Finally, the features are submitted to a Recurrent Neural Network (RNN) to recognise the involved action. The proposed method was tested using four well-known action datasets: KTH, Weizmann, HMDB51, and UCF101 datasets and its efficiency was evaluated. Since the proposed approach segments salient objects based on motion, edges, and colour features, it can be added as a preprocessing step to most current HAR systems to improve performance.

Computer vision-based approach for skeleton-based action recognition, SAHC

Computer vision-based approach for skeleton-based action recognition, SAHC