Approach For Action Recognition Research Articles

ABNet: AI-Empowered Abnormal Action Recognition Method for Laboratory Mouse Behavior

The automatic recognition and quantitative analysis of abnormal behavior in mice play a crucial role in behavioral observation experiments in neuroscience, pharmacology, and toxicology. Due to the challenging definition of abnormal behavior and difficulty in collecting training samples, directly applying behavior recognition methods to identify abnormal behavior is often infeasible. This paper proposes ABNet, an AI-empowered abnormal action recognition approach for mice. ABNet utilizes an enhanced Spatio-Temporal Graph Convolutional Network (ST-GCN) as an encoder; ST-GCN combines graph convolution and temporal convolution to efficiently capture and analyze spatio-temporal dynamic features in graph-structured data, making it suitable for complex tasks such as action recognition and traffic prediction. ABNet trains the encoding network with normal behavior samples, then employs unsupervised clustering to identify abnormal behavior in mice. Compared to the original ST-GCN network, the method significantly enhances the capabilities of feature extraction and encoding. We conduct comprehensive experiments on the Kinetics-Skeleton dataset and the mouse behavior dataset to evaluate and validate the performance of ABNet in behavior recognition and abnormal motion detection. In the behavior recognition experiments conducted on the Kinetics-Skeleton dataset, ABNet achieves an accuracy of 32.7% for the top one and 55.2% for the top five. Moreover, in the abnormal behavior analysis experiments conducted on the mouse behavior dataset, ABNet achieves an average accuracy of 83.1%.

A Dual Pipeline With Spatio-Temporal Attention Fusion Approach for Human Activity Recognition

A Dual Pipeline With Spatio-Temporal Attention Fusion Approach for Human Activity Recognition

From CNNs to Transformers in Multimodal Human Action Recognition: A Survey

Due to its widespread applications, human action recognition is one of the most widely studied research problems in Computer Vision. Recent studies have shown that addressing it using multimodal data leads to superior performance as compared to relying on a single data modality. During the adoption of deep learning for visual modelling in the past decade, action recognition approaches have mainly relied on Convolutional Neural Networks (CNNs). However, the recent rise of Transformers in visual modelling is now also causing a paradigm shift for the action recognition task. This survey captures this transition while focusing on Multimodal Human Action Recognition (MHAR). Unique to the induction of multimodal computational models is the process of ‘fusing’ the features of the individual data modalities. Hence, we specifically focus on the fusion design aspects of the MHAR approaches. We analyze the classic and emerging techniques in this regard, while also highlighting the popular trends in the adaption of CNN and Transformer building blocks for the overall problem. In particular, we emphasize on recent design choices that have led to more efficient MHAR models. Unlike existing reviews, which discuss Human Action Recognition from a broad perspective, this survey is specifically aimed at pushing the boundaries of MHAR research by identifying promising architectural and fusion design choices to train practicable models. We also provide an outlook of the multimodal datasets from their scale and evaluation viewpoint. Finally, building on the reviewed literature, we discuss the challenges and future avenues for MHAR.

Classroom teacher action recognition based on spatio-temporal dual-branch feature fusion

The classroom teaching action recognition task refers to recognizing and understanding teacher action through video temporal and spatial information. Due to complex backgrounds and significant occlusions, recognizing teacher action in the classroom environment poses substantial challenges. In this study, we propose a classroom teacher action recognition approach based on a spatio-temporal dual-branch feature fusion architecture, where the core task involves utilizing continuous human keypoint heatmap information and single-frame image information. Specifically, we fuse features from two modalities to propose a method combining image spatial information with temporal human keypoint heatmap information for teacher action recognition. Our approach ensures recognition accuracy while reducing the model’s parameters and computational complexity. Additionally, we constructed a teacher action dataset (CTA) in a real classroom environment, comprising 12 action categories, 13k+ video segments, and a total duration exceeding 15 h. The experimental results on the CTA dataset validate the effectiveness of our proposed method. Our research explores action recognition tasks in real complex classroom environments, providing a technical framework for classroom teaching intelligent analysis.

FTAN: Frame-to-frame temporal alignment network with contrastive learning for few-shot action recognition

Most current few-shot action recognition approaches follow the metric learning paradigm, measuring the distance of any sub-sequences (frames, any frame combinations or clips) between different actions for classification. However, this disordered distance metric between action sub-sequences ignores the long-term temporal relations of actions, which may result in significant metric deviations. What's more, the distance metric suffers from the distinctive temporal distribution of different actions, including intra-class temporal offsets and inter-class local similarity. In this paper, a novel few-shot action recognition framework, Frame-to-frame Temporal Alignment Network (FTAN), is proposed to address the above challenges. Specifically, an attention-based temporal alignment (ATA) module is devised to calculate the distance between corresponding frames of different actions along the temporal dimension to achieve frame-to-frame temporal alignment. Meanwhile, the Temporal Context module (TCM) is proposed to increase inter-class diversity by enriching the frame-level feature representation, and the Frames Cyclic Shift Module (FCSM) performs frame-level temporal cyclic shift to reduce intra-class inconsistency. In addition, we present temporal and global contrastive objectives to assist in learning discriminative and class-agnostic visual features. Experimental results show that the proposed architecture achieves state-of-the-art on HMDB51, UCF101, Something-Something V2 and Kinetics-100 datasets.

AtomGID: An Atomic Gesture Identifier for Qualitative Spatial Reasoning

In this paper, we present a novel non-deep-learning-based approach for real-time object tracking and activity recognition within smart homes, aiming to minimize human intervention and dataset requirements. Our method utilizes discreet, easily concealable sensors and passive RFID technology to track objects in real-time, enabling precise activity recognition without the need for extensive datasets typically associated with deep learning techniques. Central to our approach is AtomGID, an algorithm tailored to extract highly generalizable spatial features from RFID data. Notably, AtomGID’s adaptability extends beyond RFID to other imprecise tracking technologies like Bluetooth beacons and radars. We validate AtomGID through simulation and real-world RFID data collection within a functioning smart home environment. To enhance recognition accuracy, we employ a clustering adaptation of the flocking algorithm, leveraging previously published Activities of Daily Living (ADLs) data. Our classifier achieves a robust classification rate ranging from 85% to 93%, underscoring the efficacy of our approach in accurately identifying activities. By prioritizing non-deep-learning techniques and harnessing the strengths of passive RFID technology, our method offers a pragmatic and scalable solution for activity recognition in smart homes, significantly reducing dataset dependencies and human intervention requirements.

Perceiving Actions via Temporal Video Frame Pairs

Video action recognition aims at classifying the action category in given videos. In general, semantic-relevant video frame pairs reflect significant action patterns such as object appearance variation and abstract temporal concepts like speed, rhythm, and so on. However, existing action recognition approaches tend to holistically extract spatiotemporal features. Though effective, there is still a risk of neglecting the crucial action features occurring across frames with a long-term temporal span. Motivated by this, in this article, we propose to perceive actions via frame pairs directly and devise a novel Nest Structure with frame pairs as basic units. Specifically, we decompose a video sequence into all possible frame pairs and hierarchically organize them according to temporal frequency and order, thus transforming the original video sequence into a Nest Structure. Through naturally decomposing actions, the proposed structure can flexibly adapt to diverse action variations such as speed or rhythm changes. Next, we devise a Temporal Pair Analysis module (TPA) to extract discriminative action patterns based on the proposed Nest Structure. The designed TPA module consists of a pair calculation part to calculate the pair features and a pair fusion part to hierarchically fuse the pair features for recognizing actions. The proposed TPA can be flexibly integrated into existing backbones, serving as a side branch to capture various action patterns from multi-level features. Extensive experiments show that the proposed TPA module can achieve consistent improvements over several typical backbones, reaching or updating CNN-based SOTA results on several challenging action recognition benchmarks.

Multimodal Score Fusion with Sparse Low-rank Bilinear Pooling for Egocentric Hand Action Recognition

With the advent of egocentric cameras, there are new challenges where traditional computer vision is not sufficient to handle this kind of video. Moreover, egocentric cameras often offer multiple modalities that need to be modeled jointly to exploit complimentary information. In this article, we propose a sparse low-rank bilinear score pooling approach for egocentric hand action recognition from RGB-D videos. It consists of five blocks: a baseline CNN to encode RGB and depth information for producing classification probabilities; a novel bilinear score pooling block to generate a score matrix; a sparse low-rank matrix recovery block to reduce redundant features, which is common in bilinear pooling; a one-layer CNN for frame-level classification; and an RNN for video-level classification. We proposed to fuse classification probabilities instead of traditional CNN features from RGB and depth modality, involving an effective yet simple sparse low-rank bilinear score pooling to produce a fused RGB-D score matrix. To demonstrate the efficacy of our method, we perform extensive experiments over two large-scale hand action datasets, namely, THU-READ and FPHA, and two smaller datasets, GUN-71 and HAD. We observe that the proposed method outperforms state-of-the-art methods and achieves accuracies of 78.55% and 96.87% over the THU-READ dataset in cross-subject and cross-group settings, respectively. Further, we achieved accuracies of 91.59% and 43.87% over the FPHA and Gun-71 datasets, respectively.

Athletes' Action Recognition and Posture Estimation Algorithm Based on Image Processing Technology

A posture estimation algorithm based on image processing technology involves analyzing images or video frames to determine the position and orientation of human bodies or specific body parts. This process typically begins with detecting key points or landmarks on the body, such as joints or anatomical features. This paper presents a novel approach for athletes' action recognition and posture estimation using image processing technology, specifically focusing on the Contour Mapping Multivariant Recognition and Estimation (CMMRE) algorithm. The primary objective is to develop a robust system capable of accurately identifying human actions and estimating postures from visual data, with potential applications in sports analysis, surveillance, and human-computer interaction. The CMMRE algorithm employs advanced techniques for contour mapping, feature extraction, and prediction, leveraging deep learning methodologies to analyze and interpret visual information effectively. Through a series of experiments and simulations, the algorithm's performance is evaluated, showcasing its ability to achieve high accuracy in action recognition tasks across various scenarios. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement. The algorithm achieved an average accuracy of 95% in recognizing walking and running actions, and 80% in identifying jumping actions. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement.

A Multi-Modal Egocentric Activity Recognition Approach towards Video Domain Generalization.

Egocentric activity recognition is a prominent computer vision task that is based on the use of wearable cameras. Since egocentric videos are captured through the perspective of the person wearing the camera, her/his body motions severely complicate the video content, imposing several challenges. In this work we propose a novel approach for domain-generalized egocentric human activity recognition. Typical approaches use a large amount of training data, aiming to cover all possible variants of each action. Moreover, several recent approaches have attempted to handle discrepancies between domains with a variety of costly and mostly unsupervised domain adaptation methods. In our approach we show that through simple manipulation of available source domain data and with minor involvement from the target domain, we are able to produce robust models, able to adequately predict human activity in egocentric video sequences. To this end, we introduce a novel three-stream deep neural network architecture combining elements of vision transformers and residual neural networks which are trained using multi-modal data. We evaluate the proposed approach using a challenging, egocentric video dataset and demonstrate its superiority over recent, state-of-the-art research works.

Improved skeleton-based activity recognition using convolutional block attention module

Inferring human activities from the skeletons extracted from activity photos or videos is a fundamental yet important issue in the research community of computer vision. Current skeleton-based activity recognition methods have accomplished this task by extracting features in either spatial or temporal perspective, or combining both views; however, the extracted features lack further after-processing like feature augmentation. Aiming at the above shortcomings, this article presents a new human activity recognition approach based on human skeleton data, named Convolutional Block Attention Module-based Spatial Temporal Convolutional Network (CBAM-STGCN). To enhance the discriminative validity of the generated features, we leverage the attention mechanism to weight feature units within each dimension by introducing two attention modules designed in terms of different aspects into the model architecture. Specifically, the original feature map of an activity sample is adaptively refined by the convolutional block attention module, where two separate attention maps obtained with regard to spatial and channel dimensions are adopted to generate the corresponding attended feature maps. Evaluation results on the Kinetics benchmark dataset showed that the proposed model has improved 1.76 % and 2.43 % according to top-1 and top-5 metrics, respectively, compared to the conventional baseline model. Further ablation experiments proved that both channel and spatial attention modules contribute to the model performance.

Enhancing temporal action localization in an end-to-end network through estimation error incorporation

Temporal action localization presents a significant challenge in computer vision, as the development of an efficient method for this task remains elusive. The objective is to identify human activities within untrimmed videos, determining when and which actions occur in each video. While using trimmed videos could potentially resolve the localization problem and enhance classification accuracy, it is impractical for real-world applications as the trimming process itself requires human intervention. This highlights the importance of temporal localization. Due to the availability of several successful approaches for action recognition in trimmed video, conventional multi-stage methods for untrimmed video, commonly employ a network to generate activity proposals, followed by a separate network for classification. These disjoint networks are optimized individually and thus usually vary from the global optimum, leading to less precise candidate action proposals. To address this challenge, we propose a novel end-to-end neural network that utilizes error estimation for precise action localization and recognition in untrimmed videos. The proposed method performs the localization and classification of action instances simultaneously, thereby optimizing the corresponding networks concurrently. To increase the precision of the action proposal boundaries, the Regression module is innovatively utilized as part of the proposed end-to-end network, along with the Evaluation and Classification modules. This module estimates the potential error in proposal time boundaries and enhances the result accuracy. We have conducted experiments on THUMOS 14 and ActivityNet-1.3, which are considered the most challenging datasets for temporal action localization. The novel, yet fairly simple, proposed network achieves remarkable performance improvement compared to the other state-of-the-art methods. This improvement, which is more pronounced in the cases of high temporal intersection with ground truth, is accomplished without requiring extra data or complicated architecture. By incorporating error estimation, we achieved improvement in mean Average Precision (mAP). The proposed approach particularly shines for the localization of challenging activities in the complex and diverse dataset ActivityNet-1.3. For instance, for the “drinking coffee” activity, the mean Average Precision (mAP) was enhanced fivefold compared to the best-reported results.

STAR-3D: A Holistic Approach for Human Activity Recognition in the Classroom Environment

The video camera is essential for reliable activity monitoring, and a robust analysis helps in efficient interpretation. The systematic assessment of classroom activity through videos can help understand engagement levels from the perspective of both students and teachers. This practice can also help in robot-assistive classroom monitoring in the context of human–robot interaction. Therefore, we propose a novel algorithm for student–teacher activity recognition using 3D CNN (STAR-3D). The experiment is carried out using India’s indigenously developed supercomputer PARAM Shivay by the Centre for Development of Advanced Computing (C-DAC), Pune, India, under the National Supercomputing Mission (NSM), with a peak performance of 837 TeraFlops. The EduNet dataset (registered under the trademark of the DRSTATM dataset), a self-developed video dataset for classroom activities with 20 action classes, is used to train the model. Due to the unavailability of similar datasets containing both students’ and teachers’ actions, training, testing, and validation are only carried out on the EduNet dataset with 83.5% accuracy. To the best of our knowledge, this is the first attempt to develop an end-to-end algorithm that recognises both the students’ and teachers’ activities in the classroom environment, and it mainly focuses on school levels (K-12). In addition, a comparison with other approaches in the same domain shows our work’s novelty. This novel algorithm will also influence the researcher in exploring research on the “Convergence of High-Performance Computing and Artificial Intelligence”. We also present future research directions to integrate the STAR-3D algorithm with robots for classroom monitoring.

A Self-Supervised Human Activity Recognition Approach via Body Sensor Networks in Smart City

In smart cities, pervasive sensing and wearable computing techniques are increasingly being employed to monitor and recognize human activities through body sensor networks, which have been widely used in urban safety, healthcare, and manufacturing. However, most researchers regard human activity recognition (HAR) as a high-cost research task requiring a large amount of labeled data, which is often unrealistic in real-world applications. To address this problem, we propose a self-supervised learning framework for HAR (SS-HAR). SS-HAR initially takes the unlabeled data generated by data augmentation as the input of the network, mines the supervised information of the unlabeled data under the effect of self-supervised learning, and uses the obtained backbone network as a feature extractor to extract activity features for subsequent classification. After that, we use part of the labeled data as the training set and extract the activity features using the backbone network for training and fitting the classifier. Then we utilize the rest of the data to verify the feasibility and effectiveness of the proposed self-supervised learning method. We have conducted multiple experiments on three publicly available datasets and one self-collected basketball activity dataset SZU_HAD_Basketball. The experimental results show that the SS-HAR method is able to achieve higher classification accuracy and stability than supervised and semi-supervised methods. Specifically, on the UCI dataset, SS-HAR achieves better classification performance compared to other approaches which improve the classification accuracy by 1% over the supervised method and by 5%-6% over the semi-supervised method respectively.

A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks

A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks

SpATr: MoCap 3D human action recognition based on spiral auto-encoder and transformer network

Recent technological advancements have significantly expanded the potential of human action recognition through harnessing the power of 3D data. This data provides a richer understanding of actions, including depth information that enables more accurate analysis of spatial and temporal characteristics. In this context, We study the challenge of 3D human action recognition. Unlike prior methods, that rely on sampling 2D depth images, skeleton points, or point clouds, often leading to substantial memory requirements and the ability to handle only short sequences, we introduce a novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network), specifically designed for fixed-topology mesh sequences. The SpATr model disentangles space and time in the mesh sequences. A lightweight auto-encoder, based on spiral convolutions, is employed to extract spatial geometrical features from each 3D mesh. These convolutions are lightweight and specifically designed for fix-topology mesh data. Subsequently, a temporal transformer, based on self-attention, captures the temporal context within the feature sequence. The self-attention mechanism enables long-range dependencies capturing and parallel processing, enabling scalability for long sequences. The proposed method is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub, from the Archive of Motion Capture As Surface Shapes (AMASS). Our results analysis demonstrates the competitive performance of our SpATr model in 3D human action recognition while maintaining efficient memory usage. The code and the training results are publicly available at https://github.com/h-bouzid/spatr.

A novel physical activity recognition approach using deep ensemble optimized transformers and reinforcement learning

In recent years, human physical activity recognition has increasingly attracted attention from different research fields such as healthcare, computer-human interaction, lifestyle monitoring, and athletics. Deep learning models have been extensively employed in developing physical activity recognition systems. To improve these models, their hyperparameters need to be initialized with optimal values. However, tuning these hyperparameters manually is time-consuming and may lead to inaccurate results. Moreover, the application of these models to different data resources and the integration of their results into the overall data processing pipeline are challenging issues in physical activity recognition systems. In this paper, we propose a novel ensemble method for physical activity recognition based on a deep transformer-based time-series classification model that uses heart rate, speed, and distance time-series data to recognize physical activities. In particular, we develop a modified arithmetic optimization algorithm to automatically adjust the optimal values of the classification models’ hyperparameters. Moreover, a reinforcement learning-based ensemble approach is proposed to optimally integrate the results of the classification models obtained using heart rate, speed, and distance time-series data and, subsequently, recognize the physical activities. Experiments performed on a real-world dataset demonstrated that the proposed method achieves promising efficiency in comparison to other state-of-the-art models. More specifically, the proposed method increases the performance compared to the second-best performer by around 3.44 %, 9.45 %, 5.43 %, 2.54 %, and 7.53 % based on accuracy, precision, recall, specificity, and F1-score evaluation metrics, respectively.

Hierarchical compositional representations for few-shot action recognition

Recently action recognition has received more and more attention for its comprehensive and practical applications in intelligent surveillance and human–computer interaction. However, few-shot action recognition has not been well explored and remains challenging because of data scarcity. In this paper, we propose a novel hierarchical compositional representations (HCR) learning approach for few-shot action recognition. Specifically, we divide a complicated action into several sub-actions by carefully designed hierarchical clustering and further decompose the sub-actions into more fine-grained spatially attentional sub-actions (SAS-actions). Although there exist large differences between base classes and novel classes, they can share similar patterns in sub-actions or SAS-actions. Furthermore, we adopt the Earth Mover’s Distance in the transportation problem to measure the similarity between video samples in terms of sub-action representations. It computes the optimal matching flows between sub-actions as distance metric, which is favorable for comparing fine-grained patterns. Extensive experiments show our method achieves the state-of-the-art results on HMDB51, UCF101 and Kinetics datasets.

A New Time-Series Classification Approach for Human Activity Recognition with Data Augmentation

A New Time-Series Classification Approach for Human Activity Recognition with Data Augmentation

SAT: A Selective Adversarial Training Approach for WiFi-based Human Activity Recognition

SAT: A Selective Adversarial Training Approach for WiFi-based Human Activity Recognition