Field Of Activity Recognition Research Articles

WiCAR: A class-incremental system for WiFi activity recognition

WiCAR: A class-incremental system for WiFi activity recognition

EPK-CLIP: External and Priori Knowledge CLIP for action recognition

Contrastive Language-Image Pretraining (CLIP) models have achieved significant success and have markedly improved the performance of various downstream tasks, including action recognition. However, how to effectively introduce knowledge into the field of action recognition remains an open question. In this work, an External and Priori Knowledge CLIP (EPK-CLIP) is proposed to introduce external knowledge into the model. To capture external knowledge, an external knowledge embedding module is proposed, which can generate and utilize human-object interaction relations as external knowledge, enabling the model to learn better features. Furthermore, the sparse regularization is introduced in the loss function, endowing the model with the ability to exploit the sparse priori knowledge inherent in the classification task. Finally, multiple inference module is proposed to obtain classification results from both direct and indirect perspectives. Specifically, the final classification result is obtained by fusing the outputs of different reasoning modules. Moreover, four external knowledge datasets: Kinetics-400-VC, Jester-VC, HMDB-51-VC, and UCF-101-VC are built and released for public usage, which is a multimodal extension of corresponding action datasets respectively. Under fully-supervised settings, our model achieves the top-1 accuracy of 84.3%, 97.1%, 82.9%, and 98.2% on Kinetics-400, Jester, HMDB-51, and UCF-101, respectively. In zero-shot experiments, our model also achieves state-of-the-art results, with top-1 accuracy of 51.6% and 77.7% on HMDB-51 and UCF-101, respectively. All related datasets and code can be found at https://github.com/geek12138/EPK-CLIP.

Spatio‐temporal attention modules in orientation‐magnitude‐response guided multi‐stream CNNs for human action recognition

AbstractThis paper introduces a new descriptor called orientation‐magnitude response maps as a single 2D image to effectively explore motion patterns. Moreover, boosted multi‐stream CNN‐based model with various attention modules is designed for human action recognition. The model incorporates a convolutional self‐attention autoencoder to represent compressed and high‐level motion features. Sequential convolutional self‐attention modules are used to exploit the implicit relationships within motion patterns. Furthermore, 2D discrete wavelet transform is employed to decompose RGB frames into discriminative coefficients, providing supplementary spatial information related to the actors actions. A spatial attention block, implemented through the weighted inception module in a CNN‐based structure, is designed to weigh the multi‐scale neighbours of various image patches. Moreover, local and global body pose features are combined by extracting informative joints based on geometry features and joint trajectories in 3D space. To provide the importance of specific channels in pose descriptors, a multi‐scale channel attention module is proposed. For each data modality, a boosted CNN‐based model is designed, and the action predictions from different streams are seamlessly integrated. The effectiveness of the proposed model is evaluated across multiple datasets, including HMDB51, UTD‐MHAD, and MSR‐daily activity, showcasing its potential in the field of action recognition.

Spatial-temporal graph neural ODE networks for skeleton-based action recognition.

In the field of skeleton-based action recognition, accurately recognizing human actions is crucial for applications such as virtual reality and motion analysis. However, this task faces challenges such intraindividual action differences and long-term temporal dependencies. To address these challenges, we propose an innovative model called spatial-temporal graph neural ordinary differential equations (STG-NODE). First, in the data preprocessing stage, the dynamic time warping (DTW) algorithm is used to normalize and calculate 3D skeleton data to facilitate the derivation of customized adjacency matrices for improving the influence of intraindividual action differences. Secondly, a custom ordinary differential equation (ODE) integrator is applied based on the initial conditions of the temporal features, producing a solution function that simulates the dynamic evolution trend of the events of interest. Finally, the outstanding ODE solver is used to numerically solve the time features based on the solution function to increase the influence of long-term dependencies on the recognition accuracy of the model and provide it with a more powerful temporal modeling ability. Through extensive experiments conducted on the NTU RGB+D 60 and Kinetics Skeleton 400 benchmark datasets, we demonstrate the superior performance of STG-NODE in the action recognition domain. The success of the STG-NODE model also provides new ideas and methods for the future development of the action recognition field.

An optimization method of human skeleton keyframes selection for action recognition

In the action recognition field based on the characteristics of human skeleton joint points, the selection of keyframes in the skeleton sequence is a significant issue, which directly affects the action recognition accuracy. In order to improve the effectiveness of keyframes selection, this paper proposes inflection point frames, and transforms keyframes selection into a multi-objective optimization problem based on it. First, the pose features are extracted from the input skeleton joint point data, which used to construct the pose feature vector of each frame in time sequence; then, the inflection point frames in the sequence are determined according to the flow of momentum of each body part. Next, the pose feature vectors are input into the keyframes multi-objective optimization model, with the fusion of domain information and the number of keyframes; finally, the output keyframes are input to the action classifier. To verify the effectiveness of the method, the MSR-Action3D, the UTKinect-Action and Florence3D-Action, and the 3 public datasets, are chosen for simulation experiments and the results show that the keyframes sequence obtained by this method can significantly improve the accuracy of multiple action classifiers, and the average recognition accuracy of the three data sets can reach 94.6%, 97.6% and 94.2% respectively. Besides, combining the optimized keyframes with deep learning classifier on the NTU RGB + D dataset can make the accuracies reaching 83.2% and 93.7%.

The identification of minor impact collisions in a long video for detecting property damages caused by fleeing vehicles using three-dimensional convolutional neural network

Abstract A parked vehicle damaged by a hit-and-run can only be repaired at the expense of the owner, unless the fleeing vehicle is identified and the driver apprehended. Identifying the fleeing vehicle involves using a video investigation method that searches for perpetrators through CCTV footage of the crime scene. When the length of the recorded video is long, the investigation may require an extended amount of time from the investigator, resulting in an added burden on their daily work. Some commercial companies are using object recognition and tracking technology to detect hit-and-run incidents; however, detecting small movements of a vehicle during a minor collision still remains a challenge. Therefore, there is a need for a system that can detect small movement in a vehicle in a lengthy video. Automatic recognition and tracking require a sufficient amount of training dataset. However, such a dataset for hit-and-run incidents is not publicly available. One of the reasons behind this scarcity is that it may violate personal information protection acts. On the other hand, instead of using real accident videos, we could use actors to simulate such accident scenes. Although this may be feasible, creating such a dataset would require substantial costs. In this paper, we describe a new dataset for hit-and-run incidents. We collected 833 hit-and-run videos by recreating a parking lot using miniaturized cars. This dataset has been made publicly available through Kaggle. We used three-dimensional convolution neural network, which is frequently used in the field of action recognition, to detect small movements of vehicles during hit-and-run incidents. In addition, the proportion of the area that surrounds the target vehicle to the min-max box of the vehicle itself and the length of the input frame are varied to compare the accuracy. As a result, we were able to achieve better accuracy by using the lowest proportion and the shortest input frame.

Prior Knowledge-guided Hierarchical Action Quality Assessment with 3D Convolution and Attention Mechanism

Recently, there has been a growing interest in the field of computer vision and deep learning regarding a newly emerging problem known as action quality assessment (AQA). However, most researchers still rely on the traditional approach of using models from the video action recognition field. Unfortunately, this approach overlooks crucial features in AQA, such as movement fluency and degree of completion. Alternatively, some researchers have employed the transformer paradigm to capture action details and overall action integrity, but the high computational cost associated with transformers makes them impractical for real-time tasks. Due to the diversity of action types, it is challenging to rely solely on a shared model for quality assessment of various types of actions. To address these issues, we propose a novel network structure for AQA, which is the first to integrate multi-model capabilities through a classification model. Specifically, we utilize a pre-trained I3D model equipped with a self-attention block for classification. This allows us to evaluate various categories of actions using just one model. Furthermore, we introduce self-attention mechanisms and multi-head attention into the traditional convolutional neural network. By systematically replacing the last few layers of the conventional convolutional network, our model gains a greater ability to sense the global coordination of different actions. We have verified the effectiveness of our approach on the AQA-7 dataset. In comparison to other popular models, our model achieves satisfactory performance while maintaining a low computational cost.

Skeleton-based action recognition based on multidimensional adaptive dynamic temporal graph convolutional network

Due to the superior capability to process the topology of graphs, graph convolutional networks are gaining popularity in the field of action recognition based on skeleton data. However, it remains difficult to effectively extract features with more distinguishing information for both spatial and temporal dimension. A novel multidimensional adaptive dynamic temporal graph convolutional network (MADT-GCN) model for skeleton-based action recognition is proposed in this work. It consists of two modules, one multidimensional adaptive graph convolutional network (MD-AGCN) module and one dynamic temporal convolutional network (DY-TCN) module. Firstly, MD-AGCN has the ability to adaptively change the graph topology in accordance with varieties of the layers and multidimensional information of spatial, temporal, and channel dimensions that are contained in various action samples to capture the complex connections of each couple of joints. Then, DY-TCN is proposed in order to boost the representation capability to capture expressive temporal features. Moreover, the information of both the joints and bones, together with their motion information, are simultaneously modeled in a multi-stream framework, which shows notable improvements in recognition accuracy. Finally, extensive experiments are conducted on two standard datasets, NTU-RGB+D and NTU-RGB+D 120. The experimental results demonstrate the effectiveness of the proposed method.

An Overview of the Vision-Based Human Action Recognition Field

Artificial intelligence’s rapid advancement has enabled various applications, including intelligent video surveillance systems, assisted living, and human–computer interaction. These applications often require one core task: video-based human action recognition. Research in human video-based human action recognition is vast and ongoing, making it difficult to assess the full scope of available methods and current trends. This survey concisely explores the vision-based human action recognition field and defines core concepts, including definitions and explanations of the common challenges and most used datasets. Additionally, we provide in an easy-to-understand manner the literature approaches and their evolution over time, emphasizing intuitive notions. Finally, we explore current research directions and potential future paths. The core goal of this work is to provide future works with a shared understanding of fundamental ideas and clear intuitions about current works and find new research opportunities.

A survey of Few-Shot Action Recognition

In recent years, with the development of network technology, countless videos are produced every day. Many achievements have also been made in the field of action recognition in computer vision. Training action recognition models requires a large number of labeled samples, but in reality, the amount of data is scarce, and it is extremely difficult to obtain a large amount of data due to costs and other reasons. The few-shot learning aims to solve the problem of using several samples to learn new categories. This paper combs the relevant research in recent years of few-shot action recognition technology. According to the classification of training process, this paper summarizes the research progress and typical models of few-shot action recognition from the perspectives of data processing, feature embedding, feature augmentation, and metric learning; finally points out the challenges faced by current research and the future development directions.

Campus Abnormal Behavior Recognition With Temporal Segment Transformers

The intelligent campus surveillance system is beneficial to improve safety in school. Abnormal behavior recognition, a field of action recognition in computer vision, plays an essential role in intelligent surveillance systems. Computer vision has been actively applied to action recognition systems based on Convolutional Neural Networks (CNNs). However, capturing sufficient motion sequence features from videos remains a significant challenge in action recognition. This work explores the challenges of video-based abnormal behavior recognition on campus. In addition, a novel framework is established on long-range temporal video structure modeling and a global sparse uniform sampling strategy that divides a video into three segments of identical durations and uniformly samples each snippet. The proposed method incorporates a consensus of three temporal segment transformers (TST) that globally connects patches and computes self-attention with joint spatiotemporal factorization. The proposed model is developed on the newly created campus abnormal behavior recognition (CABR50) dataset, which contains 50 human abnormal action classes with an average of over 700 clips per class. Experiments show that it is feasible to implement abnormal behavior recognition on campus and that the proposed method is competitive with other peer video recognition in terms of Top-1 and Top-5 recognition accuracy. The results suggest that TST-L+ can improve campus abnormal behavior recognition, corresponding to Top-1 and Top-5 accuracy results of 83.57% and 97.16%, respectively.

Traditional African Dances Preservation Using Deep Learning Techniques

Human action recognition continues to evolve and improve through deep learning techniques. There have been studies with some success in the field of action recognition, but only a few of them have focused on traditional dance. This is because dance actions, especially in traditional African dance, are long and involve fast movements. This research proposes a novel framework that applies data science algorithms to the field of cultural preservation by applying various deep learning techniques to identify, classify, and model traditional African dances from videos. Traditional dances are an important part of African culture and heritage. Digital preservation of these dances in their multitude and form is a challenging problem. The dance dataset was constituted from freely available YouTube videos. Four traditional African dances were used for the dance classification process: Adowa, Swange, Bata, and Sinte dance. Five Convolutional Neural Network (CNN) models were used for the classification and achieved an accuracy between 93% and 98%. Additionally, human pose estimation algorithms were applied to Sinte dance. A model of Sinte dance that can be exported to other environments was obtained.

MIAM: Motion information aggregation module for action recognition

In the field of action recognition based on RGB videos, it is infeasible to train deep networks on dozens or hundreds of frames because of limits on computational complexity and memory. Previous works commonly adopted a sparse sampling strategy, which unfortunately leads to missing crucial frames and insufficient modelling for short-range motions. In this letter, an effective motion information aggregation module (MIAM) that utilises convolutional neural networks to aggregate motion information from multiple frames into one single frame is proposed. This allows to train deep networks end-to-end with densely sampled frames efficiently. The MIAM enables the model to gather motion information at every instant of action, which avoids missing subtle movements. Experiments on the NTU RGB+D 60 and NTU RGB+D 120 datasets verify that the MIAM significantly improves the recognition accuracy with very limited extra computational cost and exhibits unique advantages for recognising subtle actions.

Footprint Extraction and Sports Training Action Recognition Based on Wireless Network Communication

The combination of scientific and technological achievements and sports has found new opportunities to change people’s sports habits. Sport training takes up an increasing proportion of people’s lives. In order to improve the efficiency of sports training and standardize the training actions of players, this article is based on wireless network communication and uses different types of recognition methods in the field of action recognition to build basic classifications. Iterative mutual training to improve generalization performance can reduce the cost of labeling and realize the complementary advantages of different recognition methods, thereby improving the recognition accuracy of human actions. Finally, the algorithm is used to recognize human movements. This method can effectively overcome the problem of differential degradation of base classifiers in the iterative process of collaborative training and further improve the accuracy of human action recognition. The experimental results prove that the motion recognition of wireless network communication proposed in this paper can effectively improve the accuracy of athletes’ movements, which is more than 20% higher than traditional methods, and, under the guidance of standardized movements, can reduce athletes’ sports injuries.

Digital Twins by Physical Education Teaching Practice in Visual Sensing Training System

The paper expects to improve the efficiency and intelligence of somatosensory recognition technology in the application of physical education teaching practice. Firstly, the combination of induction recognition technology and the Internet is used. Secondly, through the Kinect sensor, bone data are acquired. Finally, the hidden Markov model (HMM) is used to simulate the experimental data. On the simulation results, a gait recognition algorithm is proposed. The gait recognition algorithm is used to identify the motion behaviour, and the results are displayed in the Web (World Wide Web) end built by the cloud server. Meantime, in view of the existing problems in the practice of physical education, combined with the establishment and operation of the Digital Twins (DTs) system, the camera source recognition architecture is carried out since the twin network and the two network branches share weights. This paper analyses these problems since the application of somatosensory recognition technology and puts forward the improvement methods. For the single problem of equipment in physical education, this paper puts forward the monitoring and identification function of the cloud server. It is to transmit data through Hypertext Transfer Protocol (HTTP) and locate and collect data through a monitoring terminal. For the lack of comprehensiveness and balance of sports plans, this paper proposes a scientific training plan and process customization based on Body Mass Index (BMI), analyses real-time data in the cloud, and makes scientific customization plans according to different students’ physical conditions. Moreover, 25 participants are invited to carry out the exercise detection and analysis experiment, and the joint monitoring of their daily movements is tested. This process has completed the design of a feasible and accurate platform for information collection and processing, which is convenient for managers and educators to comprehensively and scientifically master and manage the physical level and training of college students. The proposed method improves the recognition rate of the camera source to some extent and has important exploration significance in the field of action recognition.

Deep Learning for Ballroom Dance Recognition: A Temporal and Trajectory-Aware Classification Model With Three-Dimensional Pose Estimation and Wearable Sensing

Dance performance recognition methods have been investigated and shown various applications such as picture-pose evaluation and synchronizing foot timing and direction. However, detailed analysis and feedback are still missing. To provide them, understanding the performance by component level is necessary. Specifically, we formulate it as a dance-figure classification problem using three-dimensional body joints and wearable sensors. Our model is based on long short-term memory (LSTM) and includes the temporal and trajectory-wise structure that uses the trajectory information in a timestep and the temporal masking module. As a result, we achieved 93% accuracy with our proposed method, which is highly overwhelming the baseline result (84.7%) and very close to the accuracy of the experienced dancers (93.6%). We have made the dataset of ballroom dance performance dataset open to researchers to develop the activity recognition field further.

Automatic Visual Attention Detection for Mobile Eye Tracking Using Pre-Trained Computer Vision Models and Human Gaze.

Processing visual stimuli in a scene is essential for the human brain to make situation-aware decisions. These stimuli, which are prevalent subjects of diagnostic eye tracking studies, are commonly encoded as rectangular areas of interest (AOIs) per frame. Because it is a tedious manual annotation task, the automatic detection and annotation of visual attention to AOIs can accelerate and objectify eye tracking research, in particular for mobile eye tracking with egocentric video feeds. In this work, we implement two methods to automatically detect visual attention to AOIs using pre-trained deep learning models for image classification and object detection. Furthermore, we develop an evaluation framework based on the VISUS dataset and well-known performance metrics from the field of activity recognition. We systematically evaluate our methods within this framework, discuss potentials and limitations, and propose ways to improve the performance of future automatic visual attention detection methods.

Two Stream Convolution Fusion Network based on Attention Mechanism

Due to the intuitiveness and low cost of image data, action recognition based on image data has always been the focus of action recognition research. The most of studies show that spatio-temporal two-stream network has very superior performance in the field of action recognition. However, traditional two-stream convolutional model needs multiple training, and the method of fusion two-stream models is too simple. In this research, a new two stream convolution fusion model is proposed, which can train our model end-to-end. At the same time, the basic model of spatio-temporal flow model is replaced by CBAM_Resnet. The model is improved on the basis of resnet_101, which gives weight to channel and spatial features, so as to improve the performance of the model. In this paper, the RGB image and the optical flow in X and Y directions are taken as the input features of the fusion network, and the final classification result of the model is obtained. Our method is validated on HMDB51 datasets, and the experimental results show that the accuracy of our model is improved by 4%.

Symmetrical Enhanced Fusion Network for Skeleton-Based Action Recognition

A novel method for skeleton-based action recognition by fusing multi-level spatial features and multi-level temporal features is proposed in this article. Recently, Graph Convolutional Network (GCN) for skeleton-based action recognition has attracted the eyes of many researchers and has a great performance in the field of action recognition. But most of them focus on changing architecture of single-stream network and only use simple methods like average fusion to fuse different forms of skeleton data. In this article, we shift the focus to the problem that insufficient interactions between the different forms of features for that networks are unable to fully capture efficient information from skeleton data. To tackle this problem, we propose a multi-stream network called Symmetrical Enhanced Fusion Network (SEFN). The network is composed of a spatial stream, a temporal stream and a fusion stream. The spatial stream extracts spatial features from skeleton data by GCN. The temporal stream is able to extract temporal features from skeleton data with the help of the embedded Motion Sequence Calculation Algorithm. The fusion stream provides an early fusion method and extra fusion information for the whole network. It gathers multi-level features from two feature extractions and fuses them with the Multi-perspective Attention Fusion Module (MPAFM) we propose. The MPAFM enables different forms of data to enhance each other and can strengthen feature extractions. In the final, we generalize the skeleton data from joint data to bone data and evaluate our network in three large-scale benchmarks: NTU-RGBD, NTU-RGBD 120 and Kinetics-Skeleton. Experiment results demonstrate that our method achieves competitive performance.

Activity Recognition and Creation of Web Service for Activity Recognition using Mobile Sensor Data using Azure Machine Learning Studio

With the increasing pervasive computation, “Activity Recognition” has become a vast and popular field of research. In the field of automated Activity Recognition, we use multiple sensors in wearable/portable devices in order to recognize the human activities such as standing still, sitting, relaxing, laying, walking, climbing stairs, knee bending cycling jogging etc. The main purpose of this paper is to discuss the field of Activity Recognition for patients and old- age persons or any person in general. This research paper can also be used for telemedicine purposes. Besides, different machine learning algorithm will be applied to achieve Activity Recognition rather precisely. Microsoft Azure ML Studio and a bench marking data set are used for creation as well as evaluation of Machine Learning Model. In addition, a Web Service for Activity Recognition is also developed by using Microsoft Azure ML Studio in order to help the developer and researcher while working on Activity Recognition.