Activity Recognition Model Research Articles

Power Grid Violation Action Recognition via Few-Shot Adaptive Network

To address the performance degradation of violation action recognition models due to changing operational scenes in power grid operations, this paper proposes a Few-shot Adaptive Network (FSA-Net). The method incorporates few-shot learning into the network design by adding a parameter mapping layer to the classification network and developing a task-adaptive module to adjust the network parameters for changing scenes. A task-specific linear classifier is added after the backbone, allowing the adaptive generation of classifier weights based on the changing task scene to enhance the model’s generalizability. Additionally, the model uses a strategy of freezing the backbone network and iteratively updating only certain module parameters during training in order to minimize training costs. This approach addresses the challenge of iteratively updating difficulties in the original model, which are caused by limited image data following scene changes. In this paper, 2000 samples under power grid scenarios are used as the experimental dataset; the average recognition accuracy for violation actions is 81.77% for images after scene changes, which represents a 4.58% improvement when compared to the ResNet-50 classification network. Furthermore, the model’s training efficiency is enhanced by 40%. The experimental results show that the method enhances the performance of the violation action recognition model before and after scene changes and improves the efficiency of the iterative model by updating with a smaller sample size, lower model design cost, and lower training cost.

Research on the Education and Teaching of Physical Education Dance Courses in Colleges and Universities Based on Deep Learning

Teaching reform is usually a unified adjustment based on the current teaching objectives, lacking pertinence. In order to improve the teaching quality and students’ learning quality of sports dance course in colleges and universities, according to the teaching quality evaluation data, this paper proposes the teaching method of sports dance course in colleges and universities based on in-depth learning. According to the sports dance action image after the visual error correction, the OpenPose human posture estimation algorithm is used to implement the dancer’s motion posture estimation, obtain the sequence data of the dancer’s skeleton joint points, and use the depth convolution neural network optimized by particle swarm optimization algorithm to build the sports dance action recognition model, with the obtained sequence data of the dancer’s skeleton joint points as the effective input. The result of sports dance movement recognition is obtained through training, which is used as the basis for students to correct their dance movements and improve their professional skills. According to the teaching process organized by teachers and the goal of cultivating students’ professional skills, a sports dance teaching evaluation index system is constructed, and the deep convolution neural network optimized by particle swarm optimization algorithm is used to build a sports dance teaching evaluation model. With the quantified data of the secondary indicators in the teaching evaluation index system as the model input, the sports dance teaching evaluation results are obtained through training. The findings will serve as a dependable foundation for educators to enhance the quality of instruction and assist students in enhancing their proficiency in sports dance. The experimental outcomes demonstrate that this approach is adept at identifying sports dance movements effectively and precisely evaluating sports dance classes, thereby establishing a solid footing for substantially boosting the instructional impact of sports dance courses in higher education institutions.

Safety After Dark: A Privacy Compliant and Real-Time Edge Computing Intelligent Video Analytics for Safer Public Transportation.

Public transportation systems play a vital role in modern cities, but they face growing security challenges, particularly related to incidents of violence. Detecting and responding to violence in real time is crucial for ensuring passenger safety and the smooth operation of these transport networks. To address this issue, we propose an advanced artificial intelligence (AI) solution for identifying unsafe behaviours in public transport. The proposed approach employs deep learning action recognition models and utilises technologies like NVIDIA DeepStream SDK, Amazon Web Services (AWS) DirectConnect, local edge computing server, ONNXRuntime and MQTT to accelerate the end-to-end pipeline. The solution captures video streams from remote train stations closed circuit television (CCTV) networks, processes the data in the cloud, applies the action recognition model, and transmits the results to a live web application. A temporal pyramid network (TPN) action recognition model was trained on a newly curated video dataset mixing open-source resources and live simulated trials to identify the unsafe behaviours. The base model was able to achieve a validation accuracy of 93% when trained using open-source dataset samples and was improved to 97% when live simulated dataset was included during the training. The developed AI system was deployed at Wollongong Train Station (NSW, Australia) and showcased impressive accuracy in detecting violence incidents during an 8-week test period, achieving a reliable false-positive (FP) rate of 23%. While the AI correctly identified 30 true-positive incidents, there were 6 cases of false negatives (FNs) where violence incidents were missed during the rainy weather suggesting more data in the training dataset related to bad weather. The AI model's continuous retraining capability ensures its adaptability to various real-world scenarios, making it a valuable tool for enhancing safety and the overall passenger experience in public transport settings.

Analysis of Volleyball Tactics and Movements Based on 3D Spatio-Temporal Residual Network

Sports technology and 3D motion recognition require models to be able to accurately identify athletes’ movements, which is crucial for training analysis, game strategy development and refereeing assistance decisions. To maintain a high recognition rate under different competition scenes, different athlete styles and different environmental conditions, and to ensure the practicality and reliability of the model, two independent 3D convolutional neural networks are applied to construct the action recognition model of Two-stream 3D Residual Networks. According to the temporal-spatial characteristics of human movements in video, the model introduces attention mechanism and combines time dimension to build a Two-stream 3D Residual Networks action recognition model integrating time-channel attention. The average accuracy of Top-1 and Top-5 action recognition models of Two-stream 3D Residual Networks integrated with pre-activation structure is 68.97% and 91.68%. The residual block of the pre-activated structure can enhance the model’s effectiveness. The average precision of Top-1 and Top-5 of action recognition model of two-stream 3D spatio-temporal residual network integrating time-channel attention is 85.73% and 92.05%, which has higher accuracy. The action recognition model of the two-stream 3D spatio-temporal residual network, which integrates time-channel attention, is accurate and achieves good recognition results with volleyball action recognition in real scenes.

Unsupervised Temporal Adaptation in Skeleton-Based Human Action Recognition

With deep learning approaches, the fundamental assumption of data availability can be severely compromised when a model trained on a source domain is transposed to a target application domain where data are unlabeled, making supervised fine-tuning mostly impossible. To overcome this limitation, the present work introduces an unsupervised temporal-domain adaptation framework for human action recognition from skeleton-based data that combines Contrastive Prototype Learning (CPL) and Temporal Adaptation Modeling (TAM), with the aim of transferring the knowledge learned from a source domain to an unlabeled target domain. The CPL strategy, inspired by recent success in contrastive learning applied to skeleton data, learns a compact temporal representation from the source domain, from which the TAM strategy leverages the capacity for self-training to adapt the representation to a target application domain using pseudo-labels. The research demonstrates that simultaneously solving CPL and TAM effectively enables the training of a generalizable human action recognition model that is adaptive to both domains and overcomes the requirement of a large volume of labeled skeleton data in the target domain. Experiments are conducted on multiple large-scale human action recognition datasets such as NTU RGB+D, PKU MMD, and Northwestern–UCLA to comprehensively evaluate the effectiveness of the proposed method.

An Action Evaluation Method for Virtual Reality Simulation Power Training Based on an Improved Dynamic Time Warping Algorithm

To address the shortcomings in action evaluation within VR simulation power training, this paper introduces a novel action recognition and evaluation method based on dynamic recognition of finger keypoints combined with an improved Dynamic Time Warping (DTW) algorithm. By constructing an action recognition model centered on hand keypoints, the proposed method integrates distance similarity and cosine similarity to account comprehensively for both numerical differences and directional consistency of action features. This approach effectively tackles the challenges of feature extraction and recognition for complex actions in VR power training. Furthermore, a scoring mechanism based on the improved DTW algorithm is proposed, which employs Gaussian-weighted feature-derivative Euclidean distance combined with cosine similarity. This method significantly reduces computational complexity while improving scoring accuracy and consistency. Experimental results indicated that the improved DTW algorithm outperformed traditional methods in terms of classification accuracy and robustness. In particular, cosine similarity demonstrated superior performance in capturing dynamic variations and assessing the consistency of fine hand movements. This study provides an essential technical reference for action evaluation in VR simulation power training and offers a scientific basis for advancing the intelligence and digitalization of power VR training environments.

Construction of tennis pose estimation and action recognition model based on improved ST-GCN

With the rapid growth of computer vision and deep learning technologies, the application of pose estimation and action recognition in sports training has become increasingly widespread. Due to factors such as complex movements, fast speed, and limb occlusion, pose estimation and action recognition in tennis face significant challenges. Therefore, this study first introduces selective dropout and pyramid region of interest pooling layer strategies in fast region convolutional neural networks. Secondly, a pose estimation algorithm based on multi-scale fusion pose residual network 50 is designed, and finally a spatiotemporal graph convolutional network model is constructed by fusing channel attention module and multi-scale dilated convolution module. The data showed that the average detection accuracy of the improved attitude residual network 50 was 70.4%, and the accuracy of object detection for small, medium, and large objects was 57.4%, 69.3%, and 79.2%, respectively. The continuous action recognition accuracy and inter action fluency detection time of the improved spatiotemporal graph convolutional network were 93.8% and 19.2 ms, respectively. When the sample size was 1000, its memory usage was 1378 MB and the running time was 32.7 ms. Experiments have shown that the improved model achieves high accuracy and robustness in tennis action recognition tasks, especially in complex scenes and limb occlusion conditions, where the model shows significant advantages. This study aims to provide an efficient and accurate motion recognition technology for tennis posture analysis and intelligent training.

A hybrid deep learning model for UWB radar-based human activity recognition

A hybrid deep learning model for UWB radar-based human activity recognition

Skeleton-Based Human Action Recognition with Spatial and Temporal Attention-Enhanced Graph Convolution Networks

Skeleton-based human action recognition has great potential for human behavior analysis owing to its simplicity and robustness in varying environments. This paper presents a spatial and temporal attention-enhanced graph convolution network (STAEGCN) for human action recognition. The spatial-temporal attention module in the network uses convolution embedding for positional information and adopts multi-head self-attention mechanism to extract spatial and temporal attention separately from the input series of the skeleton. The spatial and temporal attention are then concatenated into an entire attention map according to a specific ratio. The proposed spatial and temporal attention module was integrated with an adaptive graph convolution network to form the backbone of STAEGCN. Based on STAEGCN, a two-stream skeleton-based human action recognition model was trained and evaluated. The model performed better on both NTU RGB+D and Kinetics 400 than 2s-AGCN and its variants. It was proven that the strategy of decoupling spatial and temporal attention and combining them in a flexible way helps improve the performance of graph convolution networks in skeleton-based human action recognition.

Toward Improving Human Training by Combining Wearable Full-Body IoT Sensors and Machine Learning.

This paper proposes DigitalUpSkilling, a novel IoT- and AI-based framework for improving and personalising the training of workers who are involved in physical-labour-intensive jobs. DigitalUpSkilling uses wearable IoT sensors to observe how individuals perform work activities. Such sensor observations are continuously processed to synthesise an avatar-like kinematic model for each worker who is being trained, referred to as the worker's digital twins. The framework incorporates novel work activity recognition using generative adversarial network (GAN) and machine learning (ML) models for recognising the types and sequences of work activities by analysing an individual's kinematic model. Finally, the development of skill proficiency ML is proposed to evaluate each trainee's proficiency in work activities and the overall task. To illustrate DigitalUpSkilling from wearable IoT-sensor-driven kinematic models to GAN-ML models for work activity recognition and skill proficiency assessment, the paper presents a comprehensive study on how specific meat processing activities in a real-world work environment can be recognised and assessed. In the study, DigitalUpSkilling achieved 99% accuracy in recognising specific work activities performed by meat workers. The study also presents an evaluation of the proficiency of workers by comparing kinematic data from trainees performing work activities. The proposed DigitalUpSkilling framework lays the foundation for next-generation digital personalised training.

Direction and velocity kinematic features of point-light displays grasping actions are differentially coded within the action observation network

The processing of kinematic information embedded in observed actions is an essential ability for understanding others' behavior. Previous research showed that the action observation network (AON) may encode some action kinematic features. However, our understanding of how direction and velocity are encoded within the AON is still limited. In this study, we employed event-related fMRI to investigate the neural substrates specifically activated during observation of hand grasping actions presented as point-light displays, performed with different directions (right, left) and velocities (fast, slow). Twenty-three healthy adult participants took part in the study. To identify brain regions differentially recruited by grasping direction and velocity, univariate and multivariate pattern analysis (MVPA) were performed. The results of univariate analysis demonstrate that direction is encoded in occipito-temporal and posterior visual areas, while velocity recruits lateral occipito-temporal, superior parietal and intraparietal areas. Results of MVPA further show: a) a significant decoding accuracy of both velocity and direction at the network level; b) the possibility to decode within lateral occipito-temporal and parietal areas both direction and velocity; c) a contribution of bilateral premotor areas to velocity decoding models. These results indicate that posterior parietal nodes of the AON are mainly involved in coding grasping direction and that premotor regions are crucial for coding grasping velocity, while lateral occipito-temporal cortices play a key role in encoding both parameters. The current findings could have implications for observational-based rehabilitation treatments of patients with motor disorders and artificial intelligence-based hand action recognition models.

The Application of GCN Algorithm in Building Construction Knowledge Graph Updating under the Combination of Artificial Intelligence and Knowledge Management

To promote the updating and iteration of the construction field, the construction knowledge graph can expand the professional knowledge system within the field and provide scientific and reasonable management decisions. The research adopts deep learning technology to update the knowledge graph in computer vision. The specific update method is to extract entities and analyze entity relationships in the framework of the knowledge graph, to achieve the application of the knowledge graph in the field of construction. Afterwards, a graph convolutional neural network is used to integrate semantic information and construct a network model of semantic embedding vectors. Finally, zero sample learning technologies are combined to enhance prior information during the construction process, thus demonstrating a better backward process for updating iterations. Through testing and validation of the dataset, it is found that the highest TOP-1 accuracy of the zero sample object detection model is 77%, 30%, and 45%, respectively, under different proportions of the dataset. The TOP-1 accuracy of the zero sample action recognition model increases by 22%, 23%, and 41%, respectively, compared to the former. In addition, the detection model combined with semantic embedding vectors has an average accuracy of 45% for unknown class tasks at a ratio of 8:2. This proves that the performance of the zero sample human object interaction detection model based on semantic embedding vectors is superior, providing a knowledge foundation and theoretical support for the updating of knowledge graphs in the field of construction.

Combined LinkNet-MBi-LSTM for brain activity recognition with new Stockwell transform features.

Recognizing brain activity from EEG waves is an important field of study in biomedical engineering and neuroscience. Conventional approaches usually begin with signal processing techniques to extract features from the EEG data, and then machine learning algorithms are applied to classify the data. However, the spatial resolution of these EEG signals is low, which makes it difficult to pinpoint the exact location of the neural activity source in the brain. There are ongoing initiatives to use DL-based brain activity recognition algorithms to overcome these constraints. Therefore, this work presents a novel hybrid framework for brain activity detection using the enhanced Stockwell transform and an EEG signal that is called LinkNet and modified bidirectional-long short-term memory (LN-MBi-LSTM) model. This framework follows a methodical approach that includes stages for feature extraction, brain activity recognition and preprocessing. Firstly, the improved Weiner filtering (IWF) approach is used to preprocess the EEG input signal. The relevant features are then extracted using a feature extraction technique from the preprocessed EEG signal. To identify the brain activity, these recovered feature sets are subsequently processed separately using LinkNet and modified bidirectional-long short-term memory (MBi-LSTM). A thorough analysis that takes into account both simulation and experimental calculations is part of the validation process for the LN-MBi-LSTM model. Finally, this study demonstrates the therapeutic potential of the LN-MBi-LSTM framework by presenting a strong and verified model for brain activity recognition. With the highest precision of 0.997, the LinkNet-MBi-LSTM model distinguishes itself from other models and confirms its exceptional capacity to produce accurate positive predictions.

Exploring Cutout and Mixup for Robust Human Activity Recognition on Sensor and Skeleton Data

Human Activity Recognition (HAR) is an essential area of research in Artificial Intelligence and Machine Learning, with numerous applications in healthcare, sports science, and smart environments. While several advancements in the field, such as attention-based models and Graph Neural Networks, have made great strides, this work focuses on data augmentation methods that tackle issues like data scarcity and task variability in HAR. In this work, we investigate and expand the use of mixup and cutout data augmentation methods to sensor-based and skeleton-based HAR datasets. These methods were first widely used in Computer Vision and Natural Language Processing. We use both augmentation techniques, customized for time-series and skeletal data, to improve the robustness and performance of HAR models by diversifying the data and overcoming the drawbacks of having limited training data. Specifically, we customize mixup data augmentation for sensor-based datasets and cutout data augmentation for skeleton-based datasets with the goal of improving model accuracy without adding more data. Our results show that using mixup and cutout techniques improves the accuracy and generalization of activity recognition models on both sensor-based and skeleton-based human activity datasets. This work showcases the potential of data augmentation techniques on transformers and Graph Neural Networks by offering a novel method for enhancing time series and skeletal HAR tasks.

Joint Mixing Data Augmentation for Skeleton-based Action Recognition

Skeleton-based action recognition is beneficial for understanding human behavior in videos, and thus has received much attention in recent years as an important research area in action recognition. Current research focuses on designing more advanced algorithms to better extract spatio-temporal information from skeleton data. However, due to the small amount of data in the existing skeleton dataset and the lack of effective data augmentation methods, it is easy to lead to overfitting in model training. To address this challenge, we propose a mix-based data augmentation method, Joint Mixing Data Augmentation (JMDA), which can generally improve the effectiveness and robustness of various skeleton-based action recognition algorithms. In terms of spatial information, we introduce SpatialMix (SM), a method that projects the original 3D skeleton discrete information into a 2D space. Then, SM mixes the projected spatial information between two random samples during the training process to achieve the spatial-based mixing data augmentation. Concerning temporal information, we propose TemporalMix (TM). Leveraging the temporal continuity in skeleton data, we perform a temporal resize operation on the original skeleton data, and then merge two random samples during training to achieve the temporal-based mixed data augmentation. Additionally, we analyze the Feature Mismatch (FM) problem caused by introducing mix-based data augmentation into skeleton data. Then we propose a new data preprocessing method called Feature Alignment (FA) to effectively address this problem and improve model performance. Moreover, we propose a novel training pipeline, Joint Training Strategy (JTS), which combines multiple mix-based data augmentation methods for further improvement of model performance. Specifically, our proposed JMDA is plug-and-play and widely applicable to skeleton-based action recognition models. At the same time, the application of JMDA does not increase the model parameters and there is almost no additional training cost. We conduct extensive experiments on NTU RGB+D 60 and NTU RGB+D 120 datasets to demonstrate the effectiveness and robustness of the proposed JMDA on several mainstream skeleton-based action recognition algorithms.

METAHEURISTIC-ASSISTED HYBRID RECOGNITION MODEL FOR BRAIN ACTIVITY DETECTION

Electroencephalogram (EEG) recordings typically capture the integrals of active brain potentials, which vary in latencies and populations. Anomalies in EEG data, often associated with epilepsy, play a crucial role in identifying conditions such as brain death, encephalopathy, coma, depth of anesthesia, and sleep disturbances. To get early warnings for these diseases, this work intends to propose a novel approach for brain activity detection from EEG signals. A new Hybrid Classification of Combined Coot Blue Monkey Optimization (HC-CCBO) method is proposed in this work. Initially, improved [Formula: see text]-score normalization was used to preprocess the EEG signal. Further, Discrete Wavelet Transform (DWT), improved correlation and statistical features were extracted. After that, we set up hybrid classification, which exploited Bi-Directional Gated Recurrent Unit (Bi-GRU) and Deep Max Out (DMO) models. Further, weights of BI-GRU and DMO were optimized via Combined Coot Blue Monkey Optimization (CCBO) optimization. Finally, we obtain the output scores of the rest, left fist, both fists, right fist, and both feet from the suggested hybrid brain activity recognition model. The effectiveness of the suggested HC-CCBO model is compared to conventional techniques using a variety of metrics. Compared to the existing models, the suggested model obtains a high maximum accuracy of 0.93 at the 90th learning percentage.

Lower Limb Motion Recognition Based on sEMG and CNN-TL Fusion Model.

To enhance the classification accuracy of lower limb movements, a fusion recognition model integrating a surface electromyography (sEMG)-based convolutional neural network, transformer encoder, and long short-term memory network (CNN-Transformer-LSTM, CNN-TL) was proposed in this study. By combining these advanced techniques, significant improvements in movement classification were achieved. Firstly, sEMG data were collected from 20 subjects as they performed four distinct gait movements: walking upstairs, walking downstairs, walking on a level surface, and squatting. Subsequently, the gathered sEMG data underwent preprocessing, with features extracted from both the time domain and frequency domain. These features were then used as inputs for the machine learning recognition model. Finally, based on the preprocessed sEMG data, the CNN-TL lower limb action recognition model was constructed. The performance of CNN-TL was then compared with that of the CNN, LSTM, and SVM models. The results demonstrated that the accuracy of the CNN-TL model in lower limb action recognition was 3.76%, 5.92%, and 14.92% higher than that of the CNN-LSTM, CNN, and SVM models, respectively, thereby proving its superior classification performance. An effective scheme for improving lower limb motor function in rehabilitation and assistance devices was thus provided.

Spatio-temporal interactive reasoning model for multi-group activity recognition

Multi-group activity recognition aims to recognize sub-group activities in multi-person scenes. Existing works explore group-level features by simply using graph neural networks for reasoning about the individual interactions and directly aggregating individual features, which cannot fully mine the interactions between people and between sub-groups, resulting in the loss of useful information for group activity recognition. To address this problem, this paper proposes a Spatio-Temporal Interactive Reasoning Model (STIRM) to better exploit potential spatio-temporal interactions for multi-group activity recognition. In particular, we present an interactive feature extraction strategy to explore correlation features between individuals by analyzing the features of their nearest neighbor. We design a new clustering module that combines the action similarity feature and spatio-temporal trajectory feature to divide people into small groups. In addition, to obtain rich and accurate group-level features, a group interaction reasoning module is constructed to explore the interactions between different small groups and among people in the same group and exclude people who have less impact on group activities according to their importance. Extensive experiments on the Social-CAD, PLPS and JRDB-PAR datasets indicate the superiority of the proposed method over state-of-the-art methods.

The Adversarial Attacks towards Human Action Recognition Models: A Comparison between unimodal and multimodal Models

Human-action recognition models are neural networks that analyse visual inputs and provide classification or text outputs. This technology has and will significantly impact society in security, education, healthcare, etc. However, Human-Action Recognition models, like other neural networks, are still susceptible to malicious adversarial attacks. Therefore, this paper proposes an experimental adversarial attack towards ResNet-18 using FGSM. First, ResNet-18 is finetuned using the UCF-101 dataset, and keyframes are selected from sample videos. The keyframes will be given to ResNet-18 for classification while FGSM will be implemented, and ResNet will do another classification of the attached sample. The classification results (Original and Attacked) are given to the language model (GPT-4o) through a prompt that provides the language model with a specific role (e.g. a smart home assistant), and this section is regarded as unimodal. The original and attacked frames will be sent directly instead of the labels in the multimodal section. Lastly, this paper proposes to observe the effects on textual responses generated based on a given prompt and the classification result and evaluate the impact of the attack through Cosine Similarities and Human Evaluation.

Belief-Rule-Based System with Self-organizing and Multi-temporal Modeling for Sensor-based Human Activity Recognition.

Smart environment is an efficient and cost- effective way to afford intelligent supports for the elderly people. Human activity recognition (HAR) is a crucial aspect of the research field of smart environments, and it has attracted widespread attention lately. The goal of this study is to develop an effective sensor-based HAR model based on the belief-rule-based system (BRBS), which is one of representative rule-based expert systems. Specially, a new belief rule base (BRB) modeling approach is proposed by taking into account the self- organizing rule generation method and the multi-temporal rule representation scheme, in order to address the problem of combination explosion that existed in the traditional BRB modelling procedure and the time correlation found in continuous sensor data in chronological order. The new BRB modeling approach is so called self-organizing and multi-temporal BRB (SOMT-BRB) modeling procedure. A case study is further deducted to validate the effectiveness of the SOMT-BRB modeling procedure. By comparing with some conventional BRBSs and classical activity recognition models, the results show a significant improvement of the BRBS in terms of the number of belief rules, modelling efficiency, and activity recognition accuracy.