Activity Recognition Accuracy Research Articles

Belief-Rule-Based System with Self-organizing and Multi-temporal Modeling for Sensor-based Human Activity Recognition.

Smart environment is an efficient and cost- effective way to afford intelligent supports for the elderly people. Human activity recognition (HAR) is a crucial aspect of the research field of smart environments, and it has attracted widespread attention lately. The goal of this study is to develop an effective sensor-based HAR model based on the belief-rule-based system (BRBS), which is one of representative rule-based expert systems. Specially, a new belief rule base (BRB) modeling approach is proposed by taking into account the self- organizing rule generation method and the multi-temporal rule representation scheme, in order to address the problem of combination explosion that existed in the traditional BRB modelling procedure and the time correlation found in continuous sensor data in chronological order. The new BRB modeling approach is so called self-organizing and multi-temporal BRB (SOMT-BRB) modeling procedure. A case study is further deducted to validate the effectiveness of the SOMT-BRB modeling procedure. By comparing with some conventional BRBSs and classical activity recognition models, the results show a significant improvement of the BRBS in terms of the number of belief rules, modelling efficiency, and activity recognition accuracy.

CAPTURE-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition

Existing activity tracker datasets for human activity recognition are typically obtained by having participants perform predefined activities in an enclosed environment under supervision. This results in small datasets with a limited number of activities and heterogeneity, lacking the mixed and nuanced movements normally found in free-living scenarios. As such, models trained on laboratory-style datasets may not generalise out of sample. To address this problem, we introduce a new dataset involving wrist-worn accelerometers, wearable cameras, and sleep diaries, enabling data collection for over 24 hours in a free-living setting. The result is CAPTURE-24, a large activity tracker dataset collected in the wild from 151 participants, amounting to 3883 hours of accelerometer data, of which 2562 hours are annotated. CAPTURE-24 is two to three orders of magnitude larger than existing publicly available datasets, which is critical to developing accurate human activity recognition models.

Tennis teaching assistance model based on double chain shared unsupervised action recognition algorithm

The traditional self-supervised methods based on skeleton data frequently categorize the various enhancements of a given sample as positive examples, while the remaining samples are designated as negative examples. This approach results in a significant imbalance in the ratio of positive to negative samples, which in turn constrains the efficacy of samples with identical semantic information. Therefore, to further improve the training quality in tennis teaching and enhance the accuracy of action recognition, a dual chain sharing unsupervised action recognition algorithm has been proposed. This study first designs a skeleton topology data augmentation method based on the physical connections of human joints to obtain advanced semantic embeddings. Then, an improved positive sample expansion strategy is utilized to enhance the diversity and quality of training data. Next, an unsupervised learning mechanism is employed to autonomously learn and recognize complex patterns of tennis movements. To measure the performance of the model, tests were conducted on its accuracy, F1 score, recognition time, and fitting degree. The experimental results showed that the proposed algorithm could reach its optimal state after 23 iterations of training, and its F1 value reached 0.918, with an average accuracy of 92.9 % and an average recognition time of 7.4 s. The research algorithms were superior to the multi-input branch graph convolutional network action recognition algorithms used for comparison, pull-push contrastive loss action recognition algorithms, and multi-granularity anchor contrastive representation learning action recognition algorithms. Its accuracy was leading by 8.6 %-23.5 %, and the recognition time has been reduced by 2.3s-7.4 s. In addition, in terms of the fitting degree of action recognition, compared with other methods, the proposed method had a fitting degree of up to 95.7 %, which was at least 8.1 % higher. This research method can improve the timeliness of feedback from trainers by accurately and quickly recognizing movements, helping coaches develop more targeted training plans, thereby improving the efficiency and quality of tennis teaching.

A novel rehabilitation action recognition approach using AF-LiteFormer from millimeter wave radar

Abstract Rehabilitation action recognition is a hot research topic in the medical field, which serves as the foundation for achieving remote healthcare, unsupervised exercise, intelligent home healthcare, and possesses extensive application value. Currently, vision-based action recognition methods are susceptible to limitations imposed by factors such as range of motion and environmental lighting during human motion capture. Due to its ability to effectively protect patient privacy and its immunity to lighting conditions, this paper proposed a millimeter-wave radar-based rehabilitation action recognition system, AF-LiteFormer. Firstly, EfficientFormerV2 is employed as the baseline, then a Lite-MSLA-FFN block-and-layer is designed to replace MHSA in EfficientFormerV2, which improves the diversity of attention and reduces computational complexity. Meanwhile, a Lite-Subsample block-and-layer is designed to replace the dual-path downsampling part that composed of attention downsampling and stride attention to realize global modeling and multi-scale learning.
Secondly, the iterative attention feature fusion (iAFF) mechanism is introduced to improve the recognition accuracy of rehabilitation actions. Finally, the effectiveness of the AF-LiteFormer model is validated on a self-collected rehabilitation action dataset and a publicly available micro-Doppler dataset. Experimental results show that the overall performance of the AF-LiteFormer model is better than the State-of-the-Art model (SOTA), the recognition accuracy of rehabilitation actions is as high as 99.7%, and it has strong generalization ability.

Control model of community elderly recreational exercise assistive robot based on improved dense trajectory algorithm

As the number of elderly population in the community grows, more efficient and precise recreation and exercise aids are needed to safeguard their quality of life. The study proposes a control model based on an improved dense trajectory algorithm to enhance the recognition and response capabilities of recreation and exercise assistance robots. The main method of the model is to improve the dense trajectory algorithm to enhance its recognition speed and accuracy for complex and small movements. Specifically, the study deeply refined the control process of a health and exercise assisted robot, and combined action capture to construct a health and exercise assisted robot model. The control model of the health and exercise assisted robot was optimized using an improved dense algorithm. The improved dense trajectory algorithm has feature embedding and attention mechanism, which can supplement the input data of the model, thus enabling more accurate action recognition. The results show that among the five samples, the recreation effectiveness score of the experimental group averaged 8.9, which was significantly higher than that of the control group, which was 7.3. The recognition accuracy has been improved by 2.7 % and 3.9 %, respectively, effectively suppressing the influence of camera motion. After using the improved dense trajectory algorithm, the fitness of the health training assistant robot reached 96.25 % under the same processing time, which is 8.93 % higher than the traditional model's fitness of 87.32 %. In summary, the control model of a community elderly health exercise assistance robot based on improved dense trajectory algorithm has achieved more accurate and faster recognition and response to the actions of the elderly, providing a more efficient technical means for health exercise and improving the health effect of the elderly.

Swimtrans Net: a multimodal robotic system for swimming action recognition driven via Swin-Transformer.

Currently, using machine learning methods for precise analysis and improvement of swimming techniques holds significant research value and application prospects. The existing machine learning methods have improved the accuracy of action recognition to some extent. However, they still face several challenges such as insufficient data feature extraction, limited model generalization ability, and poor real-time performance. To address these issues, this paper proposes an innovative approach called Swimtrans Net: A multimodal robotic system for swimming action recognition driven via Swin-Transformer. By leveraging the powerful visual data feature extraction capabilities of Swin-Transformer, Swimtrans Net effectively extracts swimming image information. Additionally, to meet the requirements of multimodal tasks, we integrate the CLIP model into the system. Swin-Transformer serves as the image encoder for CLIP, and through fine-tuning the CLIP model, it becomes capable of understanding and interpreting swimming action data, learning relevant features and patterns associated with swimming. Finally, we introduce transfer learning for pre-training to reduce training time and lower computational resources, thereby providing real-time feedback to swimmers. Experimental results show that Swimtrans Net has achieved a 2.94% improvement over the current state-of-the-art methods in swimming motion analysis and prediction, making significant progress. This study introduces an innovative machine learning method that can help coaches and swimmers better understand and improve swimming techniques, ultimately improving swimming performance.

Sports action recognition algorithm based on multi-modal data recognition

The recognition of sports action is an important research subject, which is conducive to the improvement of athletes’ own level. To improve the accuracy of multi-modal data action recognition, based on the Transformer module, this study introduces a multi-head attention mechanism, fuses multi-modal data, and constructs a multi-stream structured object relationship inference network. Based on PointNet++ network and combining five different data fusion frameworks, a motion recognition model that integrates RGB data and 3D skeleton point cloud is constructed. The results showed that the Top-1 accuracy of multi-stream structured object relationship inference network was 42.5% and 42.7%, respectively, which was better than other algorithms. The accuracy of the multi-modal fusion model was improved by 15.6% and 5.1% compared with the single mode, and by 5.4% and 2.6% compared with the dual mode, which showed its superiority in the action recognition task. This showed that the fusion of multi-modal data can provide more abundant information, so as to improve the accuracy of action recognition. The accuracy of the action recognition model combining RGB data and 3D skeleton point cloud was 84.3%, 87.5%, 90.2%, 90.6% and 91.2% after the combination of different strategies, which effectively compensated for the problem of missing information in 3D skeleton point cloud and significantly improved the accuracy of action recognition. With a small amount of data, the Top-1 accuracy of the multi-stream structured object relationship inference network in this study was superior to other algorithms, showing its advantages in dealing with complex action recognition tasks. In addition, the action recognition model that fuses RGB data and 3D skeleton point cloud also achieved higher accuracy, which is better than other algorithms. This study can meet the needs of motion recognition in different scenarios and has certain reference value.

A novel multi-stream hand-object interaction network for assembly action recognition

PurposeAssembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition.Design/methodology/approachIn this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method.FindingsThe method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production.Originality/valueThe author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

Human action recognition in immersive virtual reality based on multi‐scale spatio‐temporal attention network

AbstractWearable human action recognition (HAR) has practical applications in daily life. However, traditional HAR methods solely focus on identifying user movements, lacking interactivity and user engagement. This paper proposes a novel immersive HAR method called MovPosVR. Virtual reality (VR) technology is employed to create realistic scenes and enhance the user experience. To improve the accuracy of user action recognition in immersive HAR, a multi‐scale spatio‐temporal attention network (MSSTANet) is proposed. The network combines the convolutional residual squeeze and excitation (CRSE) module with the multi‐branch convolution and long short‐term memory (MCLSTM) module to extract spatio‐temporal features and automatically select relevant features from action signals. Additionally, a multi‐head attention with shared linear mechanism (MHASLM) module is designed to facilitate information interaction, further enhancing feature extraction and improving accuracy. The MSSTANet network achieves superior performance, with accuracy rates of 99.33% and 98.83% on the publicly available WISDM and PAMPA2 datasets, respectively, surpassing state‐of‐the‐art networks. Our method showcases the potential to display user actions and position information in a virtual world, enriching user experiences and interactions across diverse application scenarios.

Multihead-Res-SE Residual Network with Attention for Human Activity Recognition

Human activity recognition (HAR) typically uses wearable sensors to identify and analyze the time-series data they collect, enabling recognition of specific actions. As such, HAR is increasingly applied in human–computer interaction, healthcare, and other fields, making accurate and efficient recognition of various human activities. In recent years, deep learning methods have been extensively applied in sensor-based HAR, yielding remarkable results. However, complex HAR research, which involves specific human behaviors in varied contexts, still faces several challenges. To solve these problems, we propose a multi-head neural network based on the attention mechanism. This framework contains three convolutional heads, with each head designed using one-dimensional CNN to extract features from sensory data. The model uses a channel attention module (squeeze–excitation module) to enhance the representational capabilities of convolutional neural networks. We conducted experiments on two publicly available benchmark datasets, UCI-HAR and WISDM, to evaluate our model. The results were satisfactory, with overall recognition accuracies of 96.72% and 97.73% on their respective datasets. The experimental results demonstrate the effectiveness of the network structure for the HAR, which ensures a higher level of accuracy.

Using Human Skeleton Point Repair Methods to Improve the Accuracy of Multi-person Interactive Activity Recognition

Using Human Skeleton Point Repair Methods to Improve the Accuracy of Multi-person Interactive Activity Recognition

Human Fall Detection with Ultra-Wideband Radar and Adaptive Weighted Fusion.

To address the challenges in recognizing various types of falls, which often exhibit high similarity and are difficult to distinguish, this paper proposes a human fall classification system based on the SE-Residual Concatenate Network (SE-RCNet) with adaptive weighted fusion. First, we designed the innovative SE-RCNet network, incorporating SE modules after dense and residual connections to automatically recalibrate feature channel weights and suppress irrelevant features. Subsequently, this network was used to train and classify three types of radar images: time-distance images, time-distance images, and distance-distance images. By adaptively fusing the classification results of these three types of radar images, we achieved higher action recognition accuracy. Experimental results indicate that SE-RCNet achieved F1-scores of 94.0%, 94.3%, and 95.4% for the three radar image types on our self-built dataset. After applying the adaptive weighted fusion method, the F1-score further improved to 98.1%.

A Review of Deep Learning Based Video Action Recognition Techniques

Video action recognition is an important research direction in the field of computer vision and pattern recognition, with extensive applications in intelligent video surveillance, human-computer interaction, and sports analysis. The development of data storage and computing hardware over the past decade has driven a shift from traditional feature extraction and machine learning algorithms to deep learning-based approaches. This paper reviews the current state of development, problems, and future research directions of video action recognition techniques. Traditional methods are gradually being replaced by deep learning methods such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long-short-term memory networks (LSTMs). These methods automatically extract features and handle time-dependency, significantly improving the accuracy and robustness of action recognition. In particular, models based on the attention mechanism further enhance action recognition performance by dynamically adjusting the focus of attention, a current hot spot in research. Despite many advances, video action recognition still faces several challenges, including high computational resource requirements, complex model training, dataset bias issues, and variations in real-world application scenarios such as viewpoint changes, lighting changes, and occlusion. Future research can explore multi-modal fusion, lightweight models, self-supervised learning, and cross-domain transfer learning to improve the accuracy, robustness, and generalization of action recognition. The review provided aims to offer researchers a comprehensive perspective on the current state of development and future research directions of video action recognition technology.

Multi-scale control and action recognition based human-robot collaboration framework facing new generation intelligent manufacturing

Facing the new generation intelligent manufacturing, traditional manufacturing models are transitioning towards large-scale customized productions, improving the efficiency and flexibility of complex manufacturing processes. This is crucial for enhancing the stability and core competitiveness of the manufacturing industry, and human-robot collaboration systems are an important means to achieve this goal. At present, mainstream manufacturing human-robot collaboration systems are modeled for specific scenarios and actions, with poor scalability and flexibility, making it difficult to flexibly handle actions beyond the set. Therefore, this article proposes a new human-robot collaboration framework based on action recognition and multi-scale control, designs 27 basic gesture actions for motion control, and constructs a robot control instruction library containing 70 different semantics based on these actions. By integrating static gesture recognition, dynamic action process recognition, and You-Only-Look-Once V5 object recognition and positioning technology, accurate recognition of various control actions has been achieved. The recognition accuracy of 27 types of static control actions has reached 100%, and the dynamic action recognition accuracy of the gearbox assembly process based on lightweight MF-AE-NNOBJ has reached 90%. This provides new ideas for simplifying the complexity of human-robot collaboration problems, improving system accuracy, efficiency, and stability.

ChatMatch: Exploring the potential of hybrid vision–language deep learning approach for the intelligent analysis and inference of racket sports

Video understanding technology has become increasingly important in various disciplines, yet current approaches have primarily focused on lower comprehension level of video content, posing challenges for providing comprehensive and professional insights at a higher comprehension level. Video analysis plays a crucial role in athlete training and strategy development in racket sports. This study aims to demonstrate an innovative and higher-level video comprehension framework (ChatMatch), which integrates computer vision technologies with the cutting-edge large language models (LLM) to enable intelligent analysis and inference of racket sports videos. To examine the feasibility of this framework, we deployed a prototype of ChatMatch in the badminton in this study. A vision-based encoder was first proposed to extract the meta-features included the locations, actions, gestures, and action results of players in each frame of racket match videos, followed by a rule-based decoding method to transform the extracted information in both structured knowledge and unstructured knowledge. A set of LLM-based agents included namely task identifier, coach agent, statistician agent, and video manager, was developed through a prompt engineering and driven by an automated mechanism. The automatic collaborative interaction among the agents enabled the provision of a comprehensive response to professional inquiries from users. The validation findings showed that our vision models had excellent performances in meta-feature extraction, achieving a location identification accuracy of 0.991, an action recognition accuracy of 0.902, and a gesture recognition accuracy of 0.950. Additionally, a total of 100 questions were gathered from four proficient badminton players and one coach to evaluate the performance of the LLM-based agents, and the outcomes obtained from ChatMatch exhibited commendable results across general inquiries, statistical queries, and video retrieval tasks. These findings highlight the potential of using this approach that can offer valuable insights for athletes and coaches while significantly improve the efficiency of sports video analysis.

Tag-free indoor fall detection using transformer network encoder and data fusion

This work presents a radio frequency identification (RFID)-based technique to detect falls in the elderly. The proposed RFID-based approach offers a practical and efficient alternative to wearables, which can be uncomfortable to wear and may negatively impact user experience. The system utilises strategically positioned passive ultra-high frequency (UHF) tag array, enabling unobtrusive monitoring of elderly individuals. This contactless solution queries battery-less tag and processes the received signal strength indicator (RSSI) and phase data. Leveraging the powerful data-fitting capabilities of a transformer model to take raw RSSI and phase data as input with minimal preprocessing, combined with data fusion, it significantly improves activity recognition and fall detection accuracy, achieving an average rate exceeding 96.5%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$96.5\\%$$\\end{document}. This performance surpasses existing methods such as convolutional neural network (CNN), recurrent neural network (RNN), and long short-term memory (LSTM), demonstrating its reliability and potential for practical implementation. Additionally, the system maintains good accuracy beyond a 3-m range using minimal battery-less UHF tags and a single antenna, enhancing its practicality and cost-effectiveness.

2D human skeleton action recognition with spatial constraints

AbstractHuman actions are predominantly presented in 2D format in video surveillance scenarios, which hinders the accurate determination of action details not apparent in 2D data. Depth estimation can aid human action recognition tasks, enhancing accuracy with neural networks. However, reliance on images for depth estimation requires extensive computational resources and cannot utilise the connectivity between human body structures. Besides, the depth information may not accurately reflect actual depth ranges, necessitating improved reliability. Therefore, a 2D human skeleton action recognition method with spatial constraints (2D‐SCHAR) is introduced. 2D‐SCHAR employs graph convolution networks to process graph‐structured human action skeleton data comprising three parts: depth estimation, spatial transformation, and action recognition. The initial two components, which infer 3D information from 2D human skeleton actions and generate spatial transformation parameters to correct abnormal deviations in action data, support the latter in the model to enhance the accuracy of action recognition. The model is designed in an end‐to‐end, multitasking manner, allowing parameter sharing among these three components to boost performance. The experimental results validate the model's effectiveness and superiority in human skeleton action recognition.

Human movement science-informed multi-task spatio temporal graph convolutional networks for fitness action recognition and evaluation

In recent years, with the rise of health consciousness, people’s demand for fitness has steadily increased. Utilizing automated human action recognition technology to monitor users’ movements during exercise continuously would help prevent situations where incorrect movements lead to injuries while working out. Skeleton-based human action recognition methods can overcome the susceptibility of past color-based and depth-based methods to various external backgrounds and noise, becoming a more successful solution in recent years. In this study, the auto-fitness advisor system we propose not only identifies the category of the action but also assesses the quality of the action and provides suggestions. We integrate human movement science, such as the Five Primary Kinetic Chains (5PKC), which defines the primary physiological principles in human movement, to enhance the accuracy of fitness action recognition by providing a more precise relationship between the human skeleton and muscles. For assessing the quality of movements and providing suggestions, we have designed a multi-task objective function within our model. Overall, the proposed model is a multi-task model based on Spatio Temporal Graph Convolutional Networks (ST-GCN), which employs the Five Primary Kinetic Chains (5PKC) as a partitioning strategy for skeletal information. In our experiments, we not only collected a certain amount of datasets in gyms to validate the performance of our model but also compared it with other current methods using existing public datasets.

Humanactivity Recognition using Deep Learning

Human activity recognition (HAR) plays a crucial role in various fields, including healthcare, surveillance, and human-computer interaction. This study explores the application of deep learning techniques for accurate and efficient human activity recognition. Leveraging the capabilities of deep neural networks, specifically convolutional neural networks (CNNs) and recurrent neural networks (RNNs), the proposed approach aims to capture both spatial and temporal features from sensor data. The dataset utilized in this research comprises multi-model sensor inputs, such as accelerometer and gyroscope readings, collected from wearable devices. The deep learning model is designed to automatically learn hierarchical representations of the raw sensor data, enabling robust feature extraction and discrimination between different human activities. Transfer learning is employed to enhance model generalization across diverse activity categories and varying sensor setups. Experimental evaluations are conducted on real-world datasets, demonstrating the effectiveness of the proposed deep learning framework in accurately classifying activities, including walking, running, sitting, and standing. Comparative analyses against traditional machine learning methods underscore the superior performance and adaptability of deep learning in handling complex and dynamic activity patterns. The results showcase the potential of deploying deep learning models in real-time human activity recognition systems, highlighting their scalability and efficiency. The study contributes to the advancement of HAR methodologies, paving the way for the development of more reliable and robust systems in applications such as healthcare monitoring, assisted living, and smart environments.

A Residual Deep Learning Method for Accurate and Efficient Recognition of Gym Exercise Activities Using Electromyography and IMU Sensors

The accurate and efficient recognition of gym workout activities using wearable sensors holds significant implications for assessing fitness levels, tailoring personalized training regimens, and overseeing rehabilitation progress. This study introduces CNN-ResBiGRU, a novel deep learning architecture that amalgamates residual and hybrid methodologies, aiming to precisely categorize gym exercises based on multimodal sensor data. The primary goal of this model is to effectively identify various gym workouts by integrating convolutional neural networks, residual connections, and bidirectional gated recurrent units. Raw electromyography and inertial measurement unit data collected from wearable sensors worn by individuals during strength training and gym sessions serve as inputs for the CNN-ResBiGRU model. Initially, convolutional neural network layers are employed to extract unique features in both temporal and spatial dimensions, capturing localized patterns within the sensor outputs. Subsequently, the extracted features are fed into the ResBiGRU component, leveraging residual connections and bidirectional processing to capture the exercise activities’ long-term temporal dependencies and contextual information. The performance of the proposed model is evaluated using the Myogym dataset, comprising data from 10 participants engaged in 30 distinct gym activities. The model achieves a classification accuracy of 97.29% and an F1-score of 92.68%. Ablation studies confirm the effectiveness of the convolutional neural network and ResBiGRU components. The proposed hybrid model uses wearable multimodal sensor data to accurately and efficiently recognize gym exercise activity.