Activity Recognition Algorithms Research Articles

A multi-vision monitoring framework for simultaneous real-time unmanned aerial monitoring of farmer activity and crop health

Current remote sensing technologies employing Unmanned Aerial Vehicles (UAVs) for farm monitoring have shown promise in characterizing the environment through diverse sensor systems, including hyperspectral cameras, LiDAR, thermal cameras, and RGB sensors. However, these solutions often specialize in either activity recognition or crop monitoring, but not both. To address this limitation and enhance efficacy, we propose a multi-vision monitoring (MVM) framework capable of simultaneously recognizing farm activities and assessing crop health. Our approach involves computer vision techniques that transform aerial videos into sequential images to extract essential environmental features. Central to our framework are two pivotal components: the Farmer Activity Recognition (FAR) algorithm and the Crop Image Analysis (CIA). The FAR algorithm introduces a novel feature extraction method capturing motion across various maps, enabling distinct feature sets for each activity. Meanwhile, the CIA component utilizes the normalized Triangular Greenness Index (nTGI) to estimate leave chlorophyll levels, an important indicator for crop health. By unifying these components, we achieve dual functionality—activity recognition and crop health estimation—using identical input data, thereby enhancing efficiency and versatility in farm monitoring. Our framework employs a diverse range of machine learning models, demonstrating the potential of our extracted features to address the defined problem effectively in unison.

Human Activity Recognition Algorithm with Physiological and Inertial Signals Fusion: Photoplethysmography, Electrodermal Activity, and Accelerometry.

Inertial signals are the most widely used signals in human activity recognition (HAR) applications, and extensive research has been performed on developing HAR classifiers using accelerometer and gyroscope data. This study aimed to investigate the potential enhancement of HAR models through the fusion of biological signals with inertial signals. The classification of eight common low-, medium-, and high-intensity activities was assessed using machine learning (ML) algorithms, trained on accelerometer (ACC), blood volume pulse (BVP), and electrodermal activity (EDA) data obtained from a wrist-worn sensor. Two types of ML algorithms were employed: a random forest (RF) trained on features; and a pre-trained deep learning (DL) network (ResNet-18) trained on spectrogram images. Evaluation was conducted on both individual activities and more generalized activity groups, based on similar intensity. Results indicated that RF classifiers outperformed corresponding DL classifiers at both individual and grouped levels. However, the fusion of EDA and BVP signals with ACC data improved DL classifier performance compared to a baseline DL model with ACC-only data. The best performance was achieved by a classifier trained on a combination of ACC, EDA, and BVP images, yielding F1-scores of 69 and 87 for individual and grouped activity classifications, respectively. For DL models trained with additional biological signals, almost all individual activity classifications showed improvement (p-value < 0.05). In grouped activity classifications, DL model performance was enhanced for low- and medium-intensity activities. Exploring the classification of two specific activities, ascending/descending stairs and cycling, revealed significantly improved results using a DL model trained on combined ACC, BVP, and EDA spectrogram images (p-value < 0.05).

Athletes' Action Recognition and Posture Estimation Algorithm Based on Image Processing Technology

A posture estimation algorithm based on image processing technology involves analyzing images or video frames to determine the position and orientation of human bodies or specific body parts. This process typically begins with detecting key points or landmarks on the body, such as joints or anatomical features. This paper presents a novel approach for athletes' action recognition and posture estimation using image processing technology, specifically focusing on the Contour Mapping Multivariant Recognition and Estimation (CMMRE) algorithm. The primary objective is to develop a robust system capable of accurately identifying human actions and estimating postures from visual data, with potential applications in sports analysis, surveillance, and human-computer interaction. The CMMRE algorithm employs advanced techniques for contour mapping, feature extraction, and prediction, leveraging deep learning methodologies to analyze and interpret visual information effectively. Through a series of experiments and simulations, the algorithm's performance is evaluated, showcasing its ability to achieve high accuracy in action recognition tasks across various scenarios. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement. The algorithm achieved an average accuracy of 95% in recognizing walking and running actions, and 80% in identifying jumping actions. The results highlight the algorithm's strengths in accurately predicting actions and estimating postures, while also identifying areas for improvement.

Research on human behavior recognition algorithm based on HRNet and LSTM with model lightweight

This paper proposes a lightweight human action recognition algorithm for pedestrian behavior recognition. First, the skeleton feature information is input into the HRNet network model. In order to selectively enhance more details containing the target features and suppress irrelevant or weak features, an external attention mechanism is added to the HRN child model. Secondly, in order to extract the temporal characteristics of the target feature vector and ensure the continuity of actions in human behavior recognition, a dual-stream network based on HRNet and Long Short-Term Memory (LSTM) is constructed; finally, due to the huge model, it cannot be well transplanted to embedded. Therefore, this paper uses depthwise separable convolution to lightweight the network model. The experimental results show that in terms of human behavior recognition, the method in this paper has better recognition accuracy than Two-stream, Multi-streamCNN, Cov3DJ, ConvNets, JTM, ASM-3, RF+SW, hd-CNN and TPSMMs.

Self-supervised learning for human activity recognition using 700,000 person-days of wearable data

Accurate physical activity monitoring is essential to understand the impact of physical activity on one’s physical health and overall well-being. However, advances in human activity recognition algorithms have been constrained by the limited availability of large labelled datasets. This study aims to leverage recent advances in self-supervised learning to exploit the large-scale UK Biobank accelerometer dataset—a 700,000 person-days unlabelled dataset—in order to build models with vastly improved generalisability and accuracy. Our resulting models consistently outperform strong baselines across eight benchmark datasets, with an F1 relative improvement of 2.5–130.9% (median 24.4%). More importantly, in contrast to previous reports, our results generalise across external datasets, cohorts, living environments, and sensor devices. Our open-sourced pre-trained models will be valuable in domains with limited labelled data or where good sampling coverage (across devices, populations, and activities) is hard to achieve.

A micro-Doppler spectrogram denoising algorithm for radar human activity recognition

Radar signal recognition based on micro-Doppler spectrogram has been widely used in human action recognition tasks. However, in practical application scenarios, radar signals inevitably have noise, which leads to different degrees of deformation of the spectrogram graph structure and affects the accuracy of subsequent recognition algorithms. In this paper, we present “ACFL”, a novel algorithm for micro-Doppler spectrogram denoising, which aims to reduce the impact of noise on human action recognition. ACFL employs amplitude–frequency two-dimensional clustering and fuzzy logic clustering selection mechanism to remove noise elements from the spectrogram. Moreover, to address the issue of noise leakage or target missing under time-varying noise and action conditions, ACFL adopts spectrogram segmentation based on short-term Rényi entropy. By dividing the spectrogram into intervals with different time–frequency distributions, the dynamic spectrogram denoise over time is achieved. Simulation and measured data experiments demonstrate that the proposed algorithm not only achieves a higher-quality denoised spectrogram but also significantly improves the accuracy of human action recognition under noisy conditions.

Form Check: Exercise Posture Correction Application

Weightlifting is a popular sport and a hobby of millions; however, the risk of injury is high, primarily when performed with incorrect form/posture due to voluntary or involuntary reasons. It results in a significant loss of strength and hinders progress. The gradual strain on tendons and ligaments causes wear and tear. This paper aims to develop a deep learning model that detects incorrect posture and provides insight into remedial steps that need to be taken to correct the same, thereby providing increased safety and efficiency. Our Form Check model detects a user's form, vector geometry of posture is evaluated and corrected. A graphical representation of the exercise is provided as feedback to the user. Object detection, pose estimation, and action recognition algorithms infer the human skeleton and recognize the exercise. The inconsistency in the exercise performed is determined and mapped to specific imperfections in the stance of users during the exercise through Graph

HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors

The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which usually suffer from illumination, fast motion, privacy preservation, and large energy consumption. Meanwhile, the biologically inspired event cameras attracted great interest due to their unique features, such as high dynamic range, dense temporal but sparse spatial resolution, low latency, low power, etc. As it is a newly arising sensor, even there is no realistic large-scale dataset for HAR. Considering its great practical value, in this paper, we propose a large-scale benchmark dataset to bridge this gap, termed HARDVS, which contains 300 categories and more than 100K event sequences. We evaluate and report the performance of multiple popular HAR algorithms, which provide extensive baselines for future works to compare. More importantly, we propose a novel spatial-temporal feature learning and fusion framework, termed ESTF, for event stream based human activity recognition. It first projects the event streams into spatial and temporal embeddings using StemNet, then, encodes and fuses the dual-view representations using Transformer networks. Finally, the dual features are concatenated and fed into a classification head for activity prediction. Extensive experiments on multiple datasets fully validated the effectiveness of our model. Both the dataset and source code will be released at https://github.com/Event-AHU/HARDVS.

MDJ: A multi-scale difference joint keyframe extraction algorithm for infrared surveillance video action recognition

Many action recognition methods require significant computational resources to achieve good results on unedited videos. However, their performance on infrared videos, which contain less information, is often unsatisfactory. In this paper, we propose a multi-scale difference joint key frame extraction algorithm for action recognition in infrared surveillance videos. To evaluate our algorithm, we have created a self-built dataset comprising 1200 unedited infrared surveillance videos categorized into 10 action categories. Our algorithm extracts key frames by jointly analyzing the global and local differences between adjacent frames. Experimental analysis demonstrates that by using only 10 frames, our algorithm improves the accuracy of generic action recognition algorithms by more than 10% on both our self-built dataset and the Infrared-Visible dataset. Moreover, our proposed method achieves high recognition accuracy with minimal computational cost, even when using a small number of frames. It outperforms state-of-the-art methods by 1.82% on the Infrared-Visible dataset and 1.35% on the InfAR dataset. These results highlight the effectiveness of our algorithm as a preprocessing module to significantly enhance the accuracy of action recognition algorithms prior to employing generic models.

An Algorithm for Activity Recognition and Assessment of Adults Poststroke.

Stroke is the leading cause of long-term disability in the United States. Providers have no robust tools to objectively and accurately measure the activity of people with stroke living at home. To explore the integration of validated upper extremity assessments poststroke within an activity recognition system. Exploratory descriptive study using data previously collected over 3 mo to report on algorithm testing and assessment integration. Data were collected in the homes of community-dwelling participants. Participants were at least 6 mo poststroke, were able to ambulate with or without an assistive device, and self-reported some difficulty using their arm in everyday activities. The activity detection algorithm's accuracy was determined by comparing its activity labels with manual labels. The algorithm integrated assessment by describing the quality of upper extremity movement, which was determined by reporting extent of reach, mean and maximum speed during movement, and smoothness of movement. Sixteen participants (9 women, 7 men) took part in this study, with an average age of 63.38 yr (SD = 12.84). The algorithm was highly accurate in correctly identifying activities, with 87% to 95% accuracy depending on the movement. The algorithm was also able to detect the quality of movement for upper extremity movements. The algorithm was able to accurately identify in-kitchen activities performed by adults poststroke. Information about the quality of these movements was also successfully calculated. This algorithm has the potential to supplement clinical assessments in treatment planning and outcomes reporting. Plain-Language Summary: This study shows that clinical algorithms have the potential to inform occupational therapy practice by providing clinically relevant data about the in-home activities of adults poststroke. The algorithm accurately identified activities that were performed in the kitchen by adults poststroke. The algorithm also identified the quality of upper extremity movements of people poststroke who were living at home.

Video Abnormal Action Recognition Based on Multimodal Heterogeneous Transfer Learning

Human abnormal action recognition is crucial for video understanding and intelligent surveillance. However, the scarcity of labeled data for abnormal human actions often hinders the development of high-performance models. Inspired by the multimodal approach, this paper proposes a novel approach that leverages text descriptions associated with abnormal human action videos. Our method exploits the correlation between the text domain and the video domain in the semantic feature space and introduces a multimodal heterogeneous transfer learning framework from the text domain to the video domain. The text of the videos is used for feature encoding and knowledge extraction, and knowledge transfer and sharing are realized in the feature space, which is used to assist in the training of the abnormal action recognition model. The proposed method reduces the reliance on labeled video data, improves the performance of the abnormal human action recognition algorithm, and outperforms the popular video-based models, particularly in scenarios with sparse data. Moreover, our framework contributes to the advancement of automatic video analysis and abnormal action recognition, providing insights for the application of multimodal methods in a broader context.

Design of physical function monitoring system for athletes in soccer training in college sports

Abstract A comprehensive assessment of the physical capabilities and attributes of soccer players is essential for effective college soccer training. The ability to implement targeted tactical training relies on a thorough understanding of each player’s physical condition. This paper presents a monitoring system for soccer players’ bodily functions, developed using inertial sensor technology alongside Beer’s Law, tailored to recognize human body movement states and to measure blood oxygen and heart rate. The system’s efficacy in movement recognition and physiological monitoring was evaluated through comparative testing, assessing its performance against established algorithms. Notably, this system demonstrated a 9.75% improvement in the average accuracy for recognizing walking and running actions compared to the Juha Parkka decision tree algorithm and a 6.5% increase in accuracy over the D.M. Karantonis algorithm for walking action recognition. Heart rate measurements incurred an error of approximately ±2 BPM, with a maximum error of 1.7, and oxygen saturation monitoring displayed an error margin of ±2%, with a maximum deviation of 1.6. These deviations fall within acceptable limits. The results of this study furnish valuable data to support the customization of training content and intensity for college soccer athletes and offer insights for optimizing player selection and tactical strategies during matches.

Automation of Functional Mobility Assessments at Home Using a Multimodal Sensor System Integrating Inertial Measurement Units and Computer Vision (IMU-Vision).

Functional movement assessments are routinely used to evaluate and track changes in mobility. The objective of this study was to evaluate a multimodal movement monitoring system developed for autonomous, home-based, functional movement assessment. Fifty frail and prefrail adults were recruited from the Brigham and Women's Hospital at Home program to evaluate the feasibility and accuracy of applying the multimodal movement monitoring system to autonomously recognize and score functional activities collected in the home. Study subjects completed sit-to-stand, standing balance (Romberg, semitandem, and tandem), and walking test activities in likeness to the Short Physical Performance Battery. Test activities were identified and scored manually and by the multimodal movement monitoring system's activity recognition and scoring algorithms, which were previously trained on lab-based biomechanical data to integrate wearable inertial measurement unit (IMU) and external red-blue-green-depth vision data. Feasibility was quantified as the proportion of completed tests that were analyzable. Accuracy was quantified as the degree of agreement between the actual and system-identified activities. In an exploratory analysis of a subset of functional activity data, the accuracy of a preliminary activity-scoring algorithm was also evaluated. Activity recognition by the IMU-vision system had good feasibility and high accuracy. Of 271 test activities collected in the home, 217 (80%) were analyzable by the activity-recognition algorithm, which overall correctly identified 206 (95%) of the analyzable activities: 100% of walking, 97% of balance, and 82% of sit-to-stand activities (χ2(2) = 19.9). In the subset of 152 tests suitable for activity scoring, automatic and manual scores showed substantial agreement (Kw = 0.76 [0.69, 0.83]). Autonomous recognition and scoring of home-based functional activities is enabled by a multimodal movement monitoring system that integrates inertial measurement unit and vision data. Further algorithm training with ecologically valid data and a kitted system that is independently usable by patients are needed before fully autonomous, functional movement assessment is realizable. Functional movement assessments that can be administered in the home without a clinician present have the potential to democratize these evaluations and improve care access.

Self-attention deep ConvLSTM with sparse-learned channel dependencies for wearable sensor-based human activity recognition

In this study, we propose a novel deep-learning architecture with sparse learning for human activity recognition. The proposed model contains 1D CNNs and LSTM layers with a self-attention mechanism to enhance a substantial number of time points in time-series data for human activity recognition systems. Based on the recent success of squeeze-and-excite (SE) networks, the proposed deep learning model utilizes the SE module to enhance channel-wise interdependencies, which in turn leads to a boost in performance. In addition, we utilized sparse learning to retrain only weak nodes and freeze stronger nodes in a fully connected layer prior to classification layer. Furthermore, we utilized an entropy-inspired formula to find sparsely located weaker nodes and validated our model on various datasets, including Opportunity, UCI-HAR, and WISDM. Herein, we present an extensive analysis and survey of state-of-the-art studies, in addition to our proposed research. For a fair comparison, we evaluated our deep learning architecture using various performance metrics and achieved better results; the proposed model outperformed state-of-the-art algorithms for human activity recognition.

Motion Capture Algorithm for Students' Physical Activity Recognition in Physical Education Curriculum

Physical training learning is one of the important ways to raise the national physical quality and health level. However, there are many problems in traditional physical education, such as the difficulty to identifying the effectiveness of physical education curriculum and low level of repetitive exercise content. In order to solve this problem and improve the curriculum quality of current middle school physical education courses, a motion capture algorithm based on convolutional neural network and long-term memory network is proposed, and a student physical activity capture model is constructed based on the fusion algorithm. In the performance comparison test of the fusion algorithm proposed in this study, the loss value and accuracy of this fusion algorithm are 0.045 and 0.921, respectively, significantly superior to the comparison algorithm. Then in the empirical analysis, the accuracy rate of this motion capture algorithm model proposed in this study for students' walking posture recognition in physical education courses is 91.5%, which is better than the comparative capture method. This motion capture algorithm can accurately capture the physical activities of students in physical education courses, which has practical application significance.

Using complex networks and multiple artificial intelligence algorithms for table tennis match action recognition and technical-tactical analysis

This study developed two action recognition models using the YOLOv8-Alphapose two-stream spatial temporal graph convolutional networks (2s-STGCN), and the networks were used to recognize technical actions in table tennis. This study proposed a novel framework that merges dynamic and static complex network analysis with a community detection algorithm aimed at evaluating table tennis players' techniques, tactical patterns and styles. Two datasets that contain 8015 high-definition action videos of 37 elite players were constructed: a front-facing player technical action dataset (4154 videos) and a backwards-facing player technical action dataset (3861 videos). The results showed that YOLOv8-Alphapose-2s-STGCN achieved better recognition performance than seven other YOLOv8-Alphapose-based artificial intelligence algorithms (transformer, BiGRU, BiLSTM, GRU, LSTM, TCN and RNN algorithms) on both datasets and exhibited robust performance in practical applications. In the case study, multiple indicators were used to measure the importance of nodes (players' techniques) within the serving and receiving networks and within the two-round (winning and losing) networks. Dynamic complex network analysis was adopted to evaluate tactical styles and patterns. Furthermore, this study examined whether players and their opponents exhibit variability or similarity in their tactical patterns, focusing on the player networks and the two-round winning and losing networks. By integrating action recognition with process-focused match analysis, this study explored an innovative and comprehensive way to analyse matches, with implications for the performance analysis of table tennis players and players in related racket sports.

A Lightweight Graph Neural Network Algorithm for Action Recognition Based on Self-Distillation

Recognizing human actions can help in numerous ways, such as health monitoring, intelligent surveillance, virtual reality and human–computer interaction. A quick and accurate detection algorithm is required for daily real-time detection. This paper first proposes to generate a lightweight graph neural network by self-distillation for human action recognition tasks. The lightweight graph neural network was evaluated on the NTU-RGB+D dataset. The results demonstrate that, with competitive accuracy, the heavyweight graph neural network can be compressed by up to 80%. Furthermore, the learned representations have denser clusters, estimated by the Davies–Bouldin index, the Dunn index and silhouette coefficients. The ideal input data and algorithm capacity are also discussed.

Enabling collaborative assembly between humans and robots using a digital twin system

Human–robot collaboration (HRC) systems are intelligent systems that guide robots to collaborate with humans based on a cognitive understanding of human intention, ensuring safe, flexible, and efficient collaboration between humans and robots in shared workspaces. In industrial settings, the current methods for constructing a human digital twin model rely on motion capture devices that require personnel to wear cumbersome equipment, which goes against the principle of flexible interaction advocated for HRC. Furthermore, the current methods do not model humans and robots in a unified space, which is both unintuitive and inconvenient for perceiving and understanding the overall environment. To address these limitations, this paper proposes a digital twin system for HRC. This system facilitates the construction of a digital twin scene, the mapping from the real space to the virtual space, and the planning and execution of collaborative strategies from the virtual to the real space. Designed explicitly for common workstation settings, a robust human mesh recovery algorithm is introduced to address the challenge of reconstructing occluded human bodies. Additionally, uncertainty estimation is employed to enhance the action recognition algorithm, ensuring a controllable level of risk in the recognition process. Experimental results demonstrate the superiority of the proposed methods over baseline methods. Finally, the feasibility and effectiveness of the HRC system are validated through a case study involving component assembly.

A novel algorithm for human action recognition in compressed domain using attention-guided approach

A novel algorithm for human action recognition in compressed domain using attention-guided approach

Design and implementation of adolescent health Latin dance teaching system under artificial intelligence technology.

Since various dance teaching systems have attracted much attention with the development of Artificial Intelligence (AI) technology, this paper improves the recognition performance of Latin dance teaching systems by optimizing the action recognition model. Firstly, the object detection and action recognition technology under the current AI technology is analyzed, and the Two-stage object detection algorithm and One-stage object detection algorithm are evaluated. Secondly, the technologies and functions contained in the adolescent health Latin dance teaching system are described, including image acquisition, feature extraction, object detection, and action recognition. Finally, the action recognition algorithm is optimized based on object detection, and the rationality and feasibility of the proposed algorithm are verified by experiments. The experimental results show that the optimization algorithm can search the optimal feature subset after five iterations on Undefine Classes of 101 (UCF101) dataset, but it needs seven iterations on Human Motion Database 51 (HMDB51) dataset. Meanwhile, when using support vector machine classifier, the optimization algorithm can achieve the highest accuracy of motion recognition. Regressive Function, Multinomial Naive Bayes and Gaussian Naive Bayes Algorithms have lower prediction delay, as low as 0.01s. Therefore, this paper has certain reference significance for the design and implementation of adolescent health Latin dance teaching system.