Approach For Action Recognition Research Articles

Human Action Recognition Based on Skeleton Information and Multi-Feature Fusion

Action assessment and feedback can effectively assist fitness practitioners in improving exercise benefits. In this paper, we address key challenges in human action recognition and assessment by proposing innovative methods that enhance performance while reducing computational complexity. Firstly, we present Oct-MobileNet, a lightweight backbone network, to overcome the limitations of the traditional OpenPose algorithm’s VGG19 network, which exhibits a large parameter size and high device requirements. Oct-MobileNet employs octave convolution and attention mechanisms to improve the extraction of high-frequency features from the human body contour, resulting in enhanced accuracy with reduced model computational burden. Furthermore, we introduce a novel approach for action recognition that combines skeleton-based information and multiple feature fusion. By extracting spatial geometric and temporal characteristics from actions, we employ a sliding window algorithm to integrate these features. Experimental results show the effectiveness of our approach, demonstrating its ability to accurately recognize and classify various human actions. Additionally, we address the evaluation of traditional fitness exercises, specifically focusing on the BaDunJin movements. We propose a multimodal information-based assessment method that combines pose detection and keypoint analysis. Label sequences are obtained through a pose detector and each frame’s keypoint coordinates are represented as pose vectors. Leveraging multimodal information, including label sequences and pose vectors, we explore action similarity and perform quantitative evaluations to help exercisers assess the quality of their exercise performance.

Similarity Segmentation Approach for Sensor-Based Activity Recognition

Similarity Segmentation Approach for Sensor-Based Activity Recognition

Spatio-Temporal Adaptive Network With Bidirectional Temporal Difference for Action Recognition

Action Recognition is a fundamental task in computer vision field, with a wide range of applications in autonomous driving, security monitoring, etc. However, previous action recognition approaches usually suffer from the inappropriate spatio-temporal modeling or high computational consumption (e.g., 3D CNN). In this paper, we propose a novel Spatio-Temporal Adaptive Network (STANet) with bidirectional temporal difference, consisting of a Temporal Adaptive module (TA) and a Spatial Adaptive (SA) module, to sufficiently extract the crucial motion information and model the spatial pivotal appearance information from both forward and backward perspectives, respectively. Specifically, the Temporal Adaptive module uses bidirectional temporal differences to learn valuable motion trends and balance the static semantics and dynamic motion for a certain action during information fusion; while the Spatial Adaptive module uses the bidirectional temporal difference to obtain the spatio-channel attention to stress the discriminative position-relevant and semantic-relevant appearance features. Extensive experiments conducted on widely-used action recognition benchmarks UCF-101, HMDB-51, Something-Something V1, and Kinetics-400 prove the effectiveness of the proposed methods compared to other state-of-the-art approaches.

Multi-Camera-Based Human Activity Recognition for Human-Robot Collaboration in Construction.

As the use of construction robots continues to increase, ensuring safety and productivity while working alongside human workers becomes crucial. To prevent collisions, robots must recognize human behavior in close proximity. However, single, or RGB-depth cameras have limitations, such as detection failure, sensor malfunction, occlusions, unconstrained lighting, and motion blur. Therefore, this study proposes a multiple-camera approach for human activity recognition during human-robot collaborative activities in construction. The proposed approach employs a particle filter, to estimate the 3D human pose by fusing 2D joint locations extracted from multiple cameras and applies long short-term memory network (LSTM) to recognize ten activities associated with human and robot collaboration tasks in construction. The study compared the performance of human activity recognition models using one, two, three, and four cameras. Results showed that using multiple cameras enhances recognition performance, providing a more accurate and reliable means of identifying and differentiating between various activities. The results of this study are expected to contribute to the advancement of human activity recognition and utilization in human-robot collaboration in construction.

Perceiving local relative motion and global correlations for weakly supervised group activity recognition

This paper presents a weakly supervised approach for group activity recognition by exploiting the local relative motion and global correlations among entities. Most existing approaches of group recognition characterize motions of group activities based on the coordinates of individuals or feature maps without excluding the camera motion, which is a combination of local relative motion and camera motion. To address this problem, we utilize a simple but effective Local Relative Motion Module (LRMM): a 3D-CNN-based network to exploit the local movement. We further employ a Global Correlation Module (GCM) to establish relationships among different feature patches for capturing the entire scene. We have evaluated the proposed method on sports and group activity video. The method has achieved state-of-the-art performance on three challenging datasets for weakly supervised group activity recognition. The method has also outperformed some approaches trained with much stronger supervision in the comparative evaluation.

Multi-person Action Recognition Based on the HARNet Network in the Park Environment

Action recognition is crucial to public security, playing an important role in pedestrian protection. Recently, numerous approaches based on deep learning models have been reported for action recognition, which mainly has problems such as poor versatility, not good robustness, low recognition effects for low resolution, and occlusion in outdoor environments. This paper addresses this issue and aims to develop an approach for multi-person action recognition in the park environment. Frame extraction operation is first used to extract images from surveillance video in real-time, then YOLOv5 is adopted to detect and segment human targets from images. Thirdly, the AlphaPose algorithm is used to estimate the keypoint heatmap of the human body according to human targets. Finally, the heatmaps are preprocessed and then fused and classified by deep learning networks to complete the multi-person action recognition. In addition, an optimization method, i.e., Heatmap-based Action Recognition Network (HARNet), for ResNet is proposed in order to overcome the difficulty of information extraction from heatmaps, where the attention mechanism is considered. The approach presented here was tested on park surveillance video with multiple different scenarios, and the corresponding results verified its performance.

A multi-resolution fusion approach for human activity recognition from video data in tiny edge devices

Human Activity Recognition (HAR) is the process of automatic recognition of Activities of Daily Living (ADL) from human motion data captured in various data modalities by wearable and ambient sensors. Advances in Deep Learning, especially Convolutional Neural Networks (CNNs) and sequential models have revolutionalized HAR from video data sources. Although these models are incredibly accurate and utilize both spatial and temporal information, they require huge computation and memory resources — making them unsuitable for edge or wearable applications. Tiny Machine Learning (TinyML) aims to optimize these models in terms of compute and memory requirements – aiming to make them suitable for always-on resource constrained devices – leading to a reduction in communication latency and network traffic for HAR frameworks. In this paper, we propose a two-stream multi-resolution fusion architecture for HAR from video data modality. The context stream takes a resized image as input and the fovea stream takes the cropped center portion of the resized image as input, reducing the overall dimensionality. We tested two quantization methods: Post-Training Quantization (PTQ) and Quantization Aware Training (QAT) to optimize these models for deployment in edge devices and tested the performance in two challenging video datasets: KTH and UCF11. We performed ablation studies to validate the two-stream model performance. We deployed the proposed architecture in commercial resource constrained devices and monitored their performance in terms of inference latency and power consumption. The results indicate that the proposed architecture clearly outperforms other relevant single-stream models tested in this work in terms of accuracy, precision, recall, and F1-Score while also reducing the overall model size.

Dual-attention Network for View-invariant Action Recognition

View-invariant action recognition has been widely researched in various applications, such as visual surveillance and human–robot interaction. However, view-invariant human action recognition is challenging due to the action occlusions and information loss caused by view changes. Modeling spatiotemporal dynamics of body joints and minimizing representation discrepancy between different views could be a valuable solution for view-invariant human action recognition. Therefore, we propose a Dual-Attention Network (DANet) aims to learn robust video representation for view-invariant action recognition. The DANet is composed of relation-aware spatiotemporal self-attention and spatiotemporal cross-attention modules. The relation-aware spatiotemporal self-attention module learns representative and discriminative action features. This module captures local and global long-range dependencies, as well as pairwise relations among human body parts and joints in the spatial and temporal domains. The cross-attention module learns view-invariant attention maps and generates discriminative features for semantic representations of actions in different views. We exhaustively evaluate our proposed approach on the NTU-60, NTU-120, and UESTC large-scale challenging datasets with multi-type evaluation metrics including Cross-Subject, Cross-View, Cross-Set, and Arbitrary-view. The experimental results demonstrate that our proposed approach significantly outperforms state-of-the-art approaches in view-invariant action recognition.

Sensing within Smart Buildings: A Survey

Increasingly, buildings are being fitted with sensors for the needs of different sectors, such as education, industry and business. Using Internet of Things devices combined with analysis of data being generated by these devices, it is possible to infer a number of metrics, e.g., building occupancy and activities of occupants. The information thus gathered can be used to develop software applications to support energy management, occupant comfort, and space utilization. This survey explores the use of sensors in smart building environments, identifying different approaches to employ sensors in buildings. The most commonly used data-driven approaches for activity recognition in such buildings is also investigated, concluding by highlighting current research challenges and future research directions in this area.

System-Identification-Based Activity Recognition Algorithms With Inertial Sensors.

This paper focuses on activity recognition using a single wearable inertial measurement sensor placed on the subject's chest. The ten activities that need to be identified include lying down, standing, sitting, bending and walking, among others. The activity recognition approach is based on using and identifying a transfer function associated with each activity. The appropriate input and output signals for each transfer function are first determined based on the norms of the sensor signals excited by that specific activity. Then the transfer function is identified using training data and a Wiener filter based on the auto-correlation and cross-correlation of the output and input signals. The activity occurring in real-time is recognized by computing and comparing the input-output errors associated with all the transfer functions. The performance of the developed system is evaluated using data from a group of Parkinson's disease subjects, including data obtained in a clinical setting and data obtained through remote home monitoring. On average, the developed system provides better than 90% accuracy in identifying each activity as it occurs. Activity recognition is particularly useful for PD patients in order to monitor their level of activity, characterize their postural instability and recognize high risk-activities in real-time that could lead to falls.

SkateboardAI: The Coolest Video Action Recognition for Skateboarding (Student Abstract)

Impressed by the coolest skateboarding sports program from 2021 Tokyo Olympic Games, we are the first to curate the original real-world video datasets "SkateboardAI" in the wild, even self-design and implement diverse uni-modal and multi-modal video action recognition approaches to recognize different tricks accurately. For uni-modal methods, we separately apply (1)CNN and LSTM; (2)CNN and BiLSTM; (3)CNN and BiLSTM with effective attention mechanisms; (4)Transformer-based action recognition pipeline. Transferred to the multi-modal conditions, we investigated the two-stream Inflated-3D architecture on "SkateboardAI" datasets to compare its performance with uni-modal cases. In sum, our objective is developing an excellent AI sport referee for the coolest skateboarding competitions.

A Review on Challenges in Telerehabilitation and Human Activity Recognition Approaches during Covid-19 Pandemic

Impact of COVID-19 pandemic is widespread imposing limitations on the healthcare services all over the world. Due to this pandemic, governments around the world have imposed restrictions that limit individual freedom and have enforced social distance to prevent the collapse of national health care systems. In such situation, to offer medical care and rehabilitation to the patients, Telerehabilitation (TR) is a promising way of delivering healthcare facilities remotely using telecommunication and internet. Technological advancement has played the vital role to establish this TR technology to remotely assess patient’s physical condition and act accordingly during this pandemic. Likewise, Human Activity Recognition (HAR) is a key part of the recovery process for a wide variety of conditions, such as stroke, arthritis, brain injury, musculoskeletal injuries, Parkinson’s disease, and others. Different approaches of human activity recognition can be utilized to monitor the health and activity levels of such a patient effectively and TR allows to do this remotely. Therefore, in situations where conventional care is inadequate, combination of telerehabilitation and HAR approaches can be an effective means of providing treatment and these opportunities have become patently apparent during the COVID-19 outbreak. However, this new era of technical progress has significant limitations, and in this paper, our main focus is on the challenges of telerehabilitation and the various human activity recognition approaches. This study will help researchers identify a good activity detection platform for a TR system during and after COVID-19, considering TR and HAR challenges.

Cross-domain few-shot action recognition with unlabeled videos

Current few-shot action recognition approaches have achieved impressive performance using only a few labeled examples. However, they usually assume the base (train) and target (test) videos typically come from the same domain, which may limit their further applications. In this paper, we introduce a new practical task, termed as cross-domain few-shot action recognition, and hypothesize there is a domain shift between the base and target videos and the unlabeled target videos are available. To address this task, we further propose a Self-supervised learning Enhanced tEmporal Network (SEEN), which incorporates temporal modeling and self-supervised learning techniques to learn more transferable representations. Concretely, the temporal modeling mechanism aims to learn long-range temporal semantics from the features output by the backbone, and the self-supervised learning focuses on exploring the underlying data patterns to reduce domain shifts under the few-shot setting, which can help to improve the generalization ability. Therefore, the proposed SEEN can capture broader variations of the feature distributions and is more appropriate for the cross-domain few-shot action recognition task. Extensive experiments on multiple cross-domain benchmarks show that our SEEN consistently outperforms several strong baseline methods by a convincing margin.

A Fine-Tuning Based Approach for Daily Activity Recognition between Smart Homes

Daily activity recognition between different smart home environments faces some challenges, such as an insufficient amount of data and differences in data distribution. However, a deep network requires a large amount of labeled data for training. Additionally, inconsistent data distribution can lead to over-fitting of network learning. Additionally, the time cost of training the network from scratch is too high. In order to solve the above problems, this paper proposes a fine-tuning method suitable for daily activity recognition, which is the first application of this method in our field. Firstly, we unify the sensor space and activity space to reduce the variability in heterogeneous environments. Then, the Word2Vec algorithm is used to transform the activity samples into digital vectors recognizable by the network. Finally, the deep network is fine-tuned to transfer knowledge and complete the recognition task. Additionally, we try to train the network on public datasets. The results show that the network trained on a small dataset also has good transferability. It effectively improves the recognition accuracy and reduces the time cost and heavy data annotation.

Unsupervised Video-Based Action Recognition With Imagining Motion and Perceiving Appearance

Video-based action recognition is a challenging task, which demands carefully considering the temporal property of videos in addition to the appearance attributes. Particularly, the temporal domain of raw videos usually contains significantly more redundant or irrelevant information than still images. For that, this paper proposes an unsupervised video-based action recognition approach with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">imagining motion</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">perceiving appearance</i> , called IMPA, by comprehensively learning the spatio-temporal characteristics inherited in videos, with a particular emphasis on the moving object for action recognition. Specifically, a self-supervised <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Motion Extracting Block</i> (MEB) is designed to extract the principal motion features by focusing on the large movement of the moving object, based on the observation that humans can infer complete motion trajectories from partial moving objects. To further take the indispensable appearance attribute in videos into account, an unsupervised <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Appearance Learning Block</i> (ALB) is developed to perceive the static appearance, thus in combination with the MEB to recognize actions. Extensive validation experiments and ablation studies on multiple datasets demonstrate that our proposed IMPA approach obtains superior performance and surpasses other classical and state-of-the-art unsupervised action recognition methods.

A Review and Comparative Study of Explainable Deep Learning Models Applied on Action Recognition in Real Time

Video surveillance and image acquisition systems represent one of the most active research topics in computer vision and smart city domains. The growing concern for public and workers’ safety has led to a significant increase in the use of surveillance cameras that provide high-definition images and even depth maps when 3D cameras are available. Consequently, the need for automatic techniques for behavior analysis and action recognition is also increasing for several applications such as dangerous actions detection in railway stations or construction sites, event detection in crowd videos, behavior analysis, optimization in industrial sites, etc. In this context, several computer vision and deep learning solutions have been proposed recently where deep neural networks provided more accurate solutions, but they are not so efficient in terms of explainability and flexibility since they remain adapted for specific situations only. Moreover, the complexity of deep neural architectures requires the use of high computing resources to provide fast and real-time computations. In this paper, we propose a review and a comparative analysis of deep learning solutions in terms of precision, explainability, computation time, memory size, and flexibility. Experimental results are conducted within simulated and real-world dangerous actions in railway construction sites. Thanks to our comparative analysis and evaluation, we propose a personalized approach for dangerous action recognition depending on the type of collected data (image) and users’ requirements.

Extended Application of Inertial Measurement Units in Biomechanics: From Activity Recognition to Force Estimation.

Abnormal posture or movement is generally the indicator of musculoskeletal injuries or diseases. Mechanical forces dominate the injury and recovery processes of musculoskeletal tissue. Using kinematic data collected from wearable sensors (notably IMUs) as input, activity recognition and musculoskeletal force (typically represented by ground reaction force, joint force/torque, and muscle activity/force) estimation approaches based on machine learning models have demonstrated their superior accuracy. The purpose of the present study is to summarize recent achievements in the application of IMUs in biomechanics, with an emphasis on activity recognition and mechanical force estimation. The methodology adopted in such applications, including data pre-processing, noise suppression, classification models, force/torque estimation models, and the corresponding application effects, are reviewed. The extent of the applications of IMUs in daily activity assessment, posture assessment, disease diagnosis, rehabilitation, and exoskeleton control strategy development are illustrated and discussed. More importantly, the technical feasibility and application opportunities of musculoskeletal force prediction using IMU-based wearable devices are indicated and highlighted. With the development and application of novel adaptive networks and deep learning models, the accurate estimation of musculoskeletal forces can become a research field worthy of further attention.

HiTIM: Hierarchical Task Information Mining for Few-Shot Action Recognition

Although the existing few-shot action recognition methods have achieved impressive results, they suffer from two major shortcomings. (a) During feature extraction, few-shot tasks are not distinguished and task-irrelevant features are obtained, resulting in the loss of task-specific critical discriminative information. (b) During feature matching, information critical to the features within the task, i.e., self-information and mutual information, is ignored, resulting in the accuracy being affected by redundant or irrelevant information. To overcome these two limitations, we propose a hierarchical task information mining (HiTIM) approach for few-shot action recognition that incorporates two key components: an inter-task learner (Kinter) and an attention-matching module with an intra-task learner (Kintra). The purpose of the Kinter is to learn the knowledge of different tasks and build a task-related feature space for obtaining task-specific features. The proposed matching module with Kintra consists of two branches: the spatiotemporal self-attention matching (STM) and correlated cross-attention matching (CM), which reinforce key spatiotemporal information in features and mine regions with strong correlations between features, respectively. The shared Kintra can further optimize STM and CM. In our method, we can use either a 2D convolutional neural network (CNN) or 3D CNN as embedding. In comparable experiments using two different embeddings in the five-way one-shot and five-way five-shot task, the proposed method achieved recognition accuracy that outperformed other state-of-the-art (SOTA) few-shot action recognition methods on the HMDB51 dataset and was comparable to SOTA few-shot action recognition methods on the UCF101 and Kinetics datasets.

An efficient deep learning-based approach for human activity recognition using smartphone inertial sensors

Human activity recognition (HAR) has recently witnessed outstanding growth in health and entertainment applications. Owing to the availability of smartphones, many new methods and protocols for using the data from smartphones’ embedded sensors are emerging. Nonetheless, the methods carried out and published in the literature leave a wide area for improvement, in terms of accuracy, resource economy, and adaptation to real-world nuisances. On top of that, a novel classification method that is more economical and efficient is proposed in this paper using both 1D convolutional neural network (1D-CNN) parameters and handcrafted temporal and frequency features with the proficiency of a multilayer perceptron neural network (MLP) classifier. The method proposed requires only tri-axial accelerometer data, allowing it to be deployed even into lower equipment devices; it was tested within the two well-known benchmark datasets: UCI-HAR and Uni-MIB SHAR. Experimental results yield a classification accuracy exceeding 99%, outperforming many of the methods recently shown in the literature.

A multimodal fusion enabled ensemble approach for human activity recognition in smart homes.

How to deal with multi-modality data from different types of devices is a challenging issue for accurate recognition of human activities in a smart environment. In this paper, we propose a multimodal fusion enabled ensemble approach. Firstly, useful features collected from Bluetooth beacons, binary sensors, and smart floor are extracted and presented by fuzzy logic based-method with variable-size temporal windows. Secondly, a group of support vector machine classifiers are used to perform the classification task. Finally, a weighted ensemble method is used to obtain the final prediction. Especially, by applying the geometric framework, we are able to obtain the optimal weights for the ensemble. The proposed approach is evaluated on the UJAmI dataset. The experimental results demonstrate the efficacy and robustness of the proposed method.