Human Activity Recognition System Research Articles

Subject review: Human activity recognition based on depth images

Human activity recognition (HAR) has becomes a well-known area of study for researchers due to its multiple possible applications. HAR systems are applied in many areas including supervisory nursing care, medical supervision, surveillance, human- machine interaction, gaming and entertainment, etc. However, developing a robust HAR system that incorporates a comprehensive and scalable set of feature spaces is a challenging task. Much research has been done in this area to develop efficient HAR systems. In this article, a review of the developments in Human Activity Recognition (HAR) will be presented. Human activity recognition works on two basic principles; feature extraction from sensor data and classification of features into action classes. The review presented in this article is hence divided into two parts. The first part deals with feature representation in HAR systems such as Space-time Features, Frequency Features, Local Descriptors, Optical Flow Based, and Skeleton Joints. The second part deals with the classification approaches used in the literature such as Bayesian, and HMM.

AutoDLAR: A Semi-supervised Cross-modal Contact-free Human Activity Recognition System

WiFi-based human activity recognition (HAR) plays an essential role in various applications such as security surveillance, health monitoring, and smart home. Existing HAR methods, though yielding promising performance in indoor scenarios, highly depend on a massive labeled dataset for training which is extremely difficult to acquire in practical applications. In this paper, we present an automatic data labeling and HAR system, termed AutoDLAR. Taking a semi-supervised cross-modal learning framework with a hybrid loss function as the core, AutoDLAR transfers rich visual information to automatically label WiFi signals for WiFi-based HAR. Specifically, we devise a lightweight and multi-view WiFi sensing model with a parallel feature embedding method to accurately identify activities and accelerate recognition speed. Then, we exploit the video data to fine-tune a well-established visual HAR model, generating effective pseudo-labels for guiding the WiFi model’s training. We also build a synchronized Video-WiFi dataset with seven types of human activities under different scenarios to enable training and validating the semi-supervised HAR system. Extensive experiments on our collected activity dataset and the emotion recognition benchmark demonstrate that AutoDLAR attains an average accuracy of over 95.89% without manual labeling and only spends the inference time of 3.35 ms, outperforming the state-of-the-art (SOTA) methods.

Resource-efficient, sensor-based human activity recognition with lightweight deep models boosted with attention

With their automatic feature extraction capabilities, deep learning models have become more widespread in sensor-based human activity recognition, particularly on larger datasets. However, their direct use on mobile and wearable devices is challenging due to the extensive resource requirements. Concurrently, attention-based models are emerging to improve recognition performance by dynamically emphasizing relevant parts of features and disregarding the irrelevant ones, particularly in the computer vision domain. This study introduces a novel application of attention mechanisms to smaller deep architectures, investigating whether smaller models can achieve comparable recognition performance to larger models in sensor-based human activity recognition systems while keeping resource usage at lower levels. For this purpose, we integrate the convolutional block attention module into a hybrid model, deep convolutional and long short-term memory network. Experiments are conducted using five public datasets in three model sizes: lightweight, moderate and original. The results show that applying attention to the lightweight model enables achieving similar recognition performances to the moderate-size model, and the lightweight model requires approximately 2–13 times fewer parameters and 3.5 times fewer flops. We also conduct experiments with sensor data at lower sampling rates and from fewer sensors attached to different body parts. The results show that attention improves recognition performance under lower sampling rates, as well as under higher sampling rates when model sizes are smaller, and mitigates the impact of missing data from one or more body parts, making the model more suitable for real-world sensor-based applications.

Multi-head CNN-based activity recognition and its application on chest-mounted sensor-belt

In recent years, a great deal of research has been done on the identification, monitoring, and classification of human activities. Human activity recognition (HAR) is a term commonly used to describe the automatic identification of physical activities. For activity recognition, there are primarily vision-based and sensor-based methods available. The computer vision-based method is generally effective in lab settings, but because of clutter, fluctuating light levels, and contrast, it may not perform well in real-world scenarios. Continuous monitoring and analysis of physiological signals obtained from heterogeneous sensors attached to an individual’s body is required to realise sensor-based HAR systems. Most of the previous research in human activity recognition (HAR) is biased along with feature engineering and pre-processing which requires a good amount of domain knowledge. Application-specific modelling and time-taking methods are involved in these approaches. In this work, the multi-head convolutional neural network-based human activity recognition framework is proposed where automatic feature extraction and classification are involved in the form of an end-to-end classification approach. Experiments of this approach are performed by taking raw wearable sensor data with few pre-processing steps and without the involvement of a handcrafted feature extraction technique. 99.23% and 93.55% accuracy are obtained on the WISDM and UCI-HAR datasets which denoted the much improvement in the assessment of HAR over other similar approaches. The model is also tested on locally collected data from a chest mounted belt with fabric sensors and an accuracy of 87.14% has been achieved on that data.

Real-world environmental enrichment rehabilitation paradigm in people with severe traumatic brain injury: a pilot feasibility study

ABSTRACT Background The use of Environmental Enrichment (EE) has been widely studied in animal models. However, the application of the same in humans is limited to rehabilitation settings. Objective To investigate the feasibility of a community-based EE paradigm in adults with brain injury. Methods Six individuals diagnosed with traumatic brain injury enrolled in the study. The Go Baby Go Café instrumented with a body weight harness system, provided physical and social enrichment as participants performed functional tasks for 2 hours, three times a week, for 2 months. Feasibility and safety outcomes were recorded throughout sessions. Clinical measures including 10-meter walk, timed up and go, jebsen hand function, 6-minute walk, and trail making tests were obtained pre and post intervention. Results All participants completed the study. The attendance was 100% and adherence was 87%. Positive changes in clinical measures were statistically significant for the timed up and go (p = 0.0175), TUG-cognitive (p = 0.0064), 10-meter walk (p = 0.0428), six-minute walk (p = 0.0196), TMT-A (p = 0.034). Changes in JHFT were not significant (p = 0.0506), with one subject recording values counter to the trend. Conclusion The Café was a comprehensive EE-based intervention that was feasible, safe, and has the potential to enhance motor and cognitive function in individuals with brain injury.

The Deep Learning-based Human Action Recognition System for Competitive Sports

The Deep Learning-based Human Action Recognition System for Competitive Sports

Fall Detection System based on iOS Smartphone Sensors

Activities of daily living (ADLs) are a crucial aspect of human life, especially in remote health monitoring and fall detection. As smartphones have become an integral part of our daily routines, with their ability to perform complex calculations, connect to the internet, and incorporate various sensors, researchers have been inspired to explore human activity recognition systems. This paper focuses on accelerometer and gyroscope data from iOS-based smartphones. We developed a data collection app to record fall types (e.g., Falling Right, Falling Left) and fall-like activities (e.g., Sitting Fast, Jumping). Volunteers carried smartphones naturally in their pockets during experiments, presenting a challenge of noise but enhancing user comfort. We applied different Machine Learning algorithms (Decision Trees, Random Forest, Logistic Regression, k-Nearest Neighbor, XGBoost, LightGBM, and Neural Networks) to analyze the collected dataset. In contrast to typical studies, our approach replicated real-world smartphone usage. The paper presents and analyzes promising results from the study. Furthermore, we implemented the trained model as a real-time mobile application for potential users. This research illustrates the potential of smartphones in fall detection and opens the way for user-friendly solutions in remote health monitoring.

Human activity recognition using a single-photon direct time-of-flight sensor

Single-Photon Avalanche Diode (SPAD) direct Time-of-Flight (dToF) sensors provide depth imaging over long distances, enabling the detection of objects even in the absence of contrast in colour or texture. However, distant objects are represented by just a few pixels and are subject to noise from solar interference, limiting the applicability of existing computer vision techniques for high-level scene interpretation. We present a new SPAD-based vision system for human activity recognition, based on convolutional and recurrent neural networks, which is trained entirely on synthetic data. In tests using real data from a 64×32 pixel SPAD, captured over a distance of 40 m, the scheme successfully overcomes the limited transverse resolution (in which human limbs are approximately one pixel across), achieving an average accuracy of 89% in distinguishing between seven different activities. The approach analyses continuous streams of video-rate depth data at a maximal rate of 66 FPS when executed on a GPU, making it well-suited for real-time applications such as surveillance or situational awareness in autonomous systems.

Novel hybrid optimization based adaptive deep convolution neural network approach for human activity recognition system

Novel hybrid optimization based adaptive deep convolution neural network approach for human activity recognition system

Human Activity Recognition Based on Deep Learning and Micro-Doppler Radar Data.

Activity recognition is one of the significant technologies accompanying the development of the Internet of Things (IoT). It can help in recording daily life activities or reporting emergencies, thus improving the user's quality of life and safety, and even easing the workload of caregivers. This study proposes a human activity recognition (HAR) system based on activity data obtained via the micro-Doppler effect, combining a two-stream one-dimensional convolutional neural network (1D-CNN) with a bidirectional gated recurrent unit (BiGRU). Initially, radar sensor data are used to generate information related to time and frequency responses using short-time Fourier transform (STFT). Subsequently, the magnitudes and phase values are calculated and fed into the 1D-CNN and Bi-GRU models to extract spatial and temporal features for subsequent model training and activity recognition. Additionally, we propose a simple cross-channel operation (CCO) to facilitate the exchange of magnitude and phase features between parallel convolutional layers. An open dataset collected through radar, named Rad-HAR, is employed for model training and performance evaluation. Experimental results demonstrate that the proposed 1D-CNN+CCO-BiGRU model demonstrated superior performance, achieving an impressive accuracy rate of 98.2%. This outperformance of existing systems with the radar sensor underscores the proposed model's potential applicability in real-world scenarios, marking a significant advancement in the field of HAR within the IoT framework.

A Simulation-Based Framework for the Design of Human Activity Recognition Systems Using Radar Sensors

A Simulation-Based Framework for the Design of Human Activity Recognition Systems Using Radar Sensors

Hybrid HAR-CNN Model: A Hybrid Convolutional Neural Network Model for Predicting and Recognizing the Human Activity Recognition

Human activity recognition (HAR) is an active research area in computer vision from past several years and research is still continuing in this field due to the unavailability of perfect recognition system. The human activity recognition system it covers e-health, patient monitoring, assistive daily living activities, video surveillance, security and behaviour analysis, and sports analysis. Many researchers have suggested techniques that use visual perception to detect human activities. Researchers will need to address problems including light variations in human activity detection, interclass similarity between scenes, the surroundings and recording setting, and temporal variation in order to construct an efficient vision-based human activity recognition system. However, a significant drawback of many deep learning models is their inability to achieve satisfactory results in real-world scenarios due to the conflicts mentioned above. To address this challenge, we developed a hybrid HAR-CNN classifier aimed at enhancing the learning outcomes of Deep CNNs by combining two models: Improved CNN and VGG-19. Using the KTH dataset, we collected 6,000 images for training, validation, and testing of our proposed technique. Our research findings indicate that the Hybrid HAR-CNN model, which combines Improved CNN with VGG-19 Net, outperforms individual deep learning models such as Improved CNN and VGG-19 Net.

Exploiting channel state information of WiFi signal for human activity detection: an experimental study

Ubiquitous computing aims to seamlessly integrate computing into our daily lives, and requires reliable information on human activities and state for various applications. In this paper, we propose a device-free human activity recognition system that leverages the rich information behind WiFi signals to detect human activities in indoor environments, including walking, sitting, and standing. The key idea of our system is to use the dynamic features of activities, which we carefully examine and analyze through the characteristics of channel state information. We evaluate the impact of location changes on WiFi signal distribution for different activities and design an activity detection system that employs signal processing techniques to extract discriminative features from wireless signals in the frequency and temporal domains. We implement our system on a single off-the-shelf WiFi device connecting to a commercial wireless access point and evaluate it in laboratory and conference room environments. Our experiments demonstrate the feasibility of using WiFi signals for device-free human activity recognition, which could provide a practical and non-intrusive solution for indoor monitoring and ubiquitous computing applications.

Feasibility and utility of in-home body weight support harness system use in young children treated for spinal muscular atrophy: A single-arm prospective cohort study.

This single-arm prospective cohort study aimed to evaluate the feasibility and utility of in-home body weight support harness system (BWSS) use in children treated for spinal muscular atrophy (SMA). Individuals with 2 or 3 copies of SMN2 who received pharmacotherapeutic treatment, had head control, and weight <50lbs were enrolled. Families were provided a BWSS and documented use. Motor outcome assessments were completed at baseline, month 3 and month 6. Families provided feedback in an end of study survey. All 32 participants (2.9 (SD 1.9) yrs), improved or remained stable on all outcomes. Average reported frequency of use was 4.1(2.3) hrs/week. Controlling for other covariates, frequency of use explained over 70% of the variability in change scores. Family feedback was overwhelmingly positive. Use of in-home BWSS is a safe, feasible and useful option to increase exercise dosage after treatment in SMA and may help optimize motor abilities. Study registered with: Clinicaltrials.gov Clinicaltrials.gov identifier: NCT05715749.

Real-Time Human Activity Recognition on Embedded Equipment: A Comparative Study

As living standards improve, the growing demand for energy, comfort, and health monitoring drives the increased importance of innovative solutions. Real-time recognition of human activities (HAR) in smart homes is of significant relevance, offering varied applications to improve the quality of life of fragile individuals. These applications include facilitating autonomy at home for vulnerable people, early detection of deviations or disruptions in lifestyle habits, and immediate alerting in the event of critical situations. The first objective of this work is to develop a real-time HAR algorithm in embedded equipment. The proposed approach incorporates the event dynamic windowing based on space-temporal correlation and the knowledge of activity trigger sensors to recognize activities in the case of a record of new events. The second objective is to approach the HAR task from the perspective of edge computing. In concrete terms, this involves implementing a HAR algorithm in a “home box”, a low-power, low-cost computer, while guaranteeing performance in terms of accuracy and processing time. To achieve this goal, a HAR algorithm was first developed to perform these recognition tasks in real-time. Then, the proposed algorithm is ported on three hardware architectures to be compared: (i) a NUCLEO-H753ZI microcontroller from ST-Microelectronics using two programming languages, C language and MicroPython; (ii) an ESP32 microcontroller, often used for smart-home devices; and (iii) a Raspberry-PI, optimizing it to maintain accuracy of classification of activities with a requirement of processing time, memory resources, and energy consumption. The experimental results show that the proposed algorithm can be effectively implemented on a constrained resource hardware architecture. This could allow the design of an embedded system for real-time human activity recognition.

PBVI for Optimal Photoplethysmography Noise Filter Selection Using Human Activity Recognition Observations for Improved Heart Rate Estimation on Multi-Sensor Systems

Abstract This work details the partially observable markov decision process (POMDP) and the point-based value iteration (PBVI) algorithms for use in multisensor systems, specifically, a sensor system capable of heart rate (HR) estimation through wearable photoplethysmography (PPG) and accelerometer signals. PPG sensors are highly susceptible to motion artifact (MA); however, current methods focus more on overall MA filters, rather than action specific filtering. An end-to-end embedded human activity recognition (HAR) System is developed to represent the observation uncertainty, and two action specific PPG MA reducing filters are proposed as actions. PBVI allows optimal action decision-making based on an uncertain observation, effectively balancing correct action choice and sensor system cost. Two central systems are proposed to accompany these algorithms, one for unlimited observation access and one for limited observation access. Through simulation, it can be shown that the limited observation system performs optimally when sensor cost is negligible, while limited observation access performs optimally when a negative reward for sensor use is considered. The final general framework for POMDP and PBVI was applied to a specific HR estimation example. This work can be expanded on and used as a basis for future work on similar multisensor system.

MP-HAR: A Novel Motion-Powered Real-Time Human Activity Recognition System

MP-HAR: A Novel Motion-Powered Real-Time Human Activity Recognition System

Human Activity Recognition System with Smartphones Using Machine Learning

Abstract: Human activity recognition (HAR) is a crucial task in the field of physical activity monitoring. It involves the identification of a person's activities and movements using sensors. The accuracy of HAR systems plays a significant role in enhancing physical health, preventing accidents and injuries, and improving security systems. One of the most common approaches to HAR is using smartphones as sensors. However, smartphones have limited processing power and battery life, which can impact the performance of HAR systems. To overcome these challenges, we can apply different machine learning models and compare their performances. We begin by selecting a standard dataset, the HAR dataset from the Machine Learning Repository. We then apply several Machine learning models, including Logistic Regression (LR), Decision Tree (DT), Support Vector Machine (SVM), Random Forest (RF), and Artificial Neural Network (ANN).To compare the performance of these models, we train and test each model on the HAR dataset. We also select the best set of parameters for each model using grid search. Our results show that the Support Vector Machine (SVM) performed the best (average accuracy 96.33%), significantly outperforming the other models. We can confirm the statistical significance of these results by employing statistical significance test methods.The SVM model demonstrated superior performance in recognizing human activities. This highlights the potential of machine learning models in revolutionizing the field of HAR. However, further research is needed to address the challenges of limited processing power and battery life in smartphones and to explore other potential applications of HAR systems.

The meaning of adapted ice-skating for children and youths with disabilities

Purpose Children and youths with disabilities have low levels of physical activity and limited opportunities to participate in physical activity. This study aimed to explore the lived experiences and meaning of an adapted ice-skating approach with access to a tailored harness system as a physical activity for children and youths with disabilities in Sweden. Materials and methods Semi-structured interviews were conducted with 12 guardians, 5 ice-skating trainers and 15 ice-skating children and youths with disabilities. The interviews were supplemented with participant observations of 23 children and youths with disabilities while ice-skating. Results Four categories emerged: “To grow as a person,” “To arouse zest for life,” “To train for life” and “To become part of society.” Ice-skating was found to be a source of joy of movement, zest for life and longing, enabling children and youths with disabilities to explore and develop their abilities, social interactions, confidence, identity, and independence in a safe and joy-bringing context. Conclusion The study highlights social and existential dimensions of an adapted ice-skating approach with access to a harness system tailored for children and youths with disabilities, that enables them to be included in ice-skating, regardless of disabilities, creating increased opportunities for physical activity and movement.

Hybrid time-spatial video saliency detection method to enhance human action recognition systems

AbstractSince digital media has become increasingly popular, video processing has expanded in recent years. Video processing systems require high levels of processing, which is one of the challenges in this field. Various approaches, such as hardware upgrades, algorithmic optimizations, and removing unnecessary information, have been suggested to solve this problem. This study proposes a video saliency map based method that identifies the critical parts of the video and improves the system’s overall performance. Using an image registration algorithm, the proposed method first removes the camera’s motion. Subsequently, each video frame’s color, edge, and gradient information are used to obtain a spatial saliency map. Combining spatial saliency with motion information derived from optical flow and color-based segmentation can produce a saliency map containing both motion and spatial data. A nonlinear function is suggested to properly combine the temporal and spatial saliency maps, which was optimized using a multi-objective genetic algorithm. The proposed saliency map method was added as a preprocessing step in several Human Action Recognition (HAR) systems based on deep learning, and its performance was evaluated. Furthermore, the proposed method was compared with similar methods based on saliency maps, and the superiority of the proposed method was confirmed. The results show that the proposed method can improve HAR efficiency by up to 6.5% relative to HAR methods with no preprocessing step and 3.9% compared to the HAR method containing a temporal saliency map.