Human Activity Recognition System Research Articles

Human activity recognition method using joint deep learning and acceleration signal

Many studies have been conducted on human activity recognition (HAR) in the last decade. Accordingly, deep learning (DL) algorithms have been given more attention in terms of classification of human daily activities. Deep neural networks (DNNs) compute and extract complex features on voluminous data through some hidden layers that require large memory and powerful graphics processing units (GPUs). So, this study proposes a new joint learning (JL) approach to classify human activities using inertial sensors. To this end, a large complex donor model based on a convolutional neural network (CNN) is used to transfer knowledge to a smaller model based on CNN referred to as the acceptor model. The acceptor model can be deployed on mobile devices and low-power hardware due to decreased computing costs and memory consumption. The wireless sensor data mining (WISDM) dataset is used to test the proposed model. According to the experimental results, the HAR system based on the JL algorithm outperforms than other methods.

Using postural data and recurrent learning to monitor shoplifting activities in megastores

SummaryRecently, researchers have placed a great deal of emphasis on modeling activity patterns to better understand human behavior. Several approaches have been researched so far to develop automatic human activity recognition systems that infer detailed semantics from visual images, aiming to understand real human behavior patterns. However, there is still a need for a cost effective solution to distinguish human actions in the real‐world environment. With this encouragement, a novel approach is proposed to recognize shoplifting acts by examining the posture evidence of the human being. This approach begins by obtaining the two‐dimensional pose reflecting human's body joints as a skeleton from the recorded frames. Subsequently, a preprocessing step is used to preprocess skeleton data, which can handle the occlusion too. Postural feature generation is then used to extract pertinent features from such preprocessed skeletons. Finally, feature deduction is performed to downsize the derived features to a smaller dimension, and activity classification is performed on such reduced features to identify shoplifting behaviors in real time. A synthetic shoplifting dataset and real store recorded videos are used to conduct the experiments, the findings of which appear more promising than those obtained using other cutting‐edge methods, with an accuracy of 97.36% and 91.66% for synthesized and real store recorded inputs.

Optimal Deep Recurrent Neural Networks for IoT-enabled Human Activity Recognition in Elderly and Disabled Persons

Aging is related to a decrease in the ability to execute activities of day-to-day routine and decay in physical exercise, which affect mental and physical health. Elderly patients or people can depend on a human activity recognition (HAR) system, which monitors the activity interventions and patterns if any critical event or behavioral changes occur. A HAR system incorporated with the Internet of Things (IoT) environment might allow these people to live independently. While the number of groups of activities and sensor measurements is enormous, the HAR problem could not be resolved deterministically. Hence, machine learning (ML) algorithm was broadly applied for the advancement of the HAR system to find the patterns of human activity from the sensor data. Therefore, this study presents an Optimal Deep Recurrent Neural Networks for Human Activity Recognition (ODRNN-HAR) on Elderly and Disabled Persons technique in the IoT platform. The intension of the ODRNN-HAR approach lies in the recognition and classification of various kinds of human activities in the IoT environment. Primarily, the ODRNN-HAR technique enables IoT devices to collect human activity data and employs Z-score normalization as a preprocessing step. For effectual recognition of human activities, the ODRNN-HAR technique uses the DRNN model. At the final stage, the optimal hyperparameter adjustment of the DRNN model takes place using the mayfly optimization (MFO) algorithm. The result analysis of the ODRNN-HAR algorithm takes place on benchmark HAR dataset, and the outcomes are examined. The comprehensive simulation outcomes highlighted the improved recognition results of the ODRNN-HAR approach in terms of different measures.

C-DTW for Human Action Recognition Based on Nanogenerator.

Sensor-based human action recognition (HAR) is considered to have broad practical prospects. It applies to wearable devices to collect plantar pressure or acceleration information at human joints during human actions, thereby identifying human motion patterns. Existing related works have mainly focused on improving recognition accuracy, and have rarely considered energy-efficient management of portable HAR systems. Considering the high sensitivity and energy harvesting ability of triboelectric nanogenerators (TENGs), in this research a TENG which achieved output performance of 9.98 mW/cm2 was fabricated using polydimethylsiloxane and carbon nanotube film for sensor-based HAR as a wearable sensor. Considering real-time identification, data are acquired using a sliding window approach. However, the classification accuracy is challenged by quasi-periodic characteristics of the intercepted sequence. To solve this problem, compensatory dynamic time warping (C-DTW) is proposed, which adjusts the DTW result based on the proportion of points separated by small distances under DTW alignment. Our simulation results show that the classification accuracy of C-DTW is higher than that of DTW and its improved versions (e.g., WDTW, DDTW and softDTW), with almost the same complexity. Moreover, C-DTW is much faster than shapeDTW under the same classification accuracy. Without loss of generality, the performance of the existing DTW versions can be enhanced using the compensatory mechanism of C-DTW.

A Smartphone-Based sEMG Signal Analysis System for Human Action Recognition.

In lower-limb rehabilitation, human action recognition (HAR) technology can be introduced to analyze the surface electromyography (sEMG) signal generated by movements, which can provide an objective and accurate evaluation of the patient's action. To balance the long cycle required for rehabilitation and the inconvenient factors brought by wearing sEMG devices, a portable sEMG signal acquisition device was developed that can be used under daily scenarios. Additionally, a mobile application was developed to meet the demand for real-time monitoring and analysis of sEMG signals. This application can monitor data in real time and has functions such as plotting, filtering, storage, and action capture and recognition. To build the dataset required for the recognition model, six lower-limb motions were developed for rehabilitation (kick, toe off, heel off, toe off and heel up, step back and kick, and full gait). The sEMG segment and action label were combined for training a convolutional neural network (CNN) to achieve high-precision recognition performance for human lower-limb actions (with a maximum accuracy of 97.96% and recognition accuracy for all actions reaching over 97%). The results show that the smartphone-based sEMG analysis system proposed in this paper can provide reliable information for the clinical evaluation of lower-limb rehabilitation.

Nested Binary Classifier as an Outlier Detection Method in Human Activity Recognition Systems.

The present article is devoted to outlier detection in phases of human movement. The aim was to find the most efficient machine learning method to detect abnormal segments inside physical activities in which there is a probability of origin from other activities. The problem was reduced to a classification task. The new method is proposed based on a nested binary classifier. Test experiments were then conducted using several of the most popular machine learning algorithms (linear regression, support vector machine, k-nearest neighbor, decision trees). Each method was separately tested on three datasets varying in characteristics and number of records. We set out to evaluate the effectiveness of the models, basic measures of classifier evaluation, and confusion matrices. The nested binary classifier was compared with deep neural networks. Our research shows that the method of nested binary classifiers can be considered an effective way of recognizing outlier patterns for HAR systems.

Human Activity Prediction Based on Forecasted IMU Activity Signals by Sequence-to-Sequence Deep Neural Networks.

Human Activity Recognition (HAR) has gained significant attention due to its broad range of applications, such as healthcare, industrial work safety, activity assistance, and driver monitoring. Most prior HAR systems are based on recorded sensor data (i.e., past information) recognizing human activities. In fact, HAR works based on future sensor data to predict human activities are rare. Human Activity Prediction (HAP) can benefit in multiple applications, such as fall detection or exercise routines, to prevent injuries. This work presents a novel HAP system based on forecasted activity data of Inertial Measurement Units (IMU). Our HAP system consists of a deep learning forecaster of IMU activity signals and a deep learning classifier to recognize future activities. Our deep learning forecaster model is based on a Sequence-to-Sequence structure with attention and positional encoding layers. Then, a pre-trained deep learning Bi-LSTM classifier is used to classify future activities based on the forecasted IMU data. We have tested our HAP system for five daily activities with two tri-axial IMU sensors. The forecasted signals show an average correlation of 91.6% to the actual measured signals of the five activities. The proposed HAP system achieves an average accuracy of 97.96% in predicting future activities.

Enhancing human activity recognition using features reduction in IoT edge and Azure cloud

The Internet of Things (IoT), cloud computing, and machine learning opened an opportunity for new smart systems. These technologies have triggered huge traffic and delay by continuously transmitting telemetry data to the cloud. IoT edge choice made decision-making closer to the environment, which decreases traffic and speeds up response time. Human activity recognition (HAR) systems, among other IoT applications, require systems with quick response time; reduce costs using constrained traffic to the cloud while maintaining accurate prediction results. This study proposes an application of HAR for predicting activities using up to three smartphone accelerometers. Three models are developed, trained, and deployed to achieve the necessary accuracy at the IoT edge and in the cloud, with an acceptable response time. Since each coordinate value from the three accelerometers has different importance in activity category prediction, focusing on the data from the most related values can help minimize the amount of information transferred from the edge to the cloud. Six models were trained in the cloud; three were deployed and tested at the edge with different features by selecting the most important ones using Principal Component Analysis (PCA). Different experiments showed that traffic and processing time decreased significantly based on the time required to predict HAR categories with acceptable accuracy. Since there is significant latency between the edge and the cloud and within the cloud, sending samples for verification save bandwidth, and processing requests locally at the edge speed up predictions. Results illustrate that the time required to serve one request from the environment where smartphones generate traffic through the internet connection to the cloud took about 5.8 s on average, including transmission delays and the prediction process. During this time, the model at the edge can serve 150 requests with the same accuracy using nine features. In addition, the edge can serve 286 requests in 5.8 s with 94.8 % accuracy when choosing the top four features at the edge.

Knowledge Mapping of Human Activity Recognition Techniques for Assistive Living

Purpose: Human Activity Recognition (HAR) is a subject of research that identifies an individual’s activities for assistive living. The proliferation of ICT and sensor technology prompted HAR to flourish beyond unfathomable levels, having immense human-centric applications. The development of accurate HAR systems involves complex statistical and computational tasks from signal acquisition to activity classification. This research aims to conduct a systematic review of recent techniques proposed for each stage of HAR application development. Methodology: The review is conducted following Kitchenham principles, using Scopus and Web of Science databases. Firstly, research questions were formulated, followed by the search strategy definition. Based on assessment criteria, 193 papers are shortlisted and thoroughly analyzed to extract research- related information. Results: The techniques identified in 193 articles are comprehensively mapped from four aspects: data acquisition, data preprocessing and feature engineering, learning algorithm, and evaluation. Each technique is examined for its strengths and limitations to assist application developers in selecting the best one for their needs. The prevailing challenges and upcoming research opportunities are thoroughly explored. Conclusion: The ever-expanding literature in the field necessitated an update to the status of HAR literature. Compared to other reviews that focused on specific methods, fields of application, and datatypes, to the best of our understanding, this is the first evaluation of its kind that provides a broader mapping of HAR approaches. The findings of this analysis will provide researchers and newcomers in the field an up-to-date and holistic view of the complete body of work in this area.

A low-cost real-time IoT human activity recognition system based on wearable sensor and the supervised learning algorithms

Activity recognition systems can detect human physical activities to support the assessment of health conditions. Among approaches of activity recognition systems were researched and implemented, the wearable systems based on accelerometers and machine learning classifiers offer one of the most viable solutions. These systems are cheap, comfortable, easy to use, with high recognition accuracy. The major challenge in this classification problem is required directly performed in a low-performance microcontroller. In this manuscript, an optimal time frame of an activity, a feature set, and a simple machine learning model were proposed to build a low-cost and responsive recognition system in real-time. The proposed device was verified on both public data and our experiment data. An excellent recognition rate resulted in 99.2% on the recorded dataset for four critical daily activities (standing, sitting, running, and walking).

Reducing the Impact of Sensor Orientation Variability in Human Activity Recognition Using a Consistent Reference System

Sensor- orientation is a critical aspect in a Human Activity Recognition (HAR) system based on tri-axial signals (such as accelerations); different sensors orientations introduce important errors in the activity recognition process. This paper proposes a new preprocessing module to reduce the negative impact of sensor-orientation variability in HAR. Firstly, this module estimates a consistent reference system; then, the tri-axial signals recorded from sensors with different orientations are transformed into this consistent reference system. This new preprocessing has been evaluated to mitigate the effect of different sensor orientations on the classification accuracy in several state-of-the-art HAR systems. The experiments were carried out using a subject-wise cross-validation methodology over six different datasets, including movements and postures. This new preprocessing module provided robust HAR performance even when sudden sensor orientation changes were included during data collection in the six different datasets. As an example, for the WISDM dataset, sensors with different orientations provoked a significant reduction in the classification accuracy of the state-of-the-art system (from 91.57 ± 0.23% to 89.19 ± 0.26%). This important reduction was recovered with the proposed algorithm, increasing the accuracy to 91.46 ± 0.30%, i.e., the same result obtained when all sensors had the same orientation.

Orientation-Independent Human Activity Recognition Using Complementary Radio Frequency Sensing.

RF sensing offers an unobtrusive, user-friendly, and privacy-preserving method for detecting accidental falls and recognizing human activities. Contemporary RF-based HAR systems generally employ a single monostatic radar to recognize human activities. However, a single monostatic radar cannot detect the motion of a target, e.g., a moving person, orthogonal to the boresight axis of the radar. Owing to this inherent physical limitation, a single monostatic radar fails to efficiently recognize orientation-independent human activities. In this work, we present a complementary RF sensing approach that overcomes the limitation of existing single monostatic radar-based HAR systems to robustly recognize orientation-independent human activities and falls. Our approach used a distributed mmWave MIMO radar system that was set up as two separate monostatic radars placed orthogonal to each other in an indoor environment. These two radars illuminated the moving person from two different aspect angles and consequently produced two time-variant micro-Doppler signatures. We first computed the mean Doppler shifts (MDSs) from the micro-Doppler signatures and then extracted statistical and time- and frequency-domain features. We adopted feature-level fusion techniques to fuse the extracted features and a support vector machine to classify orientation-independent human activities. To evaluate our approach, we used an orientation-independent human activity dataset, which was collected from six volunteers. The dataset consisted of more than 1350 activity trials of five different activities that were performed in different orientations. The proposed complementary RF sensing approach achieved an overall classification accuracy ranging from 98.31 to 98.54%. It overcame the inherent limitations of a conventional single monostatic radar-based HAR and outperformed it by 6%.

Human Activity Recognition Using Attention-Mechanism-Based Deep Learning Feature Combination.

Human activity recognition (HAR) performs a vital function in various fields, including healthcare, rehabilitation, elder care, and monitoring. Researchers are using mobile sensor data (i.e., accelerometer, gyroscope) by adapting various machine learning (ML) or deep learning (DL) networks. The advent of DL has enabled automatic high-level feature extraction, which has been effectively leveraged to optimize the performance of HAR systems. In addition, the application of deep-learning techniques has demonstrated success in sensor-based HAR across diverse domains. In this study, a novel methodology for HAR was introduced, which utilizes convolutional neural networks (CNNs). The proposed approach combines features from multiple convolutional stages to generate a more comprehensive feature representation, and an attention mechanism was incorporated to extract more refined features, further enhancing the accuracy of the model. The novelty of this study lies in the integration of feature combinations from multiple stages as well as in proposing a generalized model structure with CBAM modules. This leads to a more informative and effective feature extraction technique by feeding the model with more information in every block operation. This research used spectrograms of the raw signals instead of extracting hand-crafted features through intricate signal processing techniques. The developed model has been assessed on three datasets, including KU-HAR, UCI-HAR, and WISDM datasets. The experimental findings showed that the classification accuracies of the suggested technique on the KU-HAR, UCI-HAR, and WISDM datasets were 96.86%, 93.48%, and 93.89%, respectively. The other evaluation criteria also demonstrate that the proposed methodology is comprehensive and competent compared to previous works.

Optimized deep learning vision system for human action recognition from drone images.

There are several benefits to constructing a lightweight vision system that is implemented directly on limited hardware devices. Most deep learning-based computer vision systems, such as YOLO (You Only Look Once), use computationally expensive backbone feature extractor networks, such as ResNet and Inception network. To address the issue of network complexity, researchers created SqueezeNet, an alternative compressed and diminutive network. However, SqueezeNet was trained to recognize 1000 unique objects as a broad classification system. This work integrates a two-layer particle swarm optimizer (TLPSO) into YOLO to reduce the contribution of SqueezeNet convolutional filters that have contributed less to human action recognition. In short, this work introduces a lightweight vision system with an optimized SqueezeNet backbone feature extraction network. Secondly, it does so without sacrificing accuracy. This is because that the high-dimensional SqueezeNet convolutional filter selection is supported by the efficient TLPSO algorithm. The proposed vision system has been used to the recognition of human behaviors from drone-mounted camera images. This study focused on two separate motions, namely walking and running. As a consequence, a total of 300 pictures were taken at various places, angles, and weather conditions, with 100 shots capturing running and 200 images capturing walking. The TLPSO technique lowered SqueezeNet's convolutional filters by 52%, resulting in a sevenfold boost in detection speed. With an F1 score of 94.65% and an inference time of 0.061 milliseconds, the suggested system beat earlier vision systems in terms of human recognition from drone-based photographs. In addition, the performance assessment of TLPSO in comparison to other related optimizers found that TLPSO had a better convergence curve and achieved a higher fitness value. In statistical comparisons, TLPSO surpassed PSO and RLMPSO by a wide margin.

Recognition of Approximate Motions of Human Based on Micro-Doppler Features

With the aging society approaching, the daily safety of the elderly has attracted more and more people’s attention, especially the elderly who live alone without anyone to take care of them. How to judge whether there is danger from the daily behavior of those who live alone and providing timely assistance is a research topic of health protection in the field of smart home care. Modern radar sensing technology and intelligent perception technology adopted in the domain of health care have started to gain significant interest, which can replace wearable sensors or cameras without feeling uncomfortable or having privacy issues. In this paper, some preliminary results were presented to develop an indoor monitoring system for human activity recognition using micro-Doppler features of radar signals. The experimental campaign involved five different with some similar activities of 10 subjects to be collected using a Doppler radar. A total of several texture features and numerical features are extracted from the spectrogram and time-frequency domain of measurements and applied three machine learning (ML) algorithms such as Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Decision Tree (D-Tree) for the precise classification of all five different motions. The results illustrated that the SVM classifier using the Radial Basis Function (RBF) kernel reaches the best results for all three feature sets, and the classification of mixed features can reach 93.2%, as well as 88.4% and 92.4% for numerical and texture features. The performance of SVM is 1.6% better than KNN and 3.2% better than D-Tree.

Multilevel Privacy Preservation Scheme Based on Compressed Sensing

Although the extensive application of the Internet of Things brings a great convenience, it raises the concern of privacy leakage in the processes of data acquisition, analyzing and sharing as well. In this paper, a multilevel privacy protection via compressed sensing (CS) is proposed, which has the advantages of compressed sampling, protection of data privacy, and controllability of data access. At the data acquisition end, the CS technique suitable for a resource-constrained environment is employed to sample and encrypt signals with the assistance of discriminant component analysis (DCA). Then, the encrypted data will be transmitted to the cloud in time. On the cloud service, signals protected by CS rarely disclose private information to malicious attackers, and they will be accessed by two-class authorized entities. One is the semi-authorized user with low privilege who can only get the features from encrypted data for the subsequent inference; the other is the full-authorized user who is capable of reconstructing the original data. We demonstrate the scheme through two case studies of a face recognition system and a human activity recognition system and analyze its performance.

Online continual learning for human activity recognition

Sensor-based human activity recognition (HAR), with the ability to recognise human activities from wearable or embedded sensors, has been playing an important role in many applications including personal health monitoring, smart home, and manufacturing. The real-world, long-term deployment of these HAR systems drives a critical research question: how to evolve the HAR model automatically over time to accommodate changes in an environment or activity patterns. This paper presents an online continual learning (OCL) scenario for HAR, where sensor data arrives in a streaming manner which contains unlabelled samples from already learnt activities or new activities. We propose a technique, OCL-HAR, making a real-time prediction on the streaming sensor data while at the same time discovering and learning new activities. We have empirically evaluated OCL-HAR on four third-party, publicly available HAR datasets. Our results have shown that this OCL scenario is challenging to state-of-the-art continual learning techniques that have significantly underperformed. Our technique OCL-HAR has consistently outperformed them in all experiment setups, leading up to 0.17 and 0.23 improvements in micro and macro F1 scores.

Implementation Paper on Human Activity Recognition for Military Restricted Areas

Abstract: HAR is a field of study that aims to develop algorithms and techniques for automatically identifying and classifying human activities. Different people can execute an activity in various ways, and the same action might have numerous versions. Modern human activity recognition systems are largely taught and utilised on video stream and picture data in order to identify the features and activities, variations in the data having similar or related movements. Human-to-human and human-computer contact both heavily rely on human activity recognition. Systems that are manually operated take much longer and cost more money. The goal of this project is to develop a quicker, more affordable system for recognizing human activity and detect threats proactively through live camera. This system will help the end user in a variety of applications, such as surveillance and assistance in examination centres, hospitals, restricted areas, banks etc. by identifying the activity being carried out in the video or picture and give out alerts if detected any suspicious activity. Not only will this system be affordable, but it also functions as a utility-based system that can be integrated into a variety of applications to speed up and assist with various tasks that require recognition, which will result in significant time savings and high accuracy.

Exploiting Security Issues in Human Activity Recognition Systems (HARSs)

Human activity recognition systems (HARSs) are vital in a wide range of real-life applications and are a vibrant academic research area. Although they are adopted in many fields, such as the environment, agriculture, and healthcare and they are considered assistive technology, they seem to neglect the aspects of security and privacy. This problem occurs due to the pervasive nature of sensor-based HARSs. Sensors are devices with low power and computational capabilities, joining a machine learning application that lies in a dynamic and heterogeneous communication environment, and there is no generalized unified approach to evaluate their security/privacy, but rather only individual solutions. In this work, we studied HARSs in particular and tried to extend existing techniques for these systems considering the security/privacy of all participating components. Initially, in this work, we present the architecture of a real-life medical IoT application and the data flow across the participating entities. Then, we briefly review security and privacy issues and present possible vulnerabilities of each system layer. We introduce an architecture over the communication layer that offers mutual authentication, solving many security and privacy issues, particularly the man-in-the-middle attack (MitM). Relying on the proposed solutions, we manage to prevent unauthorized access to critical information by providing a trustworthy application.

Privacy-preserving activity recognition using multimodal sensors in smart office

Developing a human activity recognition (HAR) system for employees is essential to incorporate intelligence into smart office environments, enabling various human-centered applications to enhance employees’ well-being. Although remarkable progress has been made for the HAR in the smart office, several issues still exist, including lacking a privacy-preserving and unobtrusive method and demanding enhanced generalization performance across users. Therefore, a novel privacy-preserving HAR method based on multimodal sensors is investigated in this study, containing an infrared array sensor, a sensing chair, and a triaxial accelerometer built in a smartphone. The effectiveness of different multimodal combinations is examined. Moreover, a deep learning model is developed for multimodal data fusion to enhance the generalization performances across users. The model contains a residual 3D convolutional neural network (CNN) and 1D CNN for learning spatial–temporal feature representation of different modalities. Additionally, external memory units and an adaptive decision fusion operation are utilized for multimodal data fusion. Finally, extensive experiments are conducted to examine the performance of the proposed model using a self-collected dataset and the leave-one-subject-out cross-validation approach. The results verify the effectiveness of the proposed model.