Human Activity Recognition System Research Articles

Enhancing human activity recognition using features reduction in IoT edge and Azure cloud

The Internet of Things (IoT), cloud computing, and machine learning opened an opportunity for new smart systems. These technologies have triggered huge traffic and delay by continuously transmitting telemetry data to the cloud. IoT edge choice made decision-making closer to the environment, which decreases traffic and speeds up response time. Human activity recognition (HAR) systems, among other IoT applications, require systems with quick response time; reduce costs using constrained traffic to the cloud while maintaining accurate prediction results. This study proposes an application of HAR for predicting activities using up to three smartphone accelerometers. Three models are developed, trained, and deployed to achieve the necessary accuracy at the IoT edge and in the cloud, with an acceptable response time. Since each coordinate value from the three accelerometers has different importance in activity category prediction, focusing on the data from the most related values can help minimize the amount of information transferred from the edge to the cloud. Six models were trained in the cloud; three were deployed and tested at the edge with different features by selecting the most important ones using Principal Component Analysis (PCA). Different experiments showed that traffic and processing time decreased significantly based on the time required to predict HAR categories with acceptable accuracy. Since there is significant latency between the edge and the cloud and within the cloud, sending samples for verification save bandwidth, and processing requests locally at the edge speed up predictions. Results illustrate that the time required to serve one request from the environment where smartphones generate traffic through the internet connection to the cloud took about 5.8 s on average, including transmission delays and the prediction process. During this time, the model at the edge can serve 150 requests with the same accuracy using nine features. In addition, the edge can serve 286 requests in 5.8 s with 94.8 % accuracy when choosing the top four features at the edge.

Knowledge Mapping of Human Activity Recognition Techniques for Assistive Living

Purpose: Human Activity Recognition (HAR) is a subject of research that identifies an individual’s activities for assistive living. The proliferation of ICT and sensor technology prompted HAR to flourish beyond unfathomable levels, having immense human-centric applications. The development of accurate HAR systems involves complex statistical and computational tasks from signal acquisition to activity classification. This research aims to conduct a systematic review of recent techniques proposed for each stage of HAR application development. Methodology: The review is conducted following Kitchenham principles, using Scopus and Web of Science databases. Firstly, research questions were formulated, followed by the search strategy definition. Based on assessment criteria, 193 papers are shortlisted and thoroughly analyzed to extract research- related information. Results: The techniques identified in 193 articles are comprehensively mapped from four aspects: data acquisition, data preprocessing and feature engineering, learning algorithm, and evaluation. Each technique is examined for its strengths and limitations to assist application developers in selecting the best one for their needs. The prevailing challenges and upcoming research opportunities are thoroughly explored. Conclusion: The ever-expanding literature in the field necessitated an update to the status of HAR literature. Compared to other reviews that focused on specific methods, fields of application, and datatypes, to the best of our understanding, this is the first evaluation of its kind that provides a broader mapping of HAR approaches. The findings of this analysis will provide researchers and newcomers in the field an up-to-date and holistic view of the complete body of work in this area.

A low-cost real-time IoT human activity recognition system based on wearable sensor and the supervised learning algorithms

Activity recognition systems can detect human physical activities to support the assessment of health conditions. Among approaches of activity recognition systems were researched and implemented, the wearable systems based on accelerometers and machine learning classifiers offer one of the most viable solutions. These systems are cheap, comfortable, easy to use, with high recognition accuracy. The major challenge in this classification problem is required directly performed in a low-performance microcontroller. In this manuscript, an optimal time frame of an activity, a feature set, and a simple machine learning model were proposed to build a low-cost and responsive recognition system in real-time. The proposed device was verified on both public data and our experiment data. An excellent recognition rate resulted in 99.2% on the recorded dataset for four critical daily activities (standing, sitting, running, and walking).

Reducing the Impact of Sensor Orientation Variability in Human Activity Recognition Using a Consistent Reference System

Sensor- orientation is a critical aspect in a Human Activity Recognition (HAR) system based on tri-axial signals (such as accelerations); different sensors orientations introduce important errors in the activity recognition process. This paper proposes a new preprocessing module to reduce the negative impact of sensor-orientation variability in HAR. Firstly, this module estimates a consistent reference system; then, the tri-axial signals recorded from sensors with different orientations are transformed into this consistent reference system. This new preprocessing has been evaluated to mitigate the effect of different sensor orientations on the classification accuracy in several state-of-the-art HAR systems. The experiments were carried out using a subject-wise cross-validation methodology over six different datasets, including movements and postures. This new preprocessing module provided robust HAR performance even when sudden sensor orientation changes were included during data collection in the six different datasets. As an example, for the WISDM dataset, sensors with different orientations provoked a significant reduction in the classification accuracy of the state-of-the-art system (from 91.57 ± 0.23% to 89.19 ± 0.26%). This important reduction was recovered with the proposed algorithm, increasing the accuracy to 91.46 ± 0.30%, i.e., the same result obtained when all sensors had the same orientation.

Orientation-Independent Human Activity Recognition Using Complementary Radio Frequency Sensing.

RF sensing offers an unobtrusive, user-friendly, and privacy-preserving method for detecting accidental falls and recognizing human activities. Contemporary RF-based HAR systems generally employ a single monostatic radar to recognize human activities. However, a single monostatic radar cannot detect the motion of a target, e.g., a moving person, orthogonal to the boresight axis of the radar. Owing to this inherent physical limitation, a single monostatic radar fails to efficiently recognize orientation-independent human activities. In this work, we present a complementary RF sensing approach that overcomes the limitation of existing single monostatic radar-based HAR systems to robustly recognize orientation-independent human activities and falls. Our approach used a distributed mmWave MIMO radar system that was set up as two separate monostatic radars placed orthogonal to each other in an indoor environment. These two radars illuminated the moving person from two different aspect angles and consequently produced two time-variant micro-Doppler signatures. We first computed the mean Doppler shifts (MDSs) from the micro-Doppler signatures and then extracted statistical and time- and frequency-domain features. We adopted feature-level fusion techniques to fuse the extracted features and a support vector machine to classify orientation-independent human activities. To evaluate our approach, we used an orientation-independent human activity dataset, which was collected from six volunteers. The dataset consisted of more than 1350 activity trials of five different activities that were performed in different orientations. The proposed complementary RF sensing approach achieved an overall classification accuracy ranging from 98.31 to 98.54%. It overcame the inherent limitations of a conventional single monostatic radar-based HAR and outperformed it by 6%.

Human Activity Recognition Using Attention-Mechanism-Based Deep Learning Feature Combination.

Human activity recognition (HAR) performs a vital function in various fields, including healthcare, rehabilitation, elder care, and monitoring. Researchers are using mobile sensor data (i.e., accelerometer, gyroscope) by adapting various machine learning (ML) or deep learning (DL) networks. The advent of DL has enabled automatic high-level feature extraction, which has been effectively leveraged to optimize the performance of HAR systems. In addition, the application of deep-learning techniques has demonstrated success in sensor-based HAR across diverse domains. In this study, a novel methodology for HAR was introduced, which utilizes convolutional neural networks (CNNs). The proposed approach combines features from multiple convolutional stages to generate a more comprehensive feature representation, and an attention mechanism was incorporated to extract more refined features, further enhancing the accuracy of the model. The novelty of this study lies in the integration of feature combinations from multiple stages as well as in proposing a generalized model structure with CBAM modules. This leads to a more informative and effective feature extraction technique by feeding the model with more information in every block operation. This research used spectrograms of the raw signals instead of extracting hand-crafted features through intricate signal processing techniques. The developed model has been assessed on three datasets, including KU-HAR, UCI-HAR, and WISDM datasets. The experimental findings showed that the classification accuracies of the suggested technique on the KU-HAR, UCI-HAR, and WISDM datasets were 96.86%, 93.48%, and 93.89%, respectively. The other evaluation criteria also demonstrate that the proposed methodology is comprehensive and competent compared to previous works.

Optimized deep learning vision system for human action recognition from drone images.

There are several benefits to constructing a lightweight vision system that is implemented directly on limited hardware devices. Most deep learning-based computer vision systems, such as YOLO (You Only Look Once), use computationally expensive backbone feature extractor networks, such as ResNet and Inception network. To address the issue of network complexity, researchers created SqueezeNet, an alternative compressed and diminutive network. However, SqueezeNet was trained to recognize 1000 unique objects as a broad classification system. This work integrates a two-layer particle swarm optimizer (TLPSO) into YOLO to reduce the contribution of SqueezeNet convolutional filters that have contributed less to human action recognition. In short, this work introduces a lightweight vision system with an optimized SqueezeNet backbone feature extraction network. Secondly, it does so without sacrificing accuracy. This is because that the high-dimensional SqueezeNet convolutional filter selection is supported by the efficient TLPSO algorithm. The proposed vision system has been used to the recognition of human behaviors from drone-mounted camera images. This study focused on two separate motions, namely walking and running. As a consequence, a total of 300 pictures were taken at various places, angles, and weather conditions, with 100 shots capturing running and 200 images capturing walking. The TLPSO technique lowered SqueezeNet's convolutional filters by 52%, resulting in a sevenfold boost in detection speed. With an F1 score of 94.65% and an inference time of 0.061 milliseconds, the suggested system beat earlier vision systems in terms of human recognition from drone-based photographs. In addition, the performance assessment of TLPSO in comparison to other related optimizers found that TLPSO had a better convergence curve and achieved a higher fitness value. In statistical comparisons, TLPSO surpassed PSO and RLMPSO by a wide margin.

Recognition of Approximate Motions of Human Based on Micro-Doppler Features

With the aging society approaching, the daily safety of the elderly has attracted more and more people’s attention, especially the elderly who live alone without anyone to take care of them. How to judge whether there is danger from the daily behavior of those who live alone and providing timely assistance is a research topic of health protection in the field of smart home care. Modern radar sensing technology and intelligent perception technology adopted in the domain of health care have started to gain significant interest, which can replace wearable sensors or cameras without feeling uncomfortable or having privacy issues. In this paper, some preliminary results were presented to develop an indoor monitoring system for human activity recognition using micro-Doppler features of radar signals. The experimental campaign involved five different with some similar activities of 10 subjects to be collected using a Doppler radar. A total of several texture features and numerical features are extracted from the spectrogram and time-frequency domain of measurements and applied three machine learning (ML) algorithms such as Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Decision Tree (D-Tree) for the precise classification of all five different motions. The results illustrated that the SVM classifier using the Radial Basis Function (RBF) kernel reaches the best results for all three feature sets, and the classification of mixed features can reach 93.2%, as well as 88.4% and 92.4% for numerical and texture features. The performance of SVM is 1.6% better than KNN and 3.2% better than D-Tree.

Multilevel Privacy Preservation Scheme Based on Compressed Sensing

Although the extensive application of the Internet of Things brings a great convenience, it raises the concern of privacy leakage in the processes of data acquisition, analyzing and sharing as well. In this paper, a multilevel privacy protection via compressed sensing (CS) is proposed, which has the advantages of compressed sampling, protection of data privacy, and controllability of data access. At the data acquisition end, the CS technique suitable for a resource-constrained environment is employed to sample and encrypt signals with the assistance of discriminant component analysis (DCA). Then, the encrypted data will be transmitted to the cloud in time. On the cloud service, signals protected by CS rarely disclose private information to malicious attackers, and they will be accessed by two-class authorized entities. One is the semi-authorized user with low privilege who can only get the features from encrypted data for the subsequent inference; the other is the full-authorized user who is capable of reconstructing the original data. We demonstrate the scheme through two case studies of a face recognition system and a human activity recognition system and analyze its performance.

Online continual learning for human activity recognition

Sensor-based human activity recognition (HAR), with the ability to recognise human activities from wearable or embedded sensors, has been playing an important role in many applications including personal health monitoring, smart home, and manufacturing. The real-world, long-term deployment of these HAR systems drives a critical research question: how to evolve the HAR model automatically over time to accommodate changes in an environment or activity patterns. This paper presents an online continual learning (OCL) scenario for HAR, where sensor data arrives in a streaming manner which contains unlabelled samples from already learnt activities or new activities. We propose a technique, OCL-HAR, making a real-time prediction on the streaming sensor data while at the same time discovering and learning new activities. We have empirically evaluated OCL-HAR on four third-party, publicly available HAR datasets. Our results have shown that this OCL scenario is challenging to state-of-the-art continual learning techniques that have significantly underperformed. Our technique OCL-HAR has consistently outperformed them in all experiment setups, leading up to 0.17 and 0.23 improvements in micro and macro F1 scores.

Implementation Paper on Human Activity Recognition for Military Restricted Areas

Abstract: HAR is a field of study that aims to develop algorithms and techniques for automatically identifying and classifying human activities. Different people can execute an activity in various ways, and the same action might have numerous versions. Modern human activity recognition systems are largely taught and utilised on video stream and picture data in order to identify the features and activities, variations in the data having similar or related movements. Human-to-human and human-computer contact both heavily rely on human activity recognition. Systems that are manually operated take much longer and cost more money. The goal of this project is to develop a quicker, more affordable system for recognizing human activity and detect threats proactively through live camera. This system will help the end user in a variety of applications, such as surveillance and assistance in examination centres, hospitals, restricted areas, banks etc. by identifying the activity being carried out in the video or picture and give out alerts if detected any suspicious activity. Not only will this system be affordable, but it also functions as a utility-based system that can be integrated into a variety of applications to speed up and assist with various tasks that require recognition, which will result in significant time savings and high accuracy.

Exploiting Security Issues in Human Activity Recognition Systems (HARSs)

Human activity recognition systems (HARSs) are vital in a wide range of real-life applications and are a vibrant academic research area. Although they are adopted in many fields, such as the environment, agriculture, and healthcare and they are considered assistive technology, they seem to neglect the aspects of security and privacy. This problem occurs due to the pervasive nature of sensor-based HARSs. Sensors are devices with low power and computational capabilities, joining a machine learning application that lies in a dynamic and heterogeneous communication environment, and there is no generalized unified approach to evaluate their security/privacy, but rather only individual solutions. In this work, we studied HARSs in particular and tried to extend existing techniques for these systems considering the security/privacy of all participating components. Initially, in this work, we present the architecture of a real-life medical IoT application and the data flow across the participating entities. Then, we briefly review security and privacy issues and present possible vulnerabilities of each system layer. We introduce an architecture over the communication layer that offers mutual authentication, solving many security and privacy issues, particularly the man-in-the-middle attack (MitM). Relying on the proposed solutions, we manage to prevent unauthorized access to critical information by providing a trustworthy application.

Privacy-preserving activity recognition using multimodal sensors in smart office

Developing a human activity recognition (HAR) system for employees is essential to incorporate intelligence into smart office environments, enabling various human-centered applications to enhance employees’ well-being. Although remarkable progress has been made for the HAR in the smart office, several issues still exist, including lacking a privacy-preserving and unobtrusive method and demanding enhanced generalization performance across users. Therefore, a novel privacy-preserving HAR method based on multimodal sensors is investigated in this study, containing an infrared array sensor, a sensing chair, and a triaxial accelerometer built in a smartphone. The effectiveness of different multimodal combinations is examined. Moreover, a deep learning model is developed for multimodal data fusion to enhance the generalization performances across users. The model contains a residual 3D convolutional neural network (CNN) and 1D CNN for learning spatial–temporal feature representation of different modalities. Additionally, external memory units and an adaptive decision fusion operation are utilized for multimodal data fusion. Finally, extensive experiments are conducted to examine the performance of the proposed model using a self-collected dataset and the leave-one-subject-out cross-validation approach. The results verify the effectiveness of the proposed model.

Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Human activity recognition (HAR) is becoming increasingly important, especially with the growing number of elderly people living at home. However, most sensors, such as cameras, do not perform well in low-light environments. To address this issue, we designed a HAR system that combines a camera and a millimeter wave radar, taking advantage of each sensor and a fusion algorithm to distinguish between confusing human activities and to improve accuracy in low-light settings. To extract the spatial and temporal features contained in the multisensor fusion data, we designed an improved CNN-LSTM model. In addition, three data fusion algorithms were studied and investigated. Compared to camera data in low-light environments, the fusion data significantly improved the HAR accuracy by at least 26.68%, 19.87%, and 21.92% under the data level fusion algorithm, feature level fusion algorithm, and decision level fusion algorithm, respectively. Moreover, the data level fusion algorithm also resulted in a reduction of the best misclassification rate to 2%~6%. These findings suggest that the proposed system has the potential to enhance the accuracy of HAR in low-light environments and to decrease human activity misclassification rates.

A survey to assess perspectives of people with chronic stroke about customized safety harnesses for dance-based exergaming in home settings.

Early dance-based exergaming (DBExG) trials in people with chronic stroke (PwCS) have shown promising results, but there remains a lack of knowledge if PwCS are interested in receiving such training in their homes and the applicability of a fall-protection safety harness in the home environment. To survey people with chronic stroke to understand the perspectives and preferences of various customized safety harnesses for home-based harness assisted dance-based exergaming rehabilitation. Participants were included in this survey study if they had a stroke, lived in the community, and understood English. Participants completed a study-specific safety harness survey via mail, in-person, or online. Descriptive statistics were used to characterize the sample and survey responses. One hundred two survey responses were returned. The majority of participants voted for a door mountable harness (51%), side-release style buckle (58%), and preferred to receive a manual to explain how to use the harness (51%). Seventy-eight percent of the participants required permission to install or use a harness system in their homes. More than half of the participants preferred exhibiting independence by strapping the harness themselves (68%), along with choosing to live independently (the ability to live in one's own home and community safely, independently, comfortably, and able also perform their activities of daily living, regardless of physical ability level) (89%). PwCS opted to use ShA-DBExG in their homes. The current survey serves as a guideline to develop, customize, and prescribe home-based ShA-DBExG rehabilitation.

Application of artificial intelligence wearable devices based on neural network algorithm in mass sports activity evaluation

Based on the rapid development of big data, cloud computing, Internet of things and other technologies in recent years, intelligent hardware devices has been applied to all aspects of life. Under this background, some scholars have put forward relevant concepts such as “Smart Life”. In the field of mass sports life, through the development and application of science and technology, there has been new changes related to the application of neural network algorithm technology and intelligent hardware devices. Therefore, artificial intelligence wearable devices based on wearable technology came into being. This paper analyzes the application of this device in mass sports activities. Then, this paper describes the key research technologies of motion data processing based on neural network algorithm, including: depth frame differential convolution neural network structure, motion data extraction method, human motion signal processing algorithm, etc.; then it analyzes the action recognition and interaction system design based on Intelligent wearable devices. Finally, it analyzes the recognition results of human action system, the accuracy of human action recognition system and the factors that affect the performance of the recognition system. It is concluded that the artificial intelligent wearable devices designed in this paper can be well used in popular sports activities. Finally, it introduces the research on the evaluation strategy of popular sports activities based on artificial intelligence, and hopes that this equipment can help public sports activities. This paper studies the neural network algorithm and applies it to the design process of artificial intelligence wearable devices, which promotes the development of mass sports activity evaluation.

PhaseAnti: An Anti-Interference WiFi-Based Activity Recognition System Using Interference-Independent Phase Component

Driven by a wide range of essential applications, significant achievements are made to explore WiFi-based Human Activity Recognition (HAR) techniques that utilize the information collected by commercial off-the-shelf WiFi infrastructures to infer human activities without the need for subjects to carry any devices. Although existing WiFi-based HAR systems achieve satisfactory performance in some instances, they are faced with a severe challenge that the impacts of ubiquitous Co-channel Interference (CCI) on WiFi signals are inevitable. This downgrades the performance of these HAR systems significantly. To address this challenge, we propose PhaseAnti, a novel WiFi-based HAR system to exploit the CCI-independent phase component, Nonlinear Phase Error Variation (NLPEV), of WiFi Channel State Information to cope with the negative effects of CCI. The stability of NLPEV data and the sensibility of this component to motions are rigorously analyzed. Furthermore, validated by extensive properly designed experiments, this phase component across subcarriers is invariant under various CCI scenarios while sufficiently distinct for different motions. Therefore, the NLPEV data can be used and processed effectively to perform HAR in CCI scenarios. Extensive experiments with various daily activities in different indoor rooms demonstrate the superior effectiveness and generalizability of the proposed PhaseAnti system under various CCI scenarios.

Skeleton-Based Human Activity Recognition of Low-voltage Distribution Room Based on MQTT Protocol and Wireless Sensor Network

For the activity recognition of operators in low-voltage distribution rooms, a human activity recognition system based on the bidirectional gated recurrent unit model (BiGRU) is proposed to realize the skeleton information acquisition by the WiFi Channel State Information (CSI) sensor. The CSI sensor is used to collect the activity data and the Kinect v2 camera is used to capture the skeleton of the human body as annotations of the CSI data. The BiGRU model is employed to effectively extract the characteristics of the data. Aiming at the problem of poor remote communication, the MQTT connection can be established by using Message Queuing Telemetry Transport (MQTT) protocol combined with ESP8266 WiFi serial port communication module is proposed to improve the poor communication situation and carry out better data communication. The system achieves remote intelligent monitoring of the activities of the operator in the low-voltage distribution room.

Application of smartphone in recognition of human activities with machine learning

The aim of activity recognition is to determine the physical action being performed by one or more users based on a series of observations made during the user's actions in the relevant environment. Significant advancements in the field of human activity have resulted in the creation of novel ways for supporting elderly persons in doing their tasks independently. Using ambient computing, this type of service will be manageable. Many of services are provided by ambient technology, involving home automation tools, monitoring the behaviour of diseased individuals, and utility management. Numerous academics are focusing their efforts on computer software architectures, system infrastructure, and distributed applications utilising sensor devices. Aim of this project is to develop an algorithm that can perform human activity recognition (HAR) better than the existing state-of-the-art approach. Several tasks must be done to achieve this goal. To compete with an existing HAR system, this study will rely on secondary data from the cutting-edge experiment; no new data will be collected. The central experiment will be used to quantitatively identify the best classifier based on prediction accuracy. The current study entails monitoring and assessing existing literature in order to generate hypotheses that may be tested via experiment.

An Efficient System for Human activity recognition and monitoring for elderly people using Machine Learning

An Efficient System for Human activity recognition and monitoring for elderly people using Machine Learning