Accurate Human Activity Recognition Research Articles

Investigating the Impact of Information Sharing in Human Activity Recognition.

The accuracy of Human Activity Recognition is noticeably affected by the orientation of smartphones during data collection. This study utilized a public domain dataset that was specifically collected to include variations in smartphone positioning. Although the dataset contained records from various sensors, only accelerometer data were used in this study; thus, the developed methodology would preserve smartphone battery and incur low computation costs. A total of 175 different features were extracted from the pre-processed data. Data stratification was conducted in three ways to investigate the effect of information sharing between the training and testing datasets. After data balancing using only the training dataset, ten-fold and LOSO cross-validation were performed using several algorithms, including Support Vector Machine, XGBoost, Random Forest, Naïve Bayes, KNN, and Neural Network. A very simple post-processing algorithm was developed to improve the accuracy. The results reveal that XGBoost takes the least computation time while providing high prediction accuracy. Although Neural Network outperforms XGBoost, XGBoost demonstrates better accuracy with post-processing. The final detection accuracy ranges from 99.8% to 77.6% depending on the level of information sharing. This strongly suggests that when reporting accuracy values, the associated information sharing levels should be provided as well in order to allow the results to be interpreted in the correct context.

Shuffle-invariant Network for Action Recognition in Videos

The local key features in video are important for improving the accuracy of human action recognition. However, most end-to-end methods focus on global feature learning from videos, while few works consider the enhancement of the local information in a feature. In this article, we discuss how to automatically enhance the ability to discriminate the local information in an action feature and improve the accuracy of action recognition. To address these problems, we assume that the critical level of each region for the action recognition task is different and will not change with the region location shuffle. We therefore propose a novel action recognition method called the shuffle-invariant network. In the proposed method, the shuffled video is generated by regular region cutting and random confusion to enhance the input data. The proposed network adopts the multitask framework, which includes one feature backbone network and three task branches: local critical feature shuffle-invariant learning, adversarial learning, and an action classification network. To enhance the local features, the feature response of each region is predicted by a local critical feature learning network. To train this network, an L 1-based critical feature shuffle-invariant loss is defined to ensure that the ordered feature response list of these regions remains unchanged after region location shuffle. Then, the adversarial learning is applied to eliminate the noise caused by the region shuffle. Finally, the action classification network combines these two tasks to jointly guide the training of the feature backbone network and obtain more effective action features. In the testing phase, only the action classification network is applied to identify the action category of the input video. We verify the proposed method on the HMDB51 and UCF101 action datasets. Several ablation experiments are constructed to verify the effectiveness of each module. The experimental results show that our approach achieves the state-of-the-art performance.

Privacy-Preserving Collaborative Computation for Human Activity Recognition

Human Activity Recognition (HAR) enables computer systems to assist users with their tasks and improve their quality of life in rehabilitation, daily life tracking, fitness, and cognitive disorder therapy. It is a hot topic in the field of machine learning, and HAR is gaining more attention among researchers due to its unique societal and economic advantages. This paper focuses on a collaborative computation scenario where a group of participants will securely and collaboratively train an accurate HAR model. The training process requires collecting a massive number of personal activity features and labels, which raises privacy problems. We decentralize the training process locally to each client in order to ensure the privacy of training data. Furthermore, we use an advanced secure aggregation algorithm to ensure that malicious participants cannot extract private information from the updated parameters even during the aggregation phase. Edge computing nodes have been introduced into our system to address the problem of data generation devices’ insufficient computing power. We replace the traditional central server with smart contract to make the system more robust and secure. We achieve the verifiability of the packaged nodes using the publicly auditability feature of blockchain. According to the experimental data, the accuracy of the HAR model trained by our proposed framework reaches 93.24%, which meets the applicability requirements. The use of secure multiparty computation techniques unavoidably increases training time, and experimental results show that a round of iterations takes 36.4 seconds to execute, which is still acceptable.

An End-to-End Deep Learning Pipeline for Football Activity Recognition Based on Wearable Acceleration Sensors.

Action statistics in sports, such as the number of sprints and jumps, along with the details of the corresponding locomotor actions, are of high interest to coaches and players, as well as medical staff. Current video-based systems have the disadvantage that they are costly and not easily transportable to new locations. In this study, we investigated the possibility to extract these statistics from acceleration sensor data generated by a previously developed sensor garment. We used deep learning-based models to recognize five football-related activities (jogging, sprinting, passing, shooting and jumping) in an accurate, robust, and fast manner. A combination of convolutional (CNN) layers followed by recurrent (bidirectional) LSTM layers achieved up to 98.3% of accuracy. Our results showed that deep learning models performed better in evaluation time and prediction accuracy than traditional machine learning algorithms. In addition to an increase in accuracy, the proposed deep learning architecture showed to be 2.7 to 3.4 times faster in evaluation time than traditional machine learning methods. This demonstrated that deep learning models are accurate as well as time-efficient and are thus highly suitable for cost-effective, fast, and accurate human activity recognition tasks.

Footprint Extraction and Sports Training Action Recognition Based on Wireless Network Communication

The combination of scientific and technological achievements and sports has found new opportunities to change people’s sports habits. Sport training takes up an increasing proportion of people’s lives. In order to improve the efficiency of sports training and standardize the training actions of players, this article is based on wireless network communication and uses different types of recognition methods in the field of action recognition to build basic classifications. Iterative mutual training to improve generalization performance can reduce the cost of labeling and realize the complementary advantages of different recognition methods, thereby improving the recognition accuracy of human actions. Finally, the algorithm is used to recognize human movements. This method can effectively overcome the problem of differential degradation of base classifiers in the iterative process of collaborative training and further improve the accuracy of human action recognition. The experimental results prove that the motion recognition of wireless network communication proposed in this paper can effectively improve the accuracy of athletes’ movements, which is more than 20% higher than traditional methods, and, under the guidance of standardized movements, can reduce athletes’ sports injuries.

A novel approach for context-aware sensor optimization in a smart home

Ambient Intelligence is an emerging technology that allows for reasoning of activities that are carried out in a smart environment. Smart sensors play a vital role in activity recognition tasks. Though a great number of sensors provide more pinpointed information about where and when the interaction in the environment occurred, addition of sensors also results in higher power consumption and greater cost constraints. The number of training samples used for activity recognition tasks also increases and becomes computationally expensive. Thus, to overcome these constraints sensor optimization in smart environment is carried out. Mapping of sensors to a user's activity becomes tedious when more than one activity is carried out at the same timestamp. Unambiguous sensor-activity relationship is derived with the help of spatial and temporal context represented by ontology model that captures real time sensor data. Since there exits spatial noise during activity annotation process, it is eliminated with help of pointwise mutual information score. Spatial and temporal data of an activity helps sensor optimization in multi resident living with concurrent activity occurrence. Inactive sensors are removed using mutual information score and redundant sensors are contextually paired based on where an activity happens. By doing so, only unique set of sensors are paired every time even when concurrent activity occurs. The quality of features derived is tested by testing the accuracy of the model by activity recognition task using Random Forest algorithm. Nearly 24% of sensors were identified as inactive in the smart environment. The accuracy of Human Activity Recognition (HAR) task remained constant even after removing the inactive sensors. Addition of spatial context helped to improve HAR with increased accuracy. Contextual pairing of redundant sensors resulted in minimal drop in the model's accuracy.

A Comparative Study: Toward an Effective Convolutional Neural Network Architecture for Sensor-Based Human Activity Recognition

The feature extraction of human activity recognition (HAR) based on sensor data has been studied as a hand-crafted method. The significant feature extraction ability is a key factor in improving the accuracy of HAR. Recently, deep learning methods have been employed for feature extraction. In this paper, we review previous studies on deep learning methods in HAR and discuss suitable models for feature extraction. First, we applied various convolutional neural networks to clarify the effective architecture for HAR. Afterward, we developed advanced models by embedding submodules, such as self-attention and recurrent neural networks, often adopted in recent studies. Comparative experiments on HASC, UCI, and WISDM public datasets showed that Inception-V3, which used cross-channel multi-size convolution transformation, outperformed other backbones. Through comparative experiments after embedding submodules, submodules do not always have a positive effect on accuracy. Compared with other submodules, SENet has a positive effect. We conclude that it is essential to select an appropriate backbone model before applying the submodules, and submodules are unnecessary in some cases.

A Semi-Supervised Transfer Learning with Dynamic Associate Domain Adaptation for Human Activity Recognition Using WiFi Signals.

Human activity recognition without equipment plays a vital role in smart home applications, freeing humans from the shackles of wearable devices. In this paper, by using the channel state information (CSI) of the WiFi signal, semi-supervised transfer learning with dynamic associate domain adaptation is proposed for human activity recognition. In order to improve the CSI quality and denoising of CSI, we carried out missing packet filling, burst noise removal, background estimation, feature extraction, feature enhancement, and data augmentation in the data pre-processing stage. This paper considers the problem of environment-independent human activity recognition, also known as domain adaptation. The pre-trained model is trained from the source domain by collecting a complete labeled dataset of all of the CSI of human activity patterns. Then, the pre-trained model is transferred to the target environment through the semi-supervised transfer learning stage. Therefore, when humans move to different target domains, a partial labeled dataset of the target domain is required for fine-tuning. In this paper, we propose a dynamic associate domain adaptation called DADA. By modifying the existing associate domain adaptation algorithm, the target domain can provide a dynamic ratio of labeled dataset/unlabeled dataset, while the existing associate domain adaptation algorithm only allows target domains with the unlabeled dataset. The advantage of DADA is that it provides a dynamic strategy to eliminate different effects on different environments. In addition, we further designed an attention-based DenseNet model, or AD, as our training network, which is modified by an existing DenseNet by adding the attention function. The solution we proposed was simplified to DADA-AD throughout the paper. The experimental results show that for domain adaptation in different domains, the accuracy of human activity recognition of the DADA-AD scheme is 97.4%. It also shows that DADA-AD has advantages over existing semi-supervised learning schemes.

Feature Fusion of a Deep-Learning Algorithm into Wearable Sensor Devices for Human Activity Recognition.

This paper presents a wearable device, fitted on the waist of a participant that recognizes six activities of daily living (walking, walking upstairs, walking downstairs, sitting, standing, and laying) through a deep-learning algorithm, human activity recognition (HAR). The wearable device comprises a single-board computer (SBC) and six-axis sensors. The deep-learning algorithm employs three parallel convolutional neural networks for local feature extraction and for subsequent concatenation to establish feature fusion models of varying kernel size. By using kernels of different sizes, relevant local features of varying lengths were identified, thereby increasing the accuracy of human activity recognition. Regarding experimental data, the database of University of California, Irvine (UCI) and self-recorded data were used separately. The self-recorded data were obtained by having 21 participants wear the device on their waist and perform six common activities in the laboratory. These data were used to verify the proposed deep-learning algorithm on the performance of the wearable device. The accuracy of these six activities in the UCI dataset and in the self-recorded data were 97.49% and 96.27%, respectively. The accuracies in tenfold cross-validation were 99.56% and 97.46%, respectively. The experimental results have successfully verified the proposed convolutional neural network (CNN) architecture, which can be used in rehabilitation assessment for people unable to exercise vigorously.

A Lightweight Framework for Human Activity Recognition on Wearable Devices

Human Activity Recognition (HAR) is the automatic detection and understanding of human motion behavior based on data extracted from video camera, ambient sensors or wearable sensors which particularly has recently attracted increased attention from both researchers and industry. However, for running practical HAR systems on wearable devices, there are some requirements such as design and development of small, lightweight, powerful, and low-cost smart sensors. In this context, data must be continuously collected, and edge computing is a viable solution, which is an energy-efficient technique, offering real-time response and privacy requirements for HAR applications. Rather than sending data to the cloud, edge computing is a local process that minimizes the data transmission time and responds with low latency. Recently, HAR system designers have adopted deep learning techniques inspired by their outstanding performance in many application areas and achieved relevant gain in activity recognition performance, however, these techniques were not demonstrated suitable for running on resource constrained devices. Thus, designing energy-efficient deep learning models is critical for realizing efficient HAR for mobile applications. In this work, we present a lightweight framework for the deployment of low-power but accurate HAR systems for these devices. We also implement and run the system in a microcontroller and analyze computational cost and energy consumption, and how different system configurations and deep learning model complexity influence on that.

Radar-Based Human Activity Recognition Using Hybrid Neural Network Model With Multidomain Fusion

This article concerns the issue of how to combine the multidomainradar information, including range–Doppler, time–Doppler, and time–range, for human activity recognition. Specifically, to fully make use of radar information, instead of using a single-domain spectrum as inputs, a novel hybrid neural network model is developed for exploring multidomain fusion of radar information. In doing this, three kinds of 2-D domain spectra are used in a fashion of supplementing each other with a hybrid framework that combines three models: 1-D convolution neural network, recurrent neural network, and 2-D convolution network. It is advantageous to use such a hybrid model to capture much rich features through multidomain feature fusion, so as to improve the accuracy of human activity recognition effectively. Experimental results validate the proposed method.

Feature fusion using deep learning for smartphone based human activity recognition.

Identification of human physical activities is an active research area since long due to its application in personalized health and fitness monitoring. The performance accuracy of human activity recognition (HAR) models mainly depend on the features which are extracted from domain knowledge. The features are the input of the classification algorithm to efficiently identify human physical activities. Manually extracted features (handcrafted) need expert domain knowledge. Thus these features have significant importance to identify different human activities. Recently deep learning methods are utilized to extract the features automatically from raw sensory data for HAR models. However, state-of-the-art HAR literature established that the importance of handcrafted features can’t be ignored as it is extracted from expert domain knowledge. Thus, in this paper we use the fusion of both the handcrafted features and automatically extracted features using deep learning (DL) for HAR model to enhance the performance of HAR. Extensive experimental results demonstrate that our proposed feature fusion based HAR model gives higher accuracy compared with state-of-the-art HAR literature for both the self collected and public dataset.

An Adaptive Human Activity-Aided Hand-Held Smartphone-Based Pedestrian Dead Reckoning Positioning System

Pedestrian dead reckoning (PDR), enabled by smartphones’ embedded inertial sensors, is widely applied as a type of indoor positioning system (IPS). However, traditional PDR faces two challenges to improve its accuracy: lack of robustness for different PDR-related human activities and positioning error accumulation over elapsed time. To cope with these issues, we propose a novel adaptive human activity-aided PDR (HAA-PDR) IPS that consists of two main parts, human activity recognition (HAR) and PDR optimization. (1) For HAR, eight different locomotion-related activities are divided into two classes: steady-heading activities (ascending/descending stairs, stationary, normal walking, stationary stepping, and lateral walking) and non-steady-heading activities (door opening and turning). A hierarchical combination of a support vector machine (SVM) and decision tree (DT) is used to recognize steady-heading activities. An autoencoder-based deep neural network (DNN) and a heading range-based method to recognize door opening and turning, respectively. The overall HAR accuracy is over 98.44%. (2) For optimization methods, a process automatically sets the parameters of the PDR differently for different activities to enhance step counting and step length estimation. Furthermore, a method of trajectory optimization mitigates PDR error accumulation utilizing the non-steady-heading activities. We divided the trajectory into small segments and reconstructed it after targeted optimization of each segment. Our method does not use any a priori knowledge of the building layout, plan, or map. Finally, the mean positioning error of our HAA-PDR in a multilevel building is 1.79 m, which is a significant improvement in accuracy compared with a baseline state-of-the-art PDR system.

Human Activity Recognition for People with Knee Osteoarthritis-A Proof-of-Concept.

Clinicians lack objective means for monitoring if their knee osteoarthritis patients are improving outside of the clinic (e.g., at home). Previous human activity recognition (HAR) models using wearable sensor data have only used data from healthy people and such models are typically imprecise for people who have medical conditions affecting movement. HAR models designed for people with knee osteoarthritis have classified rehabilitation exercises but not the clinically relevant activities of transitioning from a chair, negotiating stairs and walking, which are commonly monitored for improvement during therapy for this condition. Therefore, it is unknown if a HAR model trained on data from people who have knee osteoarthritis can be accurate in classifying these three clinically relevant activities. Therefore, we collected inertial measurement unit (IMU) data from 18 participants with knee osteoarthritis and trained convolutional neural network models to identify chair, stairs and walking activities, and phases. The model accuracy was 85% at the first level of classification (activity), 89–97% at the second (direction of movement) and 60–67% at the third level (phase). This study is the first proof-of-concept that an accurate HAR system can be developed using IMU data from people with knee osteoarthritis to classify activities and phases of activities.

Deep ConvLSTM Network with Dataset Resampling for Upper Body Activity Recognition Using Minimal Number of IMU Sensors

Human activity recognition (HAR) is the study of the identification of specific human movement and action based on images, accelerometer data and inertia measurement unit (IMU) sensors. In the sensor based HAR application, most of the researchers used many IMU sensors to get an accurate HAR classification. The use of many IMU sensors not only limits the deployment phase but also increase the difficulty and discomfort for users. As reported in the literature, the original model used 19 sensor data consisting of accelerometers and IMU sensors. The imbalanced class distribution is another challenge to the recognition of human activity in real-life. This is a real-life scenario, and the classifier may predict some of the imbalanced classes with very high accuracy. When a model is trained using an imbalanced dataset, it can degrade model’s performance. In this paper, two approaches, namely resampling and multiclass focal loss, were used to address the imbalanced dataset. The resampling method was used to reconstruct the imbalanced class distribution of the IMU sensor dataset prior to model development and learning using the cross-entropy loss function. A deep ConvLSTM network with a minimal number of IMU sensor data was used to develop the upper-body HAR model. On the other hand, the multiclass focal loss function was used in the HAR model and classified minority classes without the need to resample the imbalanced dataset. Based on the experiments results, the developed HAR model using a cross-entropy loss function and reconstructed dataset achieved a good performance of 0.91 in the model accuracy and F1-score. The HAR model with a multiclass focal loss function and imbalanced dataset has a slightly lower model accuracy and F1-score in both 1% difference from the resampling method. In conclusion, the upper body HAR model using a minimal number of IMU sensors and proper handling of imbalanced class distribution by the resampling method is useful for the assessment of home-based rehabilitation involving activities of daily living.

Wi-Fi-Based Location-Independent Human Activity Recognition via Meta Learning

Wi-Fi-based device-free human activity recognition has recently become a vital underpinning for various emerging applications, ranging from the Internet of Things (IoT) to Human–Computer Interaction (HCI). Although this technology has been successfully demonstrated for location-dependent sensing, it relies on sufficient data samples for large-scale sensing, which is enormously labor-intensive and time-consuming. However, in real-world applications, location-independent sensing is crucial and indispensable. Therefore, how to alleviate adverse effects on recognition accuracy caused by location variations with the limited dataset is still an open question. To address this concern, we present a location-independent human activity recognition system based on Wi-Fi named WiLiMetaSensing. Specifically, we first leverage a Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM) feature representation method to focus on location-independent characteristics. Then, in order to well transfer the model across different positions with limited data samples, a metric learning-based activity recognition method is proposed. Consequently, not only the generalization ability but also the transferable capability of the model would be significantly promoted. To fully validate the feasibility of the presented approach, extensive experiments have been conducted in an office with 24 testing locations. The evaluation results demonstrate that our method can achieve more than 90% in location-independent human activity recognition accuracy. More importantly, it can adapt well to the data samples with a small number of subcarriers and a low sampling rate.

Multifeature fusion action recognition based on key frames

SummaryAs an important technology in computer vision, video‐based human action recognition has a great commercial value, which has attracted extensive attention in the field of computer vision and pattern recognition in both academia and industry. To date, there are a wide variety of applications of human action recognition, such as surveillance, robotics, health care, video searching, and human–computer interaction. However, there are many challenges involved in human action recognition in videos, such as cluttered backgrounds, occlusions, viewpoint variation, execution rate, and camera motion. However, data redundancy and single feature were largely limited the accuracy of human action recognition. In this article, adopting the key frame extraction and multifeature fusion techniques, a novel action recognition method was proposed, which can improve the recognition accuracy. The main works are as follows: 1) in order to solve the problem of data redundancy, a key frame extraction method based on node contribution weighting is proposed to extract video key frames; 2) different kinds of information flows are extracted from the obtained key frame sequences, and different convolutional neural networks are used to obtain corresponding classification results and merge, so as to better complement the information in different flows. Lastly, the experimental results show that our method improves the accuracy of action recognition.

Energy allocation for activity recognition in wearable devices with kinetic energy harvesting

AbstractHarvesting kinetic energy from body movement is regarded as a promising rechargeable energy source for wearable devices with low‐power. Energy allocation is essential for motion‐based rechargeable devices since the great variability of energy gained from movement. Based on the realistic characteristics of an ultra‐low‐power wearable devices and our measurement observations, we propose the optimization framework allocating energy to maximize the average accuracy of human activity recognition and provide an offline and online algorithm, respectively. We evaluate the proposed energy allocation approach with real‐world human activity and kinetic energy harvesting datasets. Experimental results validate that our proposed energy allocation approach can maximize the energy allocation utility and improve energy efficiency of wearable devices.

Inertial Sensor Data to Image Encoding for Human Action Recognition

Convolutional Neural Networks (CNNs) are successful deep learning models in the field of computer vision. To get the maximum advantage of CNN model for Human Action Recognition (HAR) using inertial sensor data, in this paper, we use four types of spatial domain methods for transforming inertial sensor data to activity images, which are then utilized in a novel fusion framework. These four types of activity images are Signal Images (SI), Gramian Angular Field (GAF) Images, Markov Transition Field (MTF) Images and Recurrence Plot (RP) Images. Furthermore, for creating a multimodal fusion framework and to exploit activity images, we made each type of activity images multimodal by convolving with two spatial domain filters: Prewitt filter and High-boost filter. ResNet-18, a CNN model, is used to learn deep features from multi-modalities. Learned features are extracted from the last pooling layer of each ResNet and then fused by canonical correlation based fusion (CCF) for improving the accuracy of human action recognition. These highly informative features are served as input to a multi-class Support Vector Machine (SVM). Experimental results on three publicly available inertial datasets show the superiority of the proposed method over the current state-of-the-art.

Weighted Support Tensor Machines for Human Activity Recognition with Smartphone Sensors

Along with the development of the Industrial Internet of Things, human activity recognition (HAR) has received widespread attention in many fields. Support Vector Machine (SVM), is widely used by researchers for human activity recognition. However, the inherent difference of signal properties from different sensors and various orientations is potentially lost when using the vector-based SVM for human activity recognition. What's more, the outlier sensitivity problem of the standard SVM reduces the accuracy of human activity recognition. To tackle this problem, we present a tensor-based feature representation model and a weighted support tensor machine (WSTM) for human activity recognition. Specifically, tensor-based representations are first used to model features from different sensors and various orientations to retain the latent relationship. In addition, the weighted support tensor machine is proposed to classify the human activities in tensor space while avoiding the outlier sensitivity problem. Experimental results demonstrate the proposed WSTM algorithm.