Sensor-based Activity Recognition Research Articles

Data augmentation aided excavator activity recognition using deep convolutional conditional generative adversarial networks

Excavator activity recognition is a crucial task with significant practical applications such as assessing production efficiency, injury prevention, and enabling intelligent control. However, implementing deep learning-based activity recognition methods presents considerable challenges due to the requirement of large training datasets. And sensor-based activity recognition demands more convenient and reliable sensors. This paper proposes a generalized data augmentation framework for excavator activity recognition. Main pump pressure and flow signals are generated using Deep Convolutional Conditional Generative Adversarial Networks (DC-CGAN). The proposed framework integrates Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) architectures for activity recognition. Results indicate that the framework outperforms traditional data augmentation methods regarding the quality of generated data and shallow and single deep networks regarding model accuracy and generalization. Additionally, the framework was implemented on three sizes of excavators and a wheel loader, demonstrating excellent versatility.

Deep similarity segmentation model for sensor-based activity recognition

Deep similarity segmentation model for sensor-based activity recognition

Large Receptive Field Attention: An Innovation in Decomposing Large-Kernel Convolution for Sensor-Based Activity Recognition

Large Receptive Field Attention: An Innovation in Decomposing Large-Kernel Convolution for Sensor-Based Activity Recognition

Sports activity (SA) recognition based on error correcting output codes (ECOC) and convolutional neural network (CNN)

The increasing use of motion sensors is causing major changes in the process of monitoring people's activities. One of the main applications of these sensors is the detection of sports activities, for example, they can be used to monitor the condition of athletes or analyze the quality of sports training. Although the existing sensor-based activity recognition systems can recognize basic activities such as: walking, running, or sitting; they don't perform well in recognizing different types of sports activities. This article introduces a new model based on machine learning (ML) techniques to more accurately distinguish between sports and everyday activities. In the proposed method, the necessary data to detect the type of activity is collected through his two sensors: an accelerometer and a gyroscope attached to a person's foot. For this purpose, the input signals are first preprocessed and then short-time Fourier transform (STFT) is used to describe the characteristics of each signal. In the next step, each STFT matrix is used as input to a convolutional neural network (CNN). This CNN describes various motion characteristics of the sensor in the form of vectors. Finally, a classification model based on error correction output code (ECOC) is used to classify the extracted features and detect the type of SA. The performance of the proposed AS recognition method is evaluated using the DSADS database and the results are compared with previous methods. Based on the results, the proposed method can recognize sports activities with an accuracy of 99.71. Furthermore, the performance of the proposed method based on precision and recall criteria are 99.72 and 99.71, respectively, which are better than the compared methods.

Deep neural networks for wearable sensor-based activity recognition in Parkinson’s disease: investigating generalizability and model complexity

BackgroundThe research gap addressed in this study is the applicability of deep neural network (NN) models on wearable sensor data to recognize different activities performed by patients with Parkinson’s Disease (PwPD) and the generalizability of these models to PwPD using labeled healthy data.MethodsThe experiments were carried out utilizing three datasets containing wearable motion sensor readings on common activities of daily living. The collected readings were from two accelerometer sensors. PAMAP2 and MHEALTH are publicly available datasets collected from 10 and 9 healthy, young subjects, respectively. A private dataset of a similar nature collected from 14 PwPD patients was utilized as well. Deep NN models were implemented with varying levels of complexity to investigate the impact of data augmentation, manual axis reorientation, model complexity, and domain adaptation on activity recognition performance.ResultsA moderately complex model trained on the augmented PAMAP2 dataset and adapted to the Parkinson domain using domain adaptation achieved the best activity recognition performance with an accuracy of 73.02%, which was significantly higher than the accuracy of 63% reported in previous studies. The model’s F1 score of 49.79% significantly improved compared to the best cross-testing of 33.66% F1 score with only data augmentation and 2.88% F1 score without data augmentation or domain adaptation.ConclusionThese findings suggest that deep NN models originating on healthy data have the potential to recognize activities performed by PwPD accurately and that data augmentation and domain adaptation can improve the generalizability of models in the healthy-to-PwPD transfer scenario. The simple/moderately complex architectures tested in this study could generalize better to the PwPD domain when trained on a healthy dataset compared to the most complex architectures used. The findings of this study could contribute to the development of accurate wearable-based activity monitoring solutions for PwPD, improving clinical decision-making and patient outcomes based on patient activity levels.

Channel Attention for Sensor-Based Activity Recognition: Embedding Features into all Frequencies in DCT Domain

Channel Attention for Sensor-Based Activity Recognition: Embedding Features into all Frequencies in DCT Domain

Phase Randomization: A data augmentation for domain adaptation in human action recognition

Human action recognition models often suffer from achieving both accurate recognition and subject independence when the amount of training data is limited. In this paper, we propose a data-efficient domain adaptation approach to learning a subject-agnostic action recognition classifier. The core component of our approach is a novel data augmentation called Phase Randomization. On the basis of the observation that individual body size is highly correlated with the amplitude component of the motion sequence, we disentangle the individuality and action features by using contrastive self-supervised learning with data augmentation that randomizes only the phase component of the motion sequence. This enables us to estimate the subject label of each motion sequence and to train a subject-agnostic action recognition classifier by performing adversarial learning with the estimated subject labels. We empirically demonstrate the superiority of our method on two different action recognition tasks (skeleton-based action recognition and sensor-based activity recognition).

Similarity Segmentation Approach for Sensor-Based Activity Recognition

Similarity Segmentation Approach for Sensor-Based Activity Recognition

A Boundary Consistency-Aware Multitask Learning Framework for Joint Activity Segmentation and Recognition With Wearable Sensors

With the development of industrial and sensing technology, sensor-based activity recognition has become a promising technology for informatics applications. However, in a typical activity recognition procedure, sensory data segmentation, usually considered a preprocess with sliding windows, rarely has been investigated and significantly affected the recognition performance. In this article, we propose a novel deep-learning method to jointly segment and recognize activities with wearable sensors. Our contributions are three-fold: First, we introduce a multistage temporal convolutional network for sample-level activity prediction to overcome the multiclass windows problem. Second, for alleviating oversegmentation errors, our model forms a multitask learning framework with a boundary prediction module to adjust the entire model’s gradients. Third, we innovatively propose a boundary consistency loss to enforce the consistency of the activity and boundary prediction. Our method shows impressive performance on three public datasets, especially achieving 16% improvement over very recently advanced competing methods with class-average F1-score on the Hospital dataset. The code of this work will be open source on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/xspc/Segmentation-Sensor-based-HAR</uri> .

Sensor-Based Activity Recognition Using Frequency Band Enhancement Filters and Model Ensembles.

Deep learning methods are widely used in sensor-based activity recognition, contributing to improved recognition accuracy. Accelerometer and gyroscope data are mainly used as input to the models. Accelerometer data are sometimes converted to a frequency spectrum. However, data augmentation based on frequency characteristics has not been thoroughly investigated. This study proposes an activity recognition method that uses ensemble learning and filters that emphasize the frequency that is important for recognizing a certain activity. To realize the proposed method, we experimentally identified the important frequency of various activities by masking some frequency bands in the accelerometer data and comparing the accuracy using the masked data. To demonstrate the effectiveness of the proposed method, we compared its accuracy with and without enhancement filters during training and testing and with and without ensemble learning. The results showed that applying a frequency band enhancement filter during training and testing and ensemble learning achieved the highest recognition accuracy. In order to demonstrate the robustness of the proposed method, we used four different datasets and compared the recognition accuracy between a single model and a model using ensemble learning. As a result, in three of the four datasets, the proposed method showed the highest recognition accuracy, indicating the robustness of the proposed method.

Radar Based Activity Recognition using CNN-LSTM Network Architecture

Human Activity Recognition based research has got intensified based on the evolving demand of smart systems. There has been already a lot of wearables, digital smart sensors deployed to classify various activities. Radar sensor-based Activity recognition has been an active research area during recent times. In order to classify the radar micro doppler signature images we have proposed a approach using Convolutional Neural Network-Long Short Term Memory (CNN-LSTM). Convolutional Layer is used to update the filter values to learn the features of the radar images. LSTM Layer enhances the temporal information besides the features obtained through Convolutional Neural Network. We have used a dataset published by University of Glasgow that captures six activities for 56 subjects under different ages, which is a first of its kind dataset unlike the signals captured under controlled lab environment. Our Model has achieved 96.8% for the training data and 93.5% for the testing data. The proposed work has outperformed the existing traditional deep learning Architectures.

Improving Inertial Sensor-Based Activity Recognition in Neurological Populations.

Inertial sensor-based human activity recognition (HAR) has a range of healthcare applications as it can indicate the overall health status or functional capabilities of people with impaired mobility. Typically, artificial intelligence models achieve high recognition accuracies when trained with rich and diverse inertial datasets. However, obtaining such datasets may not be feasible in neurological populations due to, e.g., impaired patient mobility to perform many daily activities. This study proposes a novel framework to overcome the challenge of creating rich and diverse datasets for HAR in neurological populations. The framework produces images from numerical inertial time-series data (initial state) and then artificially augments the number of produced images (enhanced state) to achieve a larger dataset. Here, we used convolutional neural network (CNN) architectures by utilizing image input. In addition, CNN enables transfer learning which enables limited datasets to benefit from models that are trained with big data. Initially, two benchmarked public datasets were used to verify the framework. Afterward, the approach was tested in limited local datasets of healthy subjects (HS), Parkinson's disease (PD) population, and stroke survivors (SS) to further investigate validity. The experimental results show that when data augmentation is applied, recognition accuracies have been increased in HS, SS, and PD by 25.6%, 21.4%, and 5.8%, respectively, compared to the no data augmentation state. In addition, data augmentation contributes to better detection of stair ascent and stair descent by 39.1% and 18.0%, respectively, in limited local datasets. Findings also suggest that CNN architectures that have a small number of deep layers can achieve high accuracy. The implication of this study has the potential to reduce the burden on participants and researchers where limited datasets are accrued.

Transformer With Bidirectional GRU for Nonintrusive, Sensor-Based Activity Recognition in a Multiresident Environment

Several techniques for human activity recognition (HAR) in a smart indoor environment have been developed and improved along with the rapid advancement of sensor technologies. However, recognizing multiple people’s activities is still challenging due to the complexity of their activities, such as parallel and collaborative activities. To address these challenges, we propose a transformer with a bidirectional gated recurrent unit (GRU) deep learning (DL) method, called TRANS-BiGRU, to efficiently learn and recognize different types of activities performed by multiple residents. We compare the proposed model with the state-of-the-art models and various DL models, such as Ensemble2LSTM (Ens2-LSTM), bidirectional GRUs (Bi-GRU), and traditional machine learning (ML) models, such as support vector machine (SVM). Our experimental results based on the center for advanced studies in adaptive system and ARAS public data sets show that our model significantly outperforms the existing models for complex activity recognition of multiple residents.

Triple Cross-Domain Attention on Human Activity Recognition Using Wearable Sensors

Efficiently identifying activities of daily living (ADL) provides very important contextual information that is able to improve the effectiveness of various sports tracking and healthcare applications. Recently, attention mechanism that selectively focuses on time series signals has been widely adopted in sensor based human activity recognition (HAR), which can enhance interesting target activity and ignore irrelevant background activity. Several attention mechanisms have been investigated, which achieve remarkable performance in HAR scenario. Despite their success, prior these attention methods ignore the cross-interaction between different dimensions. In the paper, in order to avoid above shortcoming, we present a triplet cross-dimension attention for sensor-based activity recognition task, where three attention branches are built to capture the cross-interaction between sensor dimension, temporal dimension and channel dimension. The effectiveness of triplet attention method is validated through extensive experiments on four public HAR dataset namely UCI-HAR, PAMAP2, WISDM and UNIMIB-SHAR as well as the weakly labeled HAR dataset. Extensive experiments show consistent improvements in classification performance with various backbone models such as plain CNN and ResNet, demonstrating a good generality ability of the triplet attention. Visualization analysis is provided to support our conclusion, and actual implementation is evaluated on a Raspberry Pi platform.

Generic semi-supervised adversarial subject translation for sensor-based activity recognition

Performance of Human Activity Recognition (HAR) models, particularly deep neural networks, is highly contingent upon the availability of the massive amount of annotated training data. Though, data collection and manual labeling in the HAR domain are prohibitively expensive due to human resource dependence in both steps. Hence, domain adaptation techniques are proposed to adapt the knowledge from the existing source of data. More recently, adversarial transfer learning methods have shown promising results for visual classification, yet limited for HAR problems, which are still prone to the unfavorable effects of the imbalanced distribution of samples. This paper presents a novel generic semi-supervised approach that takes advantage of the adversarial framework to tackle these shortcomings by leveraging knowledge from annotated samples exclusively from the source subject and unlabeled ones of the target subject. An extensive subject translation experiments is conducted on three large, middle, and small-size datasets with different levels of imbalance to assess the robustness of the proposed model to the scale as well as the imbalance in the data. The results demonstrate the effectiveness of our proposed algorithms over state-of-the-art methods, which led to up to 13%, 4%, and 13% improvement of our high-level activities recognition metrics for Opportunity, LISSI, and PAMAP2 datasets, respectively.

PGAN framework for synthesizing sensor data privately

Some credible third parties collect and share mobile sensor data with users to enhance scientific innovation. However, since the sensor data contains sensitive attributes, it may lead to an unexpected privacy leak. Despite privacy filtering, differential privacy, and inferential privacy techniques that attempt to address this problem, the preservation of greater privacy results in a considerable loss of utility and vice versa. In this research, we tackle the privacy-preserving for sensor-based activity recognition data-sharing issue via balancing between (1) data quality perspective and (2) privacy-preserving perspective. Two novel private generative adversarial approaches, namely PGAN1 and PGAN2, are proposed, whereby we generate filtered data to be shared before revealing the raw data to a data analyst. PGAN1 and PGAN2 differ in the privacy mechanism of raw data filtering. Therefore, PGAN1 relies on a transformation algorithm, while PGAN2 relies on a synthesis algorithm. We theoretically characterize the problem and formulate new objective functions, each one as an analogy of minimax formulation among more than two networks. We evaluate the utility performance by different classifiers while the privacy is checked by overcoming the attacks. Experimental results show that PGAN1 and PGAN2 improve utility and effectively prevent privacy leaks better than previous works.

Daily Living Activity Recognition In-The-Wild: Modeling and Inferring Activity-Aware Human Contexts

Advancement in smart sensing and computing technologies has provided a dynamic opportunity to develop intelligent systems for human activity monitoring and thus assisted living. Consequently, many researchers have put their efforts into implementing sensor-based activity recognition systems. However, recognizing people’s natural behavior and physical activities with diverse contexts is still a challenging problem because human physical activities are often distracted by changes in their surroundings/environments. Therefore, in addition to physical activity recognition, it is also vital to model and infer the user’s context information to realize human-environment interactions in a better way. Therefore, this research paper proposes a new idea for activity recognition in-the-wild, which entails modeling and identifying detailed human contexts (such as human activities, behavioral environments, and phone states) using portable accelerometer sensors. The proposed scheme offers a detailed/fine-grained representation of natural human activities with contexts, which is crucial for modeling human-environment interactions in context-aware applications/systems effectively. The proposed idea is validated using a series of experiments, and it achieved an average balanced accuracy of 89.43%, which proves its effectiveness.

Sensor-Based Activity Recognition and Performance Assessment in Climbing: A Review

In the past decades, a number of technological developments made it possible to continuously collect various types of sport activity data in an unobtrusive way. Machine learning and analytical methods have been applied to flows of sensor data to predict the conducted sport activity as well as to calculate key performance indicators. In that scenario, researchers started to be interested in leveraging pervasive information technologies for sport climbing, thus allowing, in day-to-day climbing practice, the realization of systems for automatic assessment of a climber’s performance, detection of injury risk factors, and virtual coaching. This article surveys recent research works on the recognition of climbing activities and the evaluation of climbing performance indicators, where data have been acquired with accelerometers, cameras, force sensors, and other types of sensors. We describe the main types of sensors and equipment adopted for data acquisition, the techniques used to extract relevant features from sensor data, and the methods that have been proposed to identify the activities performed by a climber and to calculate key performance indicators. We also present a classification taxonomy of climbing activities and climbing performance indicators with the aim to unify the existing work and facilitate comparison of different methods. Moreover, open problems that call for new approaches and solutions are here discussed. We conclude that there is considerable scope for further work, particularly in the application of recognition techniques to problems involving different climbing activities. We hope that this survey will assist in the translation of research effort into intelligent environments that climbers will benefit from.

A Multichannel CNN-GRU Model for Human Activity Recognition

Human activity recognition (HAR) is one of the important research areas in pervasive computing. Among HAR, sensor-based activity recognition refers to acquiring a high-level knowledge about human activities from readings of many low-level sensor. In recent years, although traditional methods of deep learning (DL) have been widely used for sensor-based HAR with some good performance, they still face such challenges as feature extraction and characterization, continuous action segmentation in dealing with time series problems. In this study, a multichannel fusion model is proposed by the idea of dividing. In this proposed architecture, a multichannel convolutional neural network (CNN) is used to enhance the ability to extract features at different scales, and then the fused features are fed into gated recurrent unit (GRU) for feature labeling and enhanced feature representation, through the learning of temporal relationships. Finally, the multichannel CNN-GRU model is designed using global average pooling (GAP) to connect the feature maps with the final classification. The model performance was conducted on three benchmark datasets of WISDM, UCI-HAR, and PAMAP2 with the accuracy of 96.41%, 96.67%, and 96.25% respectively. The results show that the proposed model demonstrates better activity detection capability than some of the reported results.

Fully Connected Generative Adversarial Network for Human Activity Recognition

Conditional Generative Adversarial Networks (CGAN) have shown great promise in generating synthetic data for sensor-based activity recognition. However, one key issue concerning existing CGAN is the design of the network architecture that affects sample quality. This study proposes an effective CGAN architecture that synthesizes higher quality samples than state-of-the-art CGAN architectures. This is achieved by combining convolutional layers with multiple fully connected networks in the generator’s input and discriminator’s output of the CGAN. We show the effectiveness of the proposed approach using elderly data for sensor-based activity recognition. Visual evaluation, similarity measure, and usability evaluation is used to assess the quality of generated samples by the proposed approach and validate its performance in activity recognition. In comparison to the state-of-the-art CGAN, the visual evaluation and similarity measure demonstrate that the proposed models’ synthetic data more accurately represents actual data and creates more variations in each synthetic data than the state-of-the-art approach respectively. The experimental stages of the usability evaluation, on the other hand, show a performance gain of 2.5%, 2.5%, 3.1%, and 4.4% over the state-of-the-art CGAN when using synthetic samples by the proposed architecture.