Sensor-based Activity Research Articles

Data augmentation aided excavator activity recognition using deep convolutional conditional generative adversarial networks

Excavator activity recognition is a crucial task with significant practical applications such as assessing production efficiency, injury prevention, and enabling intelligent control. However, implementing deep learning-based activity recognition methods presents considerable challenges due to the requirement of large training datasets. And sensor-based activity recognition demands more convenient and reliable sensors. This paper proposes a generalized data augmentation framework for excavator activity recognition. Main pump pressure and flow signals are generated using Deep Convolutional Conditional Generative Adversarial Networks (DC-CGAN). The proposed framework integrates Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) architectures for activity recognition. Results indicate that the framework outperforms traditional data augmentation methods regarding the quality of generated data and shallow and single deep networks regarding model accuracy and generalization. Additionally, the framework was implemented on three sizes of excavators and a wheel loader, demonstrating excellent versatility.

Deep similarity segmentation model for sensor-based activity recognition

Deep similarity segmentation model for sensor-based activity recognition

Deep Residual Network with a CBAM Mechanism for the Recognition of Symmetric and Asymmetric Human Activity Using Wearable Sensors

Wearable devices are paramount in health monitoring applications since they provide contextual information to identify and recognize human activities. Although sensor-based human activity recognition (HAR) has been thoroughly examined, prior studies have yet to definitively differentiate between symmetric and asymmetric motions. Determining these movement patterns might provide a more profound understanding of assessing physical activity. The main objective of this research is to investigate the use of wearable motion sensors and deep convolutional neural networks in the analysis of symmetric and asymmetric activities. This study provides a new approach for classifying symmetric and asymmetric motions using a deep residual network incorporating channel and spatial convolutional block attention modules (CBAMs). Two publicly accessible benchmark HAR datasets, which consist of inertial measurements obtained from wrist-worn sensors, are used to assess the model’s efficacy. The model we have presented is subjected to thorough examination and demonstrates exceptional accuracy on both datasets. The ablation experiment examination also demonstrates noteworthy contributions from the residual mappings and CBAMs. The significance of recognizing basic movement symmetries in increasing sensor-based activity identification utilizing wearable devices is shown by the enhanced accuracy and F1-score, especially in asymmetric activities. The technique under consideration can provide activity monitoring with enhanced accuracy and detail, offering prospective advantages in diverse domains like customized healthcare, fitness tracking, and rehabilitation progress evaluation.

Large Receptive Field Attention: An Innovation in Decomposing Large-Kernel Convolution for Sensor-Based Activity Recognition

Large Receptive Field Attention: An Innovation in Decomposing Large-Kernel Convolution for Sensor-Based Activity Recognition

Sports activity (SA) recognition based on error correcting output codes (ECOC) and convolutional neural network (CNN)

The increasing use of motion sensors is causing major changes in the process of monitoring people's activities. One of the main applications of these sensors is the detection of sports activities, for example, they can be used to monitor the condition of athletes or analyze the quality of sports training. Although the existing sensor-based activity recognition systems can recognize basic activities such as: walking, running, or sitting; they don't perform well in recognizing different types of sports activities. This article introduces a new model based on machine learning (ML) techniques to more accurately distinguish between sports and everyday activities. In the proposed method, the necessary data to detect the type of activity is collected through his two sensors: an accelerometer and a gyroscope attached to a person's foot. For this purpose, the input signals are first preprocessed and then short-time Fourier transform (STFT) is used to describe the characteristics of each signal. In the next step, each STFT matrix is used as input to a convolutional neural network (CNN). This CNN describes various motion characteristics of the sensor in the form of vectors. Finally, a classification model based on error correction output code (ECOC) is used to classify the extracted features and detect the type of SA. The performance of the proposed AS recognition method is evaluated using the DSADS database and the results are compared with previous methods. Based on the results, the proposed method can recognize sports activities with an accuracy of 99.71. Furthermore, the performance of the proposed method based on precision and recall criteria are 99.72 and 99.71, respectively, which are better than the compared methods.

Deep neural networks for wearable sensor-based activity recognition in Parkinson’s disease: investigating generalizability and model complexity

BackgroundThe research gap addressed in this study is the applicability of deep neural network (NN) models on wearable sensor data to recognize different activities performed by patients with Parkinson’s Disease (PwPD) and the generalizability of these models to PwPD using labeled healthy data.MethodsThe experiments were carried out utilizing three datasets containing wearable motion sensor readings on common activities of daily living. The collected readings were from two accelerometer sensors. PAMAP2 and MHEALTH are publicly available datasets collected from 10 and 9 healthy, young subjects, respectively. A private dataset of a similar nature collected from 14 PwPD patients was utilized as well. Deep NN models were implemented with varying levels of complexity to investigate the impact of data augmentation, manual axis reorientation, model complexity, and domain adaptation on activity recognition performance.ResultsA moderately complex model trained on the augmented PAMAP2 dataset and adapted to the Parkinson domain using domain adaptation achieved the best activity recognition performance with an accuracy of 73.02%, which was significantly higher than the accuracy of 63% reported in previous studies. The model’s F1 score of 49.79% significantly improved compared to the best cross-testing of 33.66% F1 score with only data augmentation and 2.88% F1 score without data augmentation or domain adaptation.ConclusionThese findings suggest that deep NN models originating on healthy data have the potential to recognize activities performed by PwPD accurately and that data augmentation and domain adaptation can improve the generalizability of models in the healthy-to-PwPD transfer scenario. The simple/moderately complex architectures tested in this study could generalize better to the PwPD domain when trained on a healthy dataset compared to the most complex architectures used. The findings of this study could contribute to the development of accurate wearable-based activity monitoring solutions for PwPD, improving clinical decision-making and patient outcomes based on patient activity levels.

Activity Detection in Indoor Environments Using Multiple 2D Lidars.

In health monitoring systems for the elderly, a crucial aspect is unobtrusively and continuously monitoring their activities to detect potentially hazardous incidents such as sudden falls as soon as they occur. However, the effectiveness of current non-contact sensor-based activity detection systems is limited by obstacles present in the environment. To overcome this limitation, a straightforward yet highly efficient approach involves utilizing multiple sensors that collaborate seamlessly. This paper proposes a method that leverages 2D Light Detection and Ranging (Lidar) technology for activity detection. Multiple 2D Lidars are positioned in an indoor environment with varying obstacles such as furniture, working cohesively to create a comprehensive representation of ongoing activities. The data from these Lidars is concatenated and transformed into a more interpretable format, resembling images. A convolutional Long Short-Term Memory (LSTM) Neural Network is then used to process these generated images to classify the activities. The proposed approach achieves high accuracy in three tasks: activity detection, fall detection, and unsteady gait detection. Specifically, it attains accuracies of 96.10%, 99.13%, and 93.13% for these tasks, respectively. This demonstrates the efficacy and promise of the method in effectively monitoring and identifying potentially hazardous events for the elderly through 2D Lidars, which are non-intrusive sensing technology.

Channel Attention for Sensor-Based Activity Recognition: Embedding Features into all Frequencies in DCT Domain

Channel Attention for Sensor-Based Activity Recognition: Embedding Features into all Frequencies in DCT Domain

Phase Randomization: A data augmentation for domain adaptation in human action recognition

Human action recognition models often suffer from achieving both accurate recognition and subject independence when the amount of training data is limited. In this paper, we propose a data-efficient domain adaptation approach to learning a subject-agnostic action recognition classifier. The core component of our approach is a novel data augmentation called Phase Randomization. On the basis of the observation that individual body size is highly correlated with the amplitude component of the motion sequence, we disentangle the individuality and action features by using contrastive self-supervised learning with data augmentation that randomizes only the phase component of the motion sequence. This enables us to estimate the subject label of each motion sequence and to train a subject-agnostic action recognition classifier by performing adversarial learning with the estimated subject labels. We empirically demonstrate the superiority of our method on two different action recognition tasks (skeleton-based action recognition and sensor-based activity recognition).

Similarity Segmentation Approach for Sensor-Based Activity Recognition

Similarity Segmentation Approach for Sensor-Based Activity Recognition

Activity sampling in the construction industry: a review and research agenda

PurposeActivity sampling is a technique to monitor onsite labourers' time utilisation, which can provide helpful information for the management level to implement suitable labour productivity improvement strategies continuously. However, there needs to be a review paper that compiles research on activity sampling studies to give readers a thorough grasp of the research trend. Hence, this paper aims to investigate the activity sampling techniques applied in earlier research from the angles of activity categories formation, data collection methods and data analysis.Design/methodology/approachThe method used in this paper is a systematic review guided by the PRISMA framework. The search was conducted in Scopus and Web of Science. The inclusion and exclusion criteria were applied, selecting 70 articles published between 2011 and 2022 for data extraction and analysis. The analysis method involved a qualitative synthesis of the findings from the selected articles.FindingsActivity sampling is broadly divided into four stages: targeting trade, determining activity categories, data collection and data analysis. This paper divides the activity categories into three levels and classifies the data collection methods into manual observation, sensor-based activity sampling and computer vision-based activity sampling. The previous studies applied activity sampling for two construction management purposes: labour productivity monitoring and ergonomic safety monitoring. This paper also further discusses the scientific research gaps and future research directions.Originality/valueThis review paper contributes to the body of knowledge in construction management by thoroughly understanding current state-of-the-art activity sampling techniques and research gaps.

Device Orientation Independent Human Activity Recognition Model for Patient Monitoring Based on Triaxial Acceleration

Tracking a person’s activities is relevant in a variety of contexts, from health and group-specific assessments, such as elderly care, to fitness tracking and human–computer interaction. In a clinical context, sensor-based activity tracking could help monitor patients’ progress or deterioration during their hospitalization time. However, during routine hospital care, devices could face displacements in their position and orientation caused by incorrect device application, patients’ physical peculiarities, or patients’ day-to-day free movement. These aspects can significantly reduce algorithms’ performances. In this work, we investigated how shifts in orientation could impact Human Activity Recognition (HAR) classification. To reach this purpose, we propose an HAR model based on a single three-axis accelerometer that can be located anywhere on the participant’s trunk, capable of recognizing activities from multiple movement patterns, and, thanks to data augmentation, can deal with device displacement. Developed models were trained and validated using acceleration measurements acquired in fifteen participants, and tested on twenty-four participants, of which twenty were from a different study protocol for external validation. The obtained results highlight the impact of changes in device orientation on a HAR algorithm and the potential of simple wearable sensor data augmentation for tackling this challenge. When applying small rotations (<20 degrees), the error of the baseline non-augmented model steeply increased. On the contrary, even when considering rotations ranging from 0 to 180 along the frontal axis, our model reached a f1-score of 0.85±0.11 against a baseline model f1-score equal to 0.49±0.12.

A Boundary Consistency-Aware Multitask Learning Framework for Joint Activity Segmentation and Recognition With Wearable Sensors

With the development of industrial and sensing technology, sensor-based activity recognition has become a promising technology for informatics applications. However, in a typical activity recognition procedure, sensory data segmentation, usually considered a preprocess with sliding windows, rarely has been investigated and significantly affected the recognition performance. In this article, we propose a novel deep-learning method to jointly segment and recognize activities with wearable sensors. Our contributions are three-fold: First, we introduce a multistage temporal convolutional network for sample-level activity prediction to overcome the multiclass windows problem. Second, for alleviating oversegmentation errors, our model forms a multitask learning framework with a boundary prediction module to adjust the entire model’s gradients. Third, we innovatively propose a boundary consistency loss to enforce the consistency of the activity and boundary prediction. Our method shows impressive performance on three public datasets, especially achieving 16% improvement over very recently advanced competing methods with class-average F1-score on the Hospital dataset. The code of this work will be open source on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/xspc/Segmentation-Sensor-based-HAR</uri> .

Sensor-Based Activity Recognition Using Frequency Band Enhancement Filters and Model Ensembles.

Deep learning methods are widely used in sensor-based activity recognition, contributing to improved recognition accuracy. Accelerometer and gyroscope data are mainly used as input to the models. Accelerometer data are sometimes converted to a frequency spectrum. However, data augmentation based on frequency characteristics has not been thoroughly investigated. This study proposes an activity recognition method that uses ensemble learning and filters that emphasize the frequency that is important for recognizing a certain activity. To realize the proposed method, we experimentally identified the important frequency of various activities by masking some frequency bands in the accelerometer data and comparing the accuracy using the masked data. To demonstrate the effectiveness of the proposed method, we compared its accuracy with and without enhancement filters during training and testing and with and without ensemble learning. The results showed that applying a frequency band enhancement filter during training and testing and ensemble learning achieved the highest recognition accuracy. In order to demonstrate the robustness of the proposed method, we used four different datasets and compared the recognition accuracy between a single model and a model using ensemble learning. As a result, in three of the four datasets, the proposed method showed the highest recognition accuracy, indicating the robustness of the proposed method.

Radar Based Activity Recognition using CNN-LSTM Network Architecture

Human Activity Recognition based research has got intensified based on the evolving demand of smart systems. There has been already a lot of wearables, digital smart sensors deployed to classify various activities. Radar sensor-based Activity recognition has been an active research area during recent times. In order to classify the radar micro doppler signature images we have proposed a approach using Convolutional Neural Network-Long Short Term Memory (CNN-LSTM). Convolutional Layer is used to update the filter values to learn the features of the radar images. LSTM Layer enhances the temporal information besides the features obtained through Convolutional Neural Network. We have used a dataset published by University of Glasgow that captures six activities for 56 subjects under different ages, which is a first of its kind dataset unlike the signals captured under controlled lab environment. Our Model has achieved 96.8% for the training data and 93.5% for the testing data. The proposed work has outperformed the existing traditional deep learning Architectures.

Discriminating pathological, reproductive or stress conditions in cows using machine learning on sensor-based activity data

Precision livestock farming (PLF) technologies help to detect animals experiencing specific conditions such as illness, stress, or oestrus. PLF systems either detect one or more conditions at a time, or several conditions together but without distinguishing them. The aim of this work was to detect and classify a wide range of conditions of health, stress and reproduction based on cow daily activity using machine learning on time series data derived from sensors. We used five datasets (120,000 cow*days) from experimental or commercial farms, with 28 to 300 Holstein cows per dataset and observations lasting from 1 to 12 months. Caretakers checked the animals every day and recorded any specific cow condition related to disease (lameness, mastitis, acidosis, accident, and so on), oestrus, calving, as well as potential stress due to mixing or handling. We labelled a number of days before and after the specific condition during which the behaviour might be altered. Sensors (location systems or accelerometers) captured the per-hour duration of key cow activities - eating, resting, in alleys - that served to compute the activity level. The distribution of the activity level in each 24 h time series was described by 21 time-domain or frequency-domain features. We then ran the random forest algorithm to relate the features of the time series to animal condition (1000 trees; 10 random splits of 70 % train / 30 % test subsets; oversampling applied on train subsets to reduce the imbalance between the occurrences of cow conditions). The most important features in the random forest algorithm were, in descending order: maximum, 90% quantile, Fourier harmonics 2 and 1, 50 % and 75 % quantiles, Fourier harmonics 3 and 4. This approach correctly classified over 90 % of the control series (i.e. with no specific condition) and classified 24 % of the specific condition-labelled series as non-control series. The condition-labelled series can be confused with control series and to a lesser extent with series labelled disturbances or oestrus. There was 56 %–86 % probability of successfully detecting at least one 24 h series around a disease, oestrus, or calving. The detection can occur 1–2 days before the day caretakers noticed the event. This study shows that machine learning on time series can detect and discriminate several conditions of cow impaired welfare and could thus be implemented in PLF systems to facilitate herd management. The sensitivity of the method must be improved before it can be spread out on farms.

Improving Inertial Sensor-Based Activity Recognition in Neurological Populations.

Inertial sensor-based human activity recognition (HAR) has a range of healthcare applications as it can indicate the overall health status or functional capabilities of people with impaired mobility. Typically, artificial intelligence models achieve high recognition accuracies when trained with rich and diverse inertial datasets. However, obtaining such datasets may not be feasible in neurological populations due to, e.g., impaired patient mobility to perform many daily activities. This study proposes a novel framework to overcome the challenge of creating rich and diverse datasets for HAR in neurological populations. The framework produces images from numerical inertial time-series data (initial state) and then artificially augments the number of produced images (enhanced state) to achieve a larger dataset. Here, we used convolutional neural network (CNN) architectures by utilizing image input. In addition, CNN enables transfer learning which enables limited datasets to benefit from models that are trained with big data. Initially, two benchmarked public datasets were used to verify the framework. Afterward, the approach was tested in limited local datasets of healthy subjects (HS), Parkinson's disease (PD) population, and stroke survivors (SS) to further investigate validity. The experimental results show that when data augmentation is applied, recognition accuracies have been increased in HS, SS, and PD by 25.6%, 21.4%, and 5.8%, respectively, compared to the no data augmentation state. In addition, data augmentation contributes to better detection of stair ascent and stair descent by 39.1% and 18.0%, respectively, in limited local datasets. Findings also suggest that CNN architectures that have a small number of deep layers can achieve high accuracy. The implication of this study has the potential to reduce the burden on participants and researchers where limited datasets are accrued.

Transformer With Bidirectional GRU for Nonintrusive, Sensor-Based Activity Recognition in a Multiresident Environment

Several techniques for human activity recognition (HAR) in a smart indoor environment have been developed and improved along with the rapid advancement of sensor technologies. However, recognizing multiple people’s activities is still challenging due to the complexity of their activities, such as parallel and collaborative activities. To address these challenges, we propose a transformer with a bidirectional gated recurrent unit (GRU) deep learning (DL) method, called TRANS-BiGRU, to efficiently learn and recognize different types of activities performed by multiple residents. We compare the proposed model with the state-of-the-art models and various DL models, such as Ensemble2LSTM (Ens2-LSTM), bidirectional GRUs (Bi-GRU), and traditional machine learning (ML) models, such as support vector machine (SVM). Our experimental results based on the center for advanced studies in adaptive system and ARAS public data sets show that our model significantly outperforms the existing models for complex activity recognition of multiple residents.

Triple Cross-Domain Attention on Human Activity Recognition Using Wearable Sensors

Efficiently identifying activities of daily living (ADL) provides very important contextual information that is able to improve the effectiveness of various sports tracking and healthcare applications. Recently, attention mechanism that selectively focuses on time series signals has been widely adopted in sensor based human activity recognition (HAR), which can enhance interesting target activity and ignore irrelevant background activity. Several attention mechanisms have been investigated, which achieve remarkable performance in HAR scenario. Despite their success, prior these attention methods ignore the cross-interaction between different dimensions. In the paper, in order to avoid above shortcoming, we present a triplet cross-dimension attention for sensor-based activity recognition task, where three attention branches are built to capture the cross-interaction between sensor dimension, temporal dimension and channel dimension. The effectiveness of triplet attention method is validated through extensive experiments on four public HAR dataset namely UCI-HAR, PAMAP2, WISDM and UNIMIB-SHAR as well as the weakly labeled HAR dataset. Extensive experiments show consistent improvements in classification performance with various backbone models such as plain CNN and ResNet, demonstrating a good generality ability of the triplet attention. Visualization analysis is provided to support our conclusion, and actual implementation is evaluated on a Raspberry Pi platform.

Generic semi-supervised adversarial subject translation for sensor-based activity recognition

Performance of Human Activity Recognition (HAR) models, particularly deep neural networks, is highly contingent upon the availability of the massive amount of annotated training data. Though, data collection and manual labeling in the HAR domain are prohibitively expensive due to human resource dependence in both steps. Hence, domain adaptation techniques are proposed to adapt the knowledge from the existing source of data. More recently, adversarial transfer learning methods have shown promising results for visual classification, yet limited for HAR problems, which are still prone to the unfavorable effects of the imbalanced distribution of samples. This paper presents a novel generic semi-supervised approach that takes advantage of the adversarial framework to tackle these shortcomings by leveraging knowledge from annotated samples exclusively from the source subject and unlabeled ones of the target subject. An extensive subject translation experiments is conducted on three large, middle, and small-size datasets with different levels of imbalance to assess the robustness of the proposed model to the scale as well as the imbalance in the data. The results demonstrate the effectiveness of our proposed algorithms over state-of-the-art methods, which led to up to 13%, 4%, and 13% improvement of our high-level activities recognition metrics for Opportunity, LISSI, and PAMAP2 datasets, respectively.