Sensor-based Human Activity Recognition Research Articles

Privacy preservation in sensor-based Human Activity Recognition through autoencoders for low-power IoT devices

Human activity recognition is increasingly recognized as a key task in many applications. However, gathering data from the variety of sensors commonly available on end devices risks compromising user’s privacy when signals are transmitted to more powerful computing units for inference offloading. It is therefore important to design and implement strategies that could prevent privacy breaches without impairing the capability of the system of recognizing activity patterns, and by taking into account the energy constraints of low-power devices. In this work, we propose an energy-aware approach aimed at preserving the privacy of users during inference of human activities. The proposed method is based on a deep learning autoencoder trained to process the signal in order to remove the most sensitive information regarding privacy attributes, without significantly impacting classification accuracy. We also perform a thorough architecture’s parameter tuning of the designed system to enable its implementation on a low-power platform, which we also characterize in terms of energy expenditure. Experimental results show that this system is capable of effectively transforming the signal in order to prevent the inference of sensitive attributes (i.e. weight, height, age, and gender) and it can be conveniently implemented on a constrained embedded system at different levels of the trade-off between accuracy and energy consumption. Indeed, a complete obfuscation of sensitive attributes can be achieved at the cost of a marginal reduction in classification accuracy (5% at most), with an expenditure of around 165 mJ for an execution time of around 30ms needed during the signal transformation step.

Detection and Validation of Macro-Activities in Human Inertial Signals Using Graph Link Prediction.

With the continuous development of new wearable devices, sensor-based human activity recognition is enjoying enormous popularity in research and industry. The signals from inertial sensors allow for the detection, classification, and analysis of human activities such as jogging, cycling, or swimming. However, human activity recognition is often limited to basic activities that occur in short, predetermined periods of time (sliding windows). Complex macro-activities, such as multi-step sports exercises or multi-step cooking recipes, are still only considered to a limited extent, while some works have investigated the classification of macro-activities, the automated understanding of how the underlying micro-activities interact remains an open challenge. This study addresses this gap through the application of graph link prediction, a well-known concept in graph theory and graph neural networks (GNNs). To this end, the presented approach transforms micro-activity sequences into micro-activity graphs that are then processed with a GNN. The evaluation on two derived real-world data sets shows that graph link prediction enables the accurate identification of interactions between micro-activities and the precise validation of composite macro-activities based on learned graph embeddings. Furthermore, this work shows that GNNs can benefit from positional encodings in sequence recognition tasks.

Active contrastive coding reducing label effort for sensor-based human activity recognition

Human activity recognition (HAR) plays a crucial role in remotely monitoring the health of the elderly. Human annotation is time-consuming and expensive, especially for abstract sensor data. Contrastive learning can extract robust features from weakly annotated data to promote the development of sensor-based HAR. However, current research mainly focuses on the exploration of data augmentation methods and pre-trained models, disregarding the impact of data quality on label effort for fine-tuning. This paper proposes a novel active contrastive coding model that focuses on using an active query strategy to evenly select small, high-quality samples in downstream tasks to complete the update of the pre-trained model. The proposed uncertainty-based balanced query strategy mines the most indistinguishable hard samples according to the data posterior probability in the unlabeled sample pool, and imposes class balance constraints to ensure equilibrium in the labeled sample pool. Extensive experiments have shown that the proposed method consistently outperforms several state-of-the-art baselines on four mainstream HAR benchmark datasets (UCI, WISDM, MotionSense, and USCHAD). With approximately only 10% labeled samples, our method achieves impressive F1-scores of 98.54%, 99.34%, 98.46%, and 87.74%, respectively.

Domain Adaptation for Sensor-Based Human Activity Recognition with a Graph Convolutional Network

Sensor-based human activity recognition (HAR) plays a fundamental role in various mobile application scenarios, but the model performance of HAR heavily relies on the richness of the dataset and the completeness of data annotation. To address the shortage of comprehensive activity types in collected datasets, we adopt the domain adaptation technique with a graph neural network-based approach by incorporating an adaptive learning mechanism to enhance the action recognition model’s generalization ability, especially when faced with limited sample sizes. To evaluate the effectiveness of our proposed approach, we conducted experiments using three well-known datasets: MHealth, PAMAP2, and TNDA. The experimental results demonstrate the efficacy of our approach in sensor-based HAR tasks, achieving impressive average accuracies of 98.88%, 98.58%, and 97.78% based on the respective datasets. Furthermore, we conducted transfer learning experiments to address the domain adaptation problem. These experiments revealed that our proposed model exhibits exceptional transferability and distinguishing ability, even in scenarios with limited available samples. Thus, our approach offers a practical and viable solution for sensor-based HAR tasks.

Wireless body area sensor networks based human activity recognition using deep learning

In the healthcare sector, the health status and biological, and physical activity of the patient are monitored among different sensors that collect the required information about these activities using Wireless body area network (WBAN) architecture. Sensor-based human activity recognition (HAR), which offers remarkable qualities of ease and privacy, has drawn increasing attention from researchers with the growth of the Internet of Things (IoT) and wearable technology. Deep learning has the ability to extract high-dimensional information automatically, making end-to-end learning. The most significant obstacles to computer vision, particularly convolutional neural networks (CNNs), are the effect of the environment background, camera shielding, and other variables. This paper aims to propose and develop a new HAR system in WBAN dependence on the Gramian angular field (GAF) and DenseNet. Once the necessary signals are obtained, the input signals undergo pre-processing through artifact removal and median filtering. In the initial stage, the time series data captured by the sensors undergoes a conversion process, transforming it into 2-dimensional images by using the GAF algorithm. Then, DenseNet automatically makes the processes and integrates the data collected from diverse sensors. The experiment results show that the proposed method achieves the best outcomes in which it achieves 97.83% accuracy, 97.83% F-measure, and 97.64 Matthews correlation coefficient (MCC).

Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity Recognition.

Human activity recognition (HAR) based on wearable sensors has emerged as a low-cost key-enabling technology for applications such as human-computer interaction and healthcare. In wearable sensor-based HAR, deep learning is desired for extracting human active features. Due to the spatiotemporal dynamic of human activity, a special deep learning network for recognizing the temporal continuous activities of humans is required to improve the recognition accuracy for supporting advanced HAR applications. To this end, a residual multifeature fusion shrinkage network (RMFSN) is proposed. The RMFSN is an improved residual network which consists of a multi-branch framework, a channel attention shrinkage block (CASB), and a classifier network. The special multi-branch framework utilizes a 1D-CNN, a lightweight temporal attention mechanism, and a multi-scale feature extraction method to capture diverse activity features via multiple branches. The CASB is proposed to automatically select key features from the diverse features for each activity, and the classifier network outputs the final recognition results. Experimental results have shown that the accuracy of the proposed RMFSN for the public datasets UCI-HAR, WISDM, and OPPORTUNITY are 98.13%, 98.35%, and 93.89%, respectively. In comparison with existing advanced methods, the proposed RMFSN could achieve higher accuracy while requiring fewer model parameters.

Optimizing Deep LSTM Model through Hyperparameter Tuning for Sensor-Based Human Activity Recognition in Smart Home

Optimizing Deep LSTM Model through Hyperparameter Tuning for Sensor-Based Human Activity Recognition in Smart Home

A hybrid TCN-GRU model for classifying human activities using smartphone inertial signals.

Recognising human activities using smart devices has led to countless inventions in various domains like healthcare, security, sports, etc. Sensor-based human activity recognition (HAR), especially smartphone-based HAR, has become popular among the research community due to lightweight computation and user privacy protection. Deep learning models are the most preferred solutions in developing smartphone-based HAR as they can automatically capture salient and distinctive features from input signals and classify them into respective activity classes. However, in most cases, the architecture of these models needs to be deep and complex for better classification performance. Furthermore, training these models requires extensive computational resources. Hence, this research proposes a hybrid lightweight model that integrates an enhanced Temporal Convolutional Network (TCN) with Gated Recurrent Unit (GRU) layers for salient spatiotemporal feature extraction without tedious manual feature extraction. Essentially, dilations are incorporated into each convolutional kernel in the TCN-GRU model to extend the kernel's field of view without imposing additional model parameters. Moreover, fewer short filters are applied for each convolutional layer to alleviate excess parameters. Despite reducing computational cost, the proposed model utilises dilations, residual connections, and GRU layers for longer-term time dependency modelling by retaining longer implicit features of the input inertial sequences throughout training to provide sufficient information for future prediction. The performance of the TCN-GRU model is verified on two benchmark smartphone-based HAR databases, i.e., UCI HAR and UniMiB SHAR. The model attains promising accuracy in recognising human activities with 97.25% on UCI HAR and 93.51% on UniMiB SHAR. Since the current study exclusively works on the inertial signals captured by smartphones, future studies will explore the generalisation of the proposed TCN-GRU across diverse datasets, including various sensor types, to ensure its adaptability across different applications.

LHAR: Lightweight Human Activity Recognition on Knowledge Distillation.

Sensor-based Human Activity Recognition (HAR) is widely used in daily life and is the basic-level bridge to virtual healthcare in the metaverse. The current challenge is the low recognition accuracy for personalized users on smart wearable devices. The limited resource cannot support large deep learning models updated locally. Besides, integrating and transmitting sensor data to the cloud would reduce the efficiency. Considering the tradeoff between performance and complexity, we propose a Lightweight Human Activity Recognition (LHAR) framework. In LHAR, we combine the cross-people HAR task with the lightweight model task. LHAR framework is designed on the teacher-student architecture and the student network consists of multiple depthwise separable convolution layers to achieve fewer parameters. The dark knowledge distilled from the complex teacher model enhances the generalization ability of LHAR. To achieve effective knowledge distillation, we propose two optimization methods. Firstly, we train the teacher model by ensemble learning to promote teacher performance. Secondly, a multi-channel data augmentation method is proposed for the diversity of the dataset, which is a plug-in operation for the ensemble teacher model. In the experiments, we compare LHAR with state-of-art models in comparison evaluation, ablation study and the hyperparameter analysis, which proves the better performance of LHAR in efficiency and effectiveness.

Sensor-Based Human Activity Recognition Based on Multi-Stream Time-Varying Features with ECA-Net Dimensionality Reduction

Sensor-Based Human Activity Recognition Based on Multi-Stream Time-Varying Features with ECA-Net Dimensionality Reduction

EGTCN: An Efficient Graph and Temporal Convolution Network for Sensor-Based Human Activity Recognition in Federated Learning

EGTCN: An Efficient Graph and Temporal Convolution Network for Sensor-Based Human Activity Recognition in Federated Learning

MLCNNwav: Multilevel Convolutional Neural Network With Wavelet Transformations for Sensor-Based Human Activity Recognition

Human activity recognition (HAR) is a rapidly growing field of research that aims to automatically identify and classify human motions and activities from different tracking devices, such as cameras and sensors. One of the most widely used sensor modalities for HAR is the smartphone, which has various sensors, such as gyroscopes, accelerometers, and GPS, that can provide rich information about a person’s movements and actions. HAR applications are essential for the Internet of Things (IoT) and smart home industries. We used the recent advances in deep learning techniques to develop a new HAR model for wearable sensors. The proposed model, MLCNNwav, relies on residual convolutional neural networks and one-dimensional trainable discrete wavelet transform. The multilevel CNN is designed to capture global features, whereas the wavelet transformation enhances the representation and generalization by learning activity-related features. Several deep learning approaches are compared to assess the superiority of the developed model. Four public benchmarks HAR datasets were used for the evaluation. The outcomes confirmed that the developed MLCNNwav recorded high accuracy rates on all datasets.

DWOSC: Dynamic Weight Optimization and Smoothness Constraint for Sensor-Based Human Activity Recognition

DWOSC: Dynamic Weight Optimization and Smoothness Constraint for Sensor-Based Human Activity Recognition

A Multi-Section Hierarchical Deep Neural Network Model for Time Series Classification: Applied to Wearable Sensor-Based Human Activity Recognition

A Multi-Section Hierarchical Deep Neural Network Model for Time Series Classification: Applied to Wearable Sensor-Based Human Activity Recognition

A profile similarity-based personalized federated learning method for wearable sensor-based human activity recognition

Wearable sensor-based human activity recognition (HAR) utilizes artificial intelligence models to analyze real-time data like accelerometer data to recognize daily human activities. While it greatly benefits the life of senior citizens and postoperative patients, it conventionally requires the collected data to be uploaded to a central server to train AI models, raising critical security and privacy concerns. Though Federated learning (FL) emerges as a viable way to cope with these problems, it is confronted by the data heterogeneity problem, where the varying activity patterns of different individuals result in non-identically distributed local data. Some FL models have been proposed to solve the data heterogeneity problem by leveraging the similarity between individuals to create a personalized global model for each individual. However, they are still limited by increased computation or unreliable relationships in the similarity computation. This study proposes a novel profile similarity-based personalized federated learning for wearable sensor-based HAR where the similarity between individuals can be reflected in their profile, such as age, gender, height, and weight. When personalizing a model for an individual, we compute the weighted sum of all clients’ local models, where the weight is determined by the similarity value computed from the profile. In this way, the local models from individuals who have higher similarity values will contribute more towards personalizing a model for a targeted individual than those who are less similar. Experiment results demonstrate that the proposed model outperformed the baseline FL and centralized learning on both RealWorld and SisFall datasets. We also discuss the tradeoff between privacy and personalization and FL's advantages over centralized learning.

Simple to Complex, Single to Concurrent Sensor-Based Human Activity Recognition: Perception and Open Challenges

Simple to Complex, Single to Concurrent Sensor-Based Human Activity Recognition: Perception and Open Challenges

Self-attention deep ConvLSTM with sparse-learned channel dependencies for wearable sensor-based human activity recognition

In this study, we propose a novel deep-learning architecture with sparse learning for human activity recognition. The proposed model contains 1D CNNs and LSTM layers with a self-attention mechanism to enhance a substantial number of time points in time-series data for human activity recognition systems. Based on the recent success of squeeze-and-excite (SE) networks, the proposed deep learning model utilizes the SE module to enhance channel-wise interdependencies, which in turn leads to a boost in performance. In addition, we utilized sparse learning to retrain only weak nodes and freeze stronger nodes in a fully connected layer prior to classification layer. Furthermore, we utilized an entropy-inspired formula to find sparsely located weaker nodes and validated our model on various datasets, including Opportunity, UCI-HAR, and WISDM. Herein, we present an extensive analysis and survey of state-of-the-art studies, in addition to our proposed research. For a fair comparison, we evaluated our deep learning architecture using various performance metrics and achieved better results; the proposed model outperformed state-of-the-art algorithms for human activity recognition.

Semantic Loss

Deep Learning models are a standard solution for sensor-based Human Activity Recognition (HAR), but their deployment is often limited by labeled data scarcity and models' opacity. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate these issues by infusing knowledge about context information into HAR deep learning classifiers. However, existing NeSy methods for context-aware HAR require computationally expensive symbolic reasoners during classification, making them less suitable for deployment on resource-constrained devices (e.g., mobile devices). Additionally, NeSy approaches for context-aware HAR have never been evaluated on in-the-wild datasets, and their generalization capabilities in real-world scenarios are questionable. In this work, we propose a novel approach based on a semantic loss function that infuses knowledge constraints in the HAR model during the training phase, avoiding symbolic reasoning during classification. Our results on scripted and in-the-wild datasets show the impact of different semantic loss functions in outperforming a purely data-driven model. We also compare our solution with existing NeSy methods and analyze each approach's strengths and weaknesses. Our semantic loss remains the only NeSy solution that can be deployed as a single DNN without the need for symbolic reasoning modules, reaching recognition rates close (and better in some cases) to existing approaches.

SF-Adapter

Wearable sensor-based human activity recognition (HAR) has gained significant attention due to the widespread use of smart wearable devices. However, variations in different subjects can cause a domain shift that impedes the scaling of the recognition model. Unsupervised domain adaptation has been proposed as a solution to recognize activities in new, unlabeled target domains by training the source and target data together. However, the need for accessing source data raises privacy concerns. Source-free domain adaptation has emerged as a practical setting, where only a pre-trained source model is provided for the unlabeled target domain. This setup aligns with the need for personalized activity model adaptation on target local devices. As the edge devices are resource-constrained with limited memory, it is crucial to take the computational efficiency, i.e., memory cost into consideration. In this paper, we develop a source-free domain adaptation framework for wearable sensor-based HAR, with a focus on computational efficiency for target edge devices. Firstly, we design a lightweight add-on module called adapter to adapt the frozen pre-trained model to the unlabeled target domain. Secondly, to optimize the adapter, we adopt a simple yet effective model adaptation method that leverages local representation similarity and prediction consistency. Additionally, we design a set of sample selection optimization strategies to select samples effective for adaptation and further enhance computational efficiency while maintaining adaptation performance. Our extensive experiments on three datasets demonstrate that our method achieves comparable recognition accuracy to the state-of-the-art source free domain adaptation methods with fewer than 1% of the parameters updated and saves up to 4.99X memory cost.

Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition

Human Activity Recognition (HAR) models often suffer from performance degradation in real-world applications due to distribution shifts in activity patterns across individuals. Test-Time Adaptation (TTA) is an emerging learning paradigm that aims to utilize the test stream to adjust predictions in real-time inference, which has not been explored in HAR before. However, the high computational cost of optimization-based TTA algorithms makes it intractable to run on resource-constrained edge devices. In this paper, we propose an Optimization-Free Test-Time Adaptation (OFTTA) framework for sensor-based HAR. OFTTA adjusts the feature extractor and linear classifier simultaneously in an optimization-free manner. For the feature extractor, we propose Exponential Decay Test-time Normalization (EDTN) to replace the conventional batch normalization (CBN) layers. EDTN combines CBN and Test-time batch Normalization (TBN) to extract reliable features against domain shifts with TBN's influence decreasing exponentially in deeper layers. For the classifier, we adjust the prediction by computing the distance between the feature and the prototype, which is calculated by a maintained support set. In addition, the update of the support set is based on the pseudo label, which can benefit from reliable features extracted by EDTN. Extensive experiments on three public cross-person HAR datasets and two different TTA settings demonstrate that OFTTA outperforms the state-of-the-art TTA approaches in both classification performance and computational efficiency. Finally, we verify the superiority of our proposed OFTTA on edge devices, indicating possible deployment in real applications. Our code is available at https://github.com/Claydon-Wang/OFTTA.