Sensor-based Human Activity Recognition Research Articles

Sensor Data Augmentation by Resampling in Contrastive Learning for Human Activity Recognition

While deep learning models have contributed to the advancement of sensor-based human activity recognition (HAR), it usually requires large amounts of annotated sensor data to extract robust features. To alleviate the limitations of data annotation, contrastive learning has been applied to sensor-based HAR. One of the essential factors of contrastive learning is data augmentation, significantly impacting the performance of pretraining. However, current popular augmentation methods do not achieve competitive performance in contrastive learning for sensor-based HAR. Motivated by this issue, we propose a new sensor data augmentation method by resampling, which introduces variable domain information and simulates realistic activity data by varying the sampling frequency to maximize the coverage of the sampling space. The resampling augmentation method was evaluated in supervised learning and contrastive learning [SimCLR for HAR (SimCLRHAR) and MoCo for HAR (MoCoHAR)]. In the experiment, we use four datasets, UCI-HAR, MotionSense, USC-HAD, and MobiAct, using the mean F1-score as the evaluation metric for downstream tasks. The experimental results show that the resampling data augmentation outperforms all state-of-the-art augmentation methods in supervised learning and contrastive learning with a small amount of labeled data. The results also demonstrate that not all data augmentation methods have positive effects in contrastive learning frameworks.

Federated Clustering and Semi-Supervised learning: A new partnership for personalized Human Activity Recognition

Federated Learning (FL) is currently studied by several research groups as a promising paradigm for sensor-based Human Activity Recognition (HAR) to mitigate the privacy and scalability issues of classic centralized approaches. However, in the HAR domain, data is non-independently and identically distributed (non-IID), and personalization is one of the major challenges. Federated Clustering has been recently proposed to mitigate this issue by creating specialized global models for groups of similar users. While this approach significantly improves personalization, it assumes that labeled data are available on each client. In this work, we propose SS-FedCLAR, a novel HAR framework that combines Federated Clustering and Semi-Supervised learning. In SS-FedCLAR , each client uses a combination of active learning and label propagation to provide pseudo labels to a large amount of unlabeled data, which is then used to collaboratively train a Federated Clustering model. We evaluated SS-FedCLAR on two well-known public datasets, showing that it outperforms existing semi-supervised FL solutions while reaching recognition rates similar to fully-supervised FL approaches.

TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation

Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human–computer interaction applications. However, most of them are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, even though several recent existing works considered the correlations between different sensor positions, they still face significant challenging problems with performance degradation due to large gaps in the distribution of training and test data, and behavioral differences between subjects. In this work, we propose a novel Transformer-based Adversarial learning framework for human activity recognition using wearable sensors via Self-KnowledgE Distillation (TASKED), that accounts for individual sensor orientations and spatial and temporal features. The proposed method is capable of learning cross-domain embedding feature representations from multiple subjects datasets using adversarial learning and the maximum mean discrepancy (MMD) regularization to align the data distribution over multiple domains. In the proposed method, we adopt the teacher-free self-knowledge distillation to improve the stability of the training procedure and the performance of human activity recognition. Experimental results show that TASKED not only outperforms state-of-the-art methods on the four real-world public HAR datasets (alone or combined) but also improves the subject generalization effectively.

Incremental Learning of Human Activities in Smart Homes.

Sensor-based human activity recognition has been extensively studied. Systems learn from a set of training samples to classify actions into a pre-defined set of ground truth activities. However, human behaviours vary over time, and so a recognition system should ideally be able to continuously learn and adapt, while retaining the knowledge of previously learned activities, and without failing to highlight novel, and therefore potentially risky, behaviours. In this paper, we propose a method based on compression that can incrementally learn new behaviours, while retaining prior knowledge. Evaluation was conducted on three publicly available smart home datasets.

Sensor-based Human Activity Recognition Using Graph LSTM and Multi-task Classification Model

This paper explores human activities recognition from sensor-based multi-dimensional streams. Recently, deep learning-based methods such as LSTM and CNN have achieved important progress in practical application scenarios. However, in most previous deep learning-based methods exist potential challenges such as class imbalance and multi-modal heterogeneity with time and sensor signals. To handle those problems, we propose a graph LSTM and Metric Learning model (GLML) with multiple construction graph fusion by modeling the sensor-aspect signals and the graph-aspect activities. GLML is a semi-supervised co-training architecture, which can be seen as several iteratively pseudo-labels sampling processing in the unlabeled data. Specifically, we construct three graphs to capture the different relations in each timestamp. Meanwhile, the graph attention model and attention mechanism are proposed to integrate multiple graph interactions for different sensor signals. Furthermore, to obtain a fixed representation of hidden state units and their neighboring nodes, we introduce the Graph LSTM to learn the graph-aspect relations from graph-structured constructed graphs. Notably, we propose a multi-task classification model combining loss function for classification distribution with deep metric learning to enhance the representation ability of the multi-modal sensor data. Experimental results on three public datasets demonstrate that our proposed GLML model has at least 2.44% improved in average against the state-of-the-art methods.

Matched Filter Interpretation of CNN Classifiers with Application to HAR.

Time series classification is an active research topic due to its wide range of applications and the proliferation of sensory data. Convolutional neural networks (CNNs) are ubiquitous in modern machine learning (ML) models. In this work, we present a matched filter (MF) interpretation of CNN classifiers accompanied by an experimental proof of concept using a carefully developed synthetic dataset. We exploit this interpretation to develop an MF CNN model for time series classification comprising a stack of a Conv1D layer followed by a GlobalMaxPooling layer acting as a typical MF for automated feature extraction and a fully connected layer with softmax activation for computing class probabilities. The presented interpretation enables developing superlight highly accurate classifier models that meet the tight requirements of edge inference. Edge inference is emerging research that addresses the latency, availability, privacy, and connectivity concerns of the commonly deployed cloud inference. The MF-based CNN model has been applied to the sensor-based human activity recognition (HAR) problem due to its significant importance in a broad range of applications. The UCI-HAR, WISDM-AR, and MotionSense datasets are used for model training and testing. The proposed classifier is tested and benchmarked on an android smartphone with average accuracy and scores of 98% and 97%, respectively, which outperforms state-of-the-art HAR methods in terms of classification accuracy and run-time performance. The proposed model size is less than 150 KB, and the average inference time is less than 1 ms. The presented interpretation helps develop a better understanding of CNN operation and decision mechanisms. The proposed model is distinguished from related work by jointly featuring interpretability, high accuracy, and low computational cost, enabling its ready deployment on a wide set of mobile devices for a broad range of applications.

A Novel CNN, Bidirectional Long-Short Term Memory, and Gated Recurrent Unit-Based Hybrid Approach for Human Activity Recognition

Human activity recognition (HAR) is a crucial and challenging classification task for a range of applications from surveillance to assistance. Existing sensor-based HAR systems have limited training data availability and lack fast and accurate methods for robust and rapid activity recognition. In this paper, a novel hybrid HAR technique based on CNN, bi-directional long short-term memory, and gated recurrent units is proposed that can accurately and quickly recognize new human activities with a limited training set and high accuracy. The experiment was conducted on UCI Machine Learning Repository's MHEALTH dataset to analyze the effectiveness of the proposed method. The confusion matrix and accuracy score are utilized to gauge the performance of the presented model. Experiments indicate that the proposed hybrid approach for human activity recognition integrating CNN, bi-directional LSTM, and gated recurrent outperforms computing complexity and efficiency. The overall findings demonstrate that the proposed hybrid model performs exceptionally well, with enhanced accuracy of 94.68%.

HIT HAR: Human Image Threshing Machine for Human Activity Recognition Using Deep Learning Models.

In recent days, research in human activity recognition (HAR) has played a significant role in healthcare systems. The accurate activity classification results from the HAR enhance the performance of the healthcare system with broad applications. HAR results are useful in monitoring a person's health, and the system predicts abnormal activities based on user movements. The HAR system's abnormal activity predictions provide better healthcare monitoring and reduce users' health issues. The conventional HAR systems use wearable sensors, such as inertial measurement unit (IMU) and stretch sensors for activity recognition. These approaches show remarkable performances to the user's basic activities such as sitting, standing, and walking. However, when the user performs complex activities, such as running, jumping, and lying, the sensor-based HAR systems have a higher degree of misclassification results due to the reading errors from sensors. These sensor errors reduce the overall performance of the HAR system with the worst classification results. Similarly, radiofrequency or vision-based HAR systems are not free from classification errors when used in real time. In this paper, we address some of the existing challenges of HAR systems by proposing a human image threshing (HIT) machine-based HAR system that uses an image dataset from a smartphone camera for activity recognition. The HIT machine effectively uses a mask region-based convolutional neural network (R-CNN) for human body detection, a facial image threshing machine (FIT) for image cropping and resizing, and a deep learning model for activity classification. We demonstrated the effectiveness of our proposed HIT machine-based HAR system through extensive experiments and results. The proposed HIT machine achieved 98.53% accuracy when the ResNet architecture was used as its deep learning model.

Dynamic Temperature Scaling in Contrastive Self-Supervised Learning for Sensor-Based Human Activity Recognition

The use of deep neural networks in sensor-based Human Activity Recognition has led to considerably improved recognition rates in comparison to more traditional techniques. Nonetheless, these improvements usually rely on collecting and annotating massive amounts of sensor data, a time-consuming and expensive task. In this paper, inspired by the impressive performance of Contrastive Learning approaches in Self-Supervised Learning settings, we introduce a novel method based on the SimCLR framework and a Transformer-like model. The proposed algorithm addresses the problem of negative pairs in SimCLR by using dynamic temperature scaling within a contrastive loss function. While the original SimCLR framework scales similarities between features of the augmented views by a constant temperature parameter, our method dynamically computes temperature values for scaling. Dynamic temperature is based on instance-level similarity values extracted by an additional model pre-trained on initial instances beforehand. The proposed approach demonstrates state-of-the-art performance on three widely used datasets in sensor-based HAR, namely MobiAct, UCI-HAR and USC-HAD. Moreover, it is more robust than the identical supervised models and models trained with constant temperature in semi-supervised and transfer learning scenarios.

Deep Transfer Learning Using Class Augmentation for Sensor-Based Human Activity Recognition

Transfer learning (TL) is an essential technique for human activity recognition (HAR) since collecting a large amount of labeled sensor data is cost-intensive. In the image recognition field, parameter-based TL using ImageNet has become the defacto standard; however, no multipurpose TL method has been established in HAR thus far. Therefore, we proposed a simple and novel technique, i.e., class augmentation (CA), which augments the diversity of the class categories of source domains based on the hypothesis that HAR datasets used as source domains have low diversity in class categories. Data augmentation (DA) enhances the diversity of input data, whereas CA enhances the diversity of output labels. We evaluated three types of CA using DA and HAR-specific auxiliary information (gender and sensor position). Our experiments using public HAR datasets revealed that using CA seems to enhance the transfer performance. Moreover, we confirmed that CA closes the interdomain distance between the source and target domains.

Dual-Branch Interactive Networks on Multichannel Time Series for Human Activity Recognition.

The popularity of convolutional architecture has made sensor-based human activity recognition (HAR) become one primary beneficiary. By simply superimposing multiple convolution layers, the local features can be effectively captured from multi-channel time series sensor data, which could output high-performance activity prediction results. On the other hand, recent years have witnessed great success of Transformer model, which uses powerful self-attention mechanism to handle long-range sequence modeling tasks, hence avoiding the shortcoming of local feature representations caused by convolutional neural networks (CNNs). In this paper, we seek to combine the merits of CNN and Transformer to model multi-channel time series sensor data, which might provide compelling recognition performance with fewer parameters and FLOPs based on lightweight wearable devices. To this end, we propose a new Dual-branch Interactive Network (DIN) that inherits the advantages from both CNN and Transformer to handle multi-channel time series for HAR. Specifically, the proposed framework utilizes two-stream architecture to disentangle local and global features by performing conv-embedding and patch-embedding, where a co-attention mechanism is used to adaptively fuse global-to-local and local-to-global feature representations. We perform extensive experiments on three mainstream HAR benchmark datasets including PAMAP2, WISDM, and OPPORTUNITY, which verify that our method consistently outperforms several state-of-the-art baselines, reaching an F1-score of 92.05%, 98.17%, and 91.55% respectively with fewer parameters and FLOPs. In addition, the practical execution time is validated on an embedded Raspberry Pi P3 system, which demonstrates that our approach is adequately efficient for real-time HAR implementations and deserves as a better alternative in ubiquitous HAR computing scenario. Our model code will be released soon.

MultiCNN-FilterLSTM: Resource-efficient sensor-based human activity recognition in IoT applications

With the recent advances in the Internet of Things (IoT) technologies, various human-centered applications have proliferated and improved the quality of users’ life. In the meantime, human activity recognition (HAR) has been considered as an essential component of human-centered applications in IoT due to its capability of providing substantial information about the user states. Whereas deep learning-based HAR methods have been recently proposed, there are still rooms to improve the HAR models, particularly from the perspective of IoT, as the models need to be precise but resource efficient. In this paper, to support IoT systems that require a resource-efficient model, we thereby propose a deep learning-based HAR model, called MultiCNN-FilterLSTM that combines a multihead convolutional neural network (CNN) with a long–short-term memory (LSTM) through a residual connection in which feature vectors are efficiently processed in hierarchical order. Accordingly, a novel approach to using LSTM cells, which we call filterwise LSTM (FilterLSTM), is proposed whereby the HAR model can learn the dependencies among the features at different hierarchical levels. The proposed HAR model has been exhaustively evaluated on two publicly available datasets. The proposed HAR model enhances the classification accuracy by 2.3%–4.4%, while it requires 21%–70% fewer operations than the state-of-the-art models. In addition, the proposed model is deployed to a Raspberry Pi 4 for further analysis in terms of deployment. The experimental results are presented to verify the merits of the proposed HAR model compared to the state-of-art models by exploiting hierarchical relationships and to deliver significant insights on the effectiveness of the proposed FilterLSTM from the perspective of the IoT system.

Two-stream transformer network for sensor-based human activity recognition

Human Activity Recognition (HAR) based on wearable devices has always been a hot topic in health applications, human-object interaction, and smart homes. Despite significant improvements achieved by convolutional neural networks, long short-term memory networks, transformer networks and various hybrid models, there are still two fundamental issues. First, the spatial–temporal dependencies of sensor signals are difficult to be effectively modeled. Second, in multimodal environments, sensors placed on various body positions contribute distinctly to the classification results. In this work, we propose a self-attention based Two-stream Transformer Network (TTN). In view of the former issue, we use two streams, named temporal stream and spatial stream, respectively, to extract the readings-over-time and time-over-readings features from sensor signals. These features extracted from two streams are complementary since the time-over-readings features are able to express additional information which cannot be captured from sensor signals directly. To deal with the latter issue, we assign attention weights to each sensor axis in the spatial channel based on their classification scores. It makes sense that different axis-readings with distinct recognition contributions caused by data heterogeneity be treated unequally. Extensive experiments on four available benchmark datasets (PAMAP2, Opportunity, USC–HAD, and Skoda) reveal that our proposed model is better suited for multimodal HAR than previous state-of-the-art methods.

Lifelong Adaptive Machine Learning for Sensor-Based Human Activity Recognition Using Prototypical Networks.

Continual learning (CL), also known as lifelong learning, is an emerging research topic that has been attracting increasing interest in the field of machine learning. With human activity recognition (HAR) playing a key role in enabling numerous real-world applications, an essential step towards the long-term deployment of such systems is to extend the activity model to dynamically adapt to changes in people’s everyday behavior. Current research in CL applied to the HAR domain is still under-explored with researchers exploring existing methods developed for computer vision in HAR. Moreover, analysis has so far focused on task-incremental or class-incremental learning paradigms where task boundaries are known. This impedes the applicability of such methods for real-world systems. To push this field forward, we build on recent advances in the area of continual learning and design a lifelong adaptive learning framework using Prototypical Networks, LAPNet-HAR, that processes sensor-based data streams in a task-free data-incremental fashion and mitigates catastrophic forgetting using experience replay and continual prototype adaptation. Online learning is further facilitated using contrastive loss to enforce inter-class separation. LAPNet-HAR is evaluated on five publicly available activity datasets in terms of its ability to acquire new information while preserving previous knowledge. Our extensive empirical results demonstrate the effectiveness of LAPNet-HAR in task-free CL and uncover useful insights for future challenges.

CNN-LSTM-Based Late Sensor Fusion for Human Activity Recognition in Big Data Networks

The technological advancement in sensor technology and pervasive computing has brought smart devices into our daily life. Due to the continuous connectivity of the internet with our everyday devices, researchers can deploy IoT sensors to health care and other applications, such as human activity recognition. Most of the state-of-the-art sensor-based human activity recognition systems can detect basic activities (such as standing, sitting, and walking), but they cannot accurately distinguish similar activities (ascending stairs or descending stairs). Such systems are not efficient for critical healthcare applications having complex activity sets. This paper proposes two sensor fusion approaches, i.e., position-based early and late sensor fusion using convolutional neural network (CNN) and convolutional long-short-term memory (CNN-LSTM). The performance of our proposed models is evaluated on two publicly available datasets. We also evaluated the effect of different normalization techniques on recognition accuracy. Our results show that the CNN-LSTM-based late sensor fusion model also improves the recognition accuracy of similar activities.

Multi-Branch Attention-Based Grouped Convolution Network for Human Activity Recognition Using Inertial Sensors

Recently, deep neural networks have become a widely used technology in the field of sensor-based human activity recognition and they have achieved good results. However, some convolutional neural networks lack further selection for the extracted features, or the networks cannot process the sensor data from different locations of the body independently and in parallel. Therefore, the accuracy of existing networks is not ideal. In particular, similar activities are easy to be confused, which limits the application of sensor-based HAR. In this paper, we propose a multi-branch neural network based on attention-based convolution. Each branch of the network consists of two layers of attention-based grouped convolution submodules. We introduce a dual attention mechanism that consists of channel attention and spatial attention to select the most important features. Sensor data collected at different positions of the human body are separated and fed into different network branches for training and testing independently. Finally, the multi-branch features are fused. We test the proposed network on three large datasets: PAMAP2, UT, and OPPORTUNITY. The experiment results show that our method outperforms the existing state-of-the-art methods.

Wearable Sensor-Based Human Activity Recognition with Hybrid Deep Learning Model

It is undeniable that mobile devices have become an inseparable part of human’s daily routines due to the persistent growth of high-quality sensor devices, powerful computational resources and massive storage capacity nowadays. Similarly, the fast development of Internet of Things technology has motivated people into the research and wide applications of sensors, such as the human activity recognition system. This results in substantial existing works that have utilized wearable sensors to identify human activities with a variety of techniques. In this paper, a hybrid deep learning model that amalgamates a one-dimensional Convolutional Neural Network with a bidirectional long short-term memory (1D-CNN-BiLSTM) model is proposed for wearable sensor-based human activity recognition. The one-dimensional Convolutional Neural Network transforms the prominent information in the sensor time series data into high level representative features. Thereafter, the bidirectional long short-term memory encodes the long-range dependencies in the features by gating mechanisms. The performance evaluation reveals that the proposed 1D-CNN-BiLSTM outshines the existing methods with a recognition rate of 95.48% on the UCI-HAR dataset, 94.17% on the Motion Sense dataset and 100% on the Single Accelerometer dataset.

RecurrentHAR: A Novel Transfer Learning-Based Deep Learning Model for Sequential, Complex, Concurrent, Interleaved, and Heterogeneous Type Human Activity Recognition

Smartphone sensor-based Human Activity Recognition (HAR) is a significant facilitator for many real-world applications such as smart homes, personal healthcare, and illness detection because it has the potential to discover activity patterns in daily life. While solutions for detecting sequential activities like walking or running are already matured, however recognizing additional forms of complex, concurrent, interleaved, and heterogeneous human activities remains a research challenge. Furthermore, several solutions have been developed for specific types of activity complexity. According to the best of our knowledge, there is no common solution for detecting all types of action-based activities. In this paper, we address the problem of recognizing all types of action-based human activities i.e. sequential, complex, concurrent, interleaved, and heterogeneous activities. The intention behind this paper is to present a generic methodology for HAR by leveraging transfer learning and evaluating the performance on a plethora of HAR datasets. In this regard, we have proposed a model called RecurrentHAR, which stands for Recurrent layers for HAR, a novel adversarial knowledge transfer approach that uses the Gated Recurrent Units (GRUs) architecture for smartphone sensor-based HAR. The performance of the proposed model is evaluated through extensive experiments using three public datasets namely WISDM (sequential activities), PAMAP2 (complex, concurrent, and interleaved activities), and KU-HAR (Heterogeneous activities), and the RecurrentHAR model has outperformed compared with other state-of-the-art approaches. The proposed model achieved 96.26%, 94.77%, and 98.98% of F1-score for WISDM, PAMAP2, and KU-HAR datasets respectively. The experimental results provide insight into the applicability of the proposed model and future research possibilities.

A new approach for physical human activity recognition from sensor signals based on motif patterns and long-short term memory

Numerous studies have been carried out in recent years on the recognition, tracking, and discrimination of human activities. Automatic recognition of physical activities is often referred to as human activity recognition (HAR). There are generally vision-based and sensor-based approaches for activity recognition. The computer vision-based approach generally works well in laboratory conditions, but it can fail in real-world problems due to clutter, variable light intensity, and contrast. Sensor-based HAR systems are realized by continuously monitoring and analyzing physiological signals measured from heterogeneous sensors connected to the person's body. In this study, the Motif Patterns (MP) approach, which extracts features from sensor signals, is proposed for HAR. The success of the HAR systems depends on the effectiveness of the features extracted from the signals. The LSTM network is a special kind of recurrent neural network that has been used to make very successful predictions on time series data where long-term dependencies are. The LSTM network type offers a successful solution approach to solving long-term dependencies problems such as human activity recognition. The classification process was carried out with Long-Short Term Memory (LSTM) using MP features extracted from accelerometer, gyroscope, and magnetometer sensor signals. A large dataset of 9120 signals was used to test the proposed approach. A high success rate of 98.42 % was achieved with the proposed MP + LSTM method. As a result, it has been seen that the proposed approach has been obtained with a high success rate for HAR using sensor signals.

Few shot learning-based fast adaptation for human activity recognition

Sensor-based human activity recognition (HAR) is a common application in the fields of mobile computing and pattern recognition. Existing approaches and models of HAR can present ideal recognition performance only in well-designed, specific, and deterministic scenarios. However, in real scenes, new types of activities, new human bodies that performs activities, and other new situations are encountered. When new situations arise, it is difficult to collect sufficient and high-quality data in time. Thus, the existing approaches and models suffer from a lack of interoperability and scalability. To address these challenges, this study proposes a Model-agnostic Meta Learnings(MAML)-Coarse-Fine Convolutional Neural Networks(CFCNN) mixed task strategy to achieve fast adaptation to the human activity recognition under new situations. This is a novel method that incorporates shot learning to recognize tasks in situations where several kinds of new scenarios exist. First, some modifications were carried out on traditional MAML for multi-scale feature extraction, and a mixed task strategy was adopted during training. The proposed method improves the generalization ability of the model, and is capable of learning quickly when dealing with diffirent new tasks. Finally, the model was compared with other models when facing two new scenes of a new activity category and a new human body. The results showed that the accuracy of the proposed MAML-CFCNN based on the mixed task strategy is higher than that of the state-of-the-art methods, including MAML, First-Order MAML (FOMAML), REPTILE, and so on.