Continuous Activity Recognition Research Articles

A Novel Framework Based on Deep Learning Architecture for Continuous Human Activity Recognition with Inertial Sensors.

Frameworks for human activity recognition (HAR) can be applied in the clinical environment for monitoring patients' motor and functional abilities either remotely or within a rehabilitation program. Deep Learning (DL) models can be exploited to perform HAR by means of raw data, thus avoiding time-demanding feature engineering operations. Most works targeting HAR with DL-based architectures have tested the workflow performance on data related to a separate execution of the tasks. Hence, a paucity in the literature has been found with regard to frameworks aimed at recognizing continuously executed motor actions. In this article, the authors present the design, development, and testing of a DL-based workflow targeting continuous human activity recognition (CHAR). The model was trained on the data recorded from ten healthy subjects and tested on eight different subjects. Despite the limited sample size, the authors claim the capability of the proposed framework to accurately classify motor actions within a feasible time, thus making it potentially useful in a clinical scenario.

Classification Strategies for Radar-Based Continuous Human Activity Recognition with Multiple Inputs and Multi-Label Output

Classification Strategies for Radar-Based Continuous Human Activity Recognition with Multiple Inputs and Multi-Label Output

Real-Time Continuous Activity Recognition With a Commercial mmWave Radar

Real-Time Continuous Activity Recognition With a Commercial mmWave Radar

Deep Learning Techniques for Radar-Based Continuous Human Activity Recognition

Human capability to perform routine tasks declines with age and age-related problems. Remote human activity recognition (HAR) is beneficial for regular monitoring of the elderly population. This paper addresses the problem of the continuous detection of daily human activities using a mm-wave Doppler radar. In this study, two strategies have been employed: the first method uses un-equalized series of activities, whereas the second method utilizes a gradient-based strategy for equalization of the series of activities. The dynamic time warping (DTW) algorithm and Long Short-term Memory (LSTM) techniques have been implemented for the classification of un-equalized and equalized series of activities, respectively. The input for DTW was provided using three strategies. The first approach uses the pixel-level data of frames (UnSup-PLevel). In the other two strategies, a convolutional variational autoencoder (CVAE) is used to extract Un-Supervised Encoded features (UnSup-EnLevel) and Supervised Encoded features (Sup-EnLevel) from the series of Doppler frames. The second approach for equalized data series involves the application of four distinct feature extraction methods: i.e., convolutional neural networks (CNN), supervised and unsupervised CVAE, and principal component Analysis (PCA). The extracted features were considered as an input to the LSTM. This paper presents a comparative analysis of a novel supervised feature extraction pipeline, employing Sup-ENLevel-DTW and Sup-EnLevel-LSTM, against several state-of-the-art unsupervised methods, including UnSUp-EnLevel-DTW, UnSup-EnLevel-LSTM, CNN-LSTM, and PCA-LSTM. The results demonstrate the superiority of the Sup-EnLevel-LSTM strategy. However, the UnSup-PLevel strategy worked surprisingly well without using annotations and frame equalization.

Lightweight accurate trigger to reduce power consumption in sensor-based continuous human activity recognition

Wearable devices have become increasingly popular in recent years, and they offer a great opportunity for sensor-based continuous human activity recognition in real-world scenarios. However, one of the major challenges is their limited battery life. In this study, we propose an energy-aware human activity recognition framework for wearable devices based on a lightweight accurate trigger. The trigger acts as a binary classifier capable of recognizing, with maximum accuracy, the presence or absence of one of the interesting activities in the real-time input signal and it is responsible for starting the energy-intensive classification procedure only when needed. The measurement results conducted on a real wearable device show that the proposed approach can reduce energy consumption by up to 95% in realistic case studies, with a cost of performance deterioration of at most 1% or 2% compared to the traditional energy-intensive classification strategy.

A Survey on Radar-Based Continuous Human Activity Recognition

Radar-based human motion and activity recognition is currently a topic of great research interest, as the aging population increases and older individuals prefer an independent lifestyle. This technology has a wide range of applications, such as fall detection in assisted living, gesture recognition for human-machine interfaces, and many more. Numerous studies exist on various approaches for radar-based activity capture and classification. However, most of these employ rather artificial data, often obtained in laboratory environments, and typically collected under particular conditions. Specifically, most research so far has aimed at distinguishing a predefined set of single activities with a defined start, stop and duration. This paper aims at drawing the attention to a so far less researched issue, one that will be of vital importance for future real-world application of radar-based human activity recognition: continuous activity recognition, i.e. recognizing specific activities in a stream of several sequential activities with unknown duration and arbitrary transitions between different classes of activities. A review on the current state of the art in this relatively new topic is given, followed by a discussion on future research directions.

CSI-Based Human Continuous Activity Recognition Using GMM–HMM

Recently, device-free human activity recognition has become a research hotspot, and great progress has been made in ubiquitous computing. Among the different kinds of implementations, activity recognition based on WiFi channel state information (CSI) has attracted enormous attention for its superiority compared with conventional approaches. In this article, a device-free human continuous activity recognition system based on WiFi CSI is proposed. First, the CSI phase difference expansion matrix is constructed as a more obvious activity recognition feature, and a method based on threshold combined with labeling is used to achieve continuous activity segmentation. Then, the Gaussian mixture model–hidden Markov model (GMM–HMM) is used to model the CSI feature data of each activity, which is originally used for human 3-D skeleton-based activity recognition. The approach is of great value not only for its high accuracy compared with other classification approaches, such as long short-term memory (LSTM) and convolutional neural network (CNN), but also for its tremendous advantage that a pretty short CSI time series could be used to identify human activities, thus saving computer memory, reducing system calculation time greatly, and improving the error tolerance rate of the segmentation. Experiments on measured activity datasets and methods comparison demonstrate the effectiveness and superiority of the proposed system. The factors affecting system performance, such as the length of the CSI time series, have been discussed in this article.

Probabilistic Cascading Classifier for Energy-Efficient Activity Monitoring in Wearables

Advances in embedded systems have given rise to integrating several small-size health monitoring devices within daily human life. This trend led to an ongoing extension of wearable sensors in a broad range of applications. Wearable technologies, which are firmly connected with the human body, utilize sensors and machine learning to describe individuals’ physical or psychological routines through activity recognition and human movement. Since wearables are used all day long, the power consumption of these systems needs to be reasonably low. Current research considers that such machine learning methods are trained with fixed properties, including sensor sampling rate and statistical features computed from the time series data. However, in reality, wearables require continuous reconfiguration of their computational algorithms due to the personalized nature of human gait and movement. Furthermore, computational algorithms must become energy- and memory-efficient due to these embedded sensors’ limited power and memory. In this paper, we propose a resource-efficient framework for real-time, continuous, and on-node human activity recognition. Typically activity recognition problem is a multi-class classification problem. However, we suggest transforming this problem based on MET (Metabolic Equivalent of Task) into a hierarchical classification model, providing personalized structure for each individual. We discuss the design and construction of this new configurable classification paradigm. Our results demonstrate that the proposed probabilistic cascading system accuracy for different personalized scenarios varies between 94.5% and 96.9% in detecting activities using a limited memory, while power usage of the system is reduced by as high as 17.2% compared to the traditional methods.

Human Activity Recognition Based on the Fading Characteristics of the On-Body Channel

Several solutions using wearable devices for continuous Human Activity Recognition (HAR) have been proposed up to now, and HAR sensor-based methods are by far the most adopted. However, many data transmission obstacles prevent the viable operation of these systems, such as those related to deep channel fading, the shadowing of body links during movements, packet loss resulting from unpredictable and dynamic body movements, and environmental variations. In this work, a novel HAR approach based solely on on-body Received Signal Strength Indicator (RSSI) measurements of different body channels is presented to discriminate various human activities. In this perspective, a novel hierarchical threshold-based algorithm is proposed to classify RSSI data into two classes. The first one encompasses static movements. The second one groups dynamic movements. In this paper, the features provided to machine learning algorithms are unconventionnaly based on the statistical characterization parameters of the body channel in an indoor environment. These parameters include the average of the slow fading channel component and the fast fading component’s cumulative distribution function (CDF). In addition, the impact of the sliding window size adopted for the extraction of the slow fading component is investigated, as well as the impingement of the placement of the radio nodes on the body for the selection of the best on-body link for the transmission. It is shown that the proposed scheme provides higher accuracy compared to the state of the art radio-based HAR techniques and that the mislabeling problem is reduced by distinguishing between static and dynamic activities. Indeed, an approximate F1 score-based accuracy of 99% is obtained with a demeaning window size of 50 ms, considering the most informative channel.

Distributed radar fusion and recurrent networks for classification of continuous human activities

Continuous Human Activity Recognition (HAR) in arbitrary directions is investigated in this paper using a network of five spatially distributed pulsed Ultra-Wideband radars. While activities performed continuously and in unconstrained trajectories provide a more realistic and natural scenario for HAR, the network of radar sensors is proposed to address the issue of unfavourable or occluded perspectives when using only a single sensor. Different techniques to combine the relevant information from the multiple radars in the network are investigated, focussing on signal level fusion directly applied on Range-Time maps, and the selection of radar nodes based on location and velocity of the target derived from multilateration processing and tracking. Recurrent Neural Networks with and without bidirectionality are used to classify the activities based on the micro-Doppler (μD) spectrograms obtained for sensor fusion techniques. To assess classification performances, novel evaluation metrics accounting for the continuous nature of the sequence of activities and inherent imbalances in the dataset are proposed and compared with existing metrics. It is shown that the conventional accuracy metric may not capture all the important aspects for continuous HAR, and the proposed metrics can be considered for a more comprehensive evaluation.

Continuous Human Activity Recognition With Distributed Radar Sensor Networks and CNN–RNN Architectures

Unconstrained human activities recognition with a radar network is considered. A hybrid classifier combining both CNNs and RNNs for spatial-temporal pattern extraction is proposed. The two-dimensional CNNs (2D-CNNs) are first applied to the radar data to perform spatial feature extraction on the input spectrograms. Subsequently, gated recurrent units with bidirectional implementations are used to capture the long- and short-term temporal dependencies in the feature maps generated by the 2D-CNNs. Three NN-based data fusion methods were explored and compared to utilize the rich information provided by the different radar nodes. The performance of the proposed classifier was validated rigorously using the K-fold CV and L1PO method. Unlike competitive research, the dataset with continuous human activities with seamless inter-activity transitions that can occur at any time and unconstrained moving trajectories of the participants has been collected and used for evaluation purposes. Classification accuracy of about 90.8% is achieved for nine-class HAR by the proposed classifier with the halfway fusion method.

Continuous Human Activity Recognition through Parallelism LSTM with Multi-Frequency Spectrograms

According to the real-living environment, radar-based human activity recognition (HAR) is dedicated to recognizing and classifying a sequence of activities rather than individual activities, thereby drawing more attention in practical applications of security surveillance, health care and human–computer interactions. This paper proposes a parallelism long short-term memory (LSTM) framework with the input of multi-frequency spectrograms to implement continuous HAR. Specifically, frequency-division short-time Fourier transformation (STFT) is performed on the data stream of continuous activities collected by a stepped-frequency continuous-wave (SFCW) radar, generating spectrograms of multiple frequencies which introduce different scattering properties and frequency resolutions. In the designed parallelism LSTM framework, multiple parallel LSTM sub-networks are trained separately to extract different temporal features from the spectrogram of each frequency and produce corresponding classification probabilities. At the decision level, the probabilities of activity classification from these sub-networks are fused by addition as the recognition output. To validate the proposed method, an experimental data set is collected by using an SFCW radar to monitor 11 participants who continuously perform six activities in sequence with three different transitions and random durations. The validation results demonstrate that the average accuracies of the designed parallelism unidirectional LSTM (Uni-LSTM) and bidirectional LSTM (Bi-LSTM) based on five frequency spectrograms are 85.41% and 96.15%, respectively, outperforming traditional Uni-LSTM and Bi-LSTM networks with only a single-frequency spectrogram by 5.35% and 6.33% at least. Additionally, the recognition accuracy of the parallelism LSTM network reveals an upward trend as the number of multi-frequency spectrograms (namely the number of LSTM subnetworks) increases, and tends to be stable when the number reaches 4.

Continuous physical activity recognition for intelligent labour monitoring

Continuous physical activity recognition for intelligent labour monitoring

Contactless Continuous Activity Recognition based on Meta-Action Temporal Correlation in Mobile Environments

Continuous activity recognition (CAR) plays an important role in human daily indoor activity monitoring and can be widely used in smart home, human-computer interaction and user authentication. Due to the privacy issue and limited coverage of video signals, RF-based CAR has attracted more and more attention in recent years. This paper focuses on three key problems in RF-based CAR: denoising, segmentation and recognition. We present the design and implementation of a contactless and sensorless continuous activity recognition system, namely WiCheck. Our basic idea is to utilize the temporal correlation between two adjacent actions in continuous activity to eliminate the cumulative error in continuous activity segmentation. Firstly, the multi-layer optimized noise elimination method is used to decrease the environment interference. Secondly, a method based on dual-swing window is proposed to reduce the cumulative error of continuous activity segmentation. Finally, WiCheck is implemented in different indoor environments, and 6 continuous activity sequences are designed to evaluate and analyze the influencing factors. The continuous activity recognition accuracy of WiCheck to two actions and three actions can approach 90% and 75%, respectively.

CHARM-Deep: Continuous Human Activity Recognition Model Based on Deep Neural Network Using IMU Sensors of Smartwatch

In the present paper, an attempt was made to achieve high-performance continuous human activity recognition (CHAR) using deep neural networks. The present study focuses on recognizing different activities in a continuous stream, which means `back-to-back' consecutive set of activities, from only inertial measurement unit (IMU) sensors mounted on smartwatches. For that purpose, a new dataset called `CHAR-SW', which includes numerous streams of daily activities, was collected using smartwatches, and feature representations and network architectures were designed. Experimental comparisons using our own dataset and public datasets (Aruba and Tulum) have been performed. They demonstrated that cascading bidirectional long short-term memory (Bi-LSTM) with featured data performed well in offline mode from the viewpoints of accuracy, computational time, and storage space required. The input to the Bi-LSTM is a descriptor which composed of a stream of the following features: autocorrelation, median, entropy, and instantaneous frequency. Additionally, a novel technique to operate the CHAR system online was introduced and shown to be effective. Experimental results can be summarized as: the offline CHARM-Deep enhanced the accuracy compared with using raw data or the existing approaches, and it reduced the processing time by 86% at least relative to the time consumed in executing the Bi-LSTM classifier directly on the raw data. It also reduced storage space by approximately 97.77% compared with using raw data. The online evaluation shows that it can recognize activities in real-time with an accuracy of 91%.

Bi-LSTM Network for Multimodal Continuous Human Activity Recognition and Fall Detection

This paper presents a framework based on multi-layer bi-LSTM network (bidirectional Long Short-Term Memory) for multimodal sensor fusion to sense and classify daily activities’ patterns and high-risk events such as falls. The data collected in this work are continuous activity streams from FMCW radar and three wearable inertial sensors on the wrist, waist, and ankle. Each activity has a variable duration in the data stream so that the transitions between activities can happen at random times within the stream, without resorting to conventional fixed-duration snapshots. The proposed bi-LSTM implements soft feature fusion between wearable sensors and radar data, as well as two robust hard-fusion methods using the confusion matrices of both sensors. A novel hybrid fusion scheme is then proposed to combine soft and hard fusion to push the classification performances to approximately 96% accuracy in identifying continuous activities and fall events. These fusion schemes implemented with the proposed bi-LSTM network are compared with conventional sliding window approach, and all are validated with realistic “leaving one participant out” (L1PO) method (i.e. testing subjects unknown to the classifier). The developed hybrid-fusion approach is capable of stabilizing the classification performance among different participants in terms of reducing accuracy variance of up to 18.1% and increasing minimum, worst-case accuracy up to 16.2%.

Continuous Human Activity Recognition in Logistics from Inertial Sensor Data using Temporal Convolutions in CNN

Human activity recognition has been an important task for the research community. With the introduction of deep learning architectures, the performance of activity recognition algorithms has improved significantly. However, most of the research in this area has focused on activity recognition for health/assisted living with other applications being given less attention. This paper considers continuous activity recognition in logistics (order picking and packing operations) using a convolutional neural network with temporal convolutions on inertial measurement sensor data from the recently released LARa dataset. Four variants of the popular CNN-IMU are experimented upon and a discussion of the results is provided. The results indicate that temporal convolutions are able to achieve satisfactory performance for some activities (hand center and cart) whereas they perform poorly for the activities of stand and hand up.

HCAR: Human continuous activity recognition using latent structure features

AbstractWith the rapid development of the Internet of Things and the improvement of computing power, edge‐computing becomes an emergency computing paradigm that communicates between the terminal and cloud. One of the most representative works of edge‐computing is to achieve human‐activity recognition at the edge side, as it has lower latency and it could reduce transmission costs, compared with processing at the cloud side. However, existing approaches have many drawbacks: (1) they can merely recognize separated actions as it is incompetence for continuous activity recognition; and (2) they are not robust to action transformation and environmental noise due to the value‐feature‐based matching strategy. In this paper, we propose HCAR, a structure‐feature–based human continuous activity recognition system, which is insensitive to action transformation and environmental noise. Firstly, we leverage word2vec to embed the CSI sequences to CSI value space. Secondly, we select representative features from the embedded vectors and use HMM‐LDA to cluster them into different action categories. Lastly, for each new coming sequence, we calculate the Hellinger distance and bi‐modality coefficient to different categories and then identify the corresponding action(s). We implement HCAR by Intel 5300 NIC to evaluate the activity recognition precision in different cases. The experiments show that HCAR can recognize actions corresponding to the unsegmented CSI sequence with high accuracy, ie, >90%.

A novel framework of continuous human-activity recognition using Kinect

Abstract Automatic human activity recognition is being studied widely by researchers for various applications. However, majority of the existing work are limited to recognition of isolated activities, though human activities are inherently continuous in nature with spatial and temporal transitions between various segments. Therefore, there are scopes to develop a robust and continuous Human Activity Recognition (HAR) system. In this paper, we present a novel Coarse-to-Fine framework for continuous HAR using Microsoft Kinect. The activity sequences are captured in the form of 3D skeleton trajectories consisting of 3D positions of 20 joints estimated from the depth data. The recorded sequences are first coarsely grouped into two activity sequences performed during sitting and standing. Next, the activities present in the segmented sequences are recognized into fine-level activities. Activity classification in both stages are performed using Bidirectional Long Short-Term Memory Neural Network (BLSTM-NN) classifier. A total of 1110 continuous activity sequences have been recorded using a combination of 24 isolated human activities. Recognition rates of 68.9% and 64.45% have been recorded using BLSTM-NN classifier when tested using length-modeling and without length-modeling, respectively. We have also computed results for isolated activity recognition performance. Finally, the performance has been compared with existing approaches.

Learning Datum-Wise Sampling Frequency for Energy-Efficient Human Activity Recognition

Continuous Human Activity Recognition (HAR) is an important application of smart mobile/wearable systems for providing dynamic assistance to users. However, HAR in real-time requires continuous sampling of data using built-in sensors (e.g., accelerometer), which significantly increases the energy cost and shortens the operating span. Reducing sampling rate can save energy but causes low recognition accuracy. Therefore, choosing adaptive sampling frequency that balances accuracy and energy efficiency becomes a critical problem in HAR. In this paper, we formalize the problem as minimizing both classification error and energy cost by choosing dynamically appropriate sampling rates. We propose Datum-Wise Frequency Selection (DWFS) to solve the problem via a continuous state Markov Decision Process (MDP). A policy function is learned from the MDP, which selects the best frequency for sampling an incoming data entity by exploiting a datum related state of the system. We propose a method for alternative learning the parameters of an activity classification model and the MDP that improves both the accuracy and the energy efficiency. We evaluate DWFS with three real-world HAR datasets, and the results show that DWFS statistically outperforms the state-of-the-arts regarding a combined measurement of accuracy and energy efficiency.