From action to activity: Sensor-based activity recognition

Research on sensor-based activity recognition has, recently, made significant progress and is attracting growing attention in a number of disciplines and application domains. However, there is a lack of high-level overview on this topic that can inform related communities of the research state of the art. In this paper, we present a comprehensive survey to examine the development and current status of various aspects of sensor-based activity recognition. We first discuss the general rationale and distinctions of vision-based and sensor-based activity recognition. Then, we review the major approaches and methods associated with sensor-based activity monitoring, modeling, and recognition from which strengths and weaknesses of those approaches are highlighted. We make a primary distinction in this paper between data-driven and knowledge-driven approaches, and use this distinction to structure our survey. We also discuss some promising directions for future research.

Sensor-based activity recognition aims to predict users' activities from multi-dimensional streams of various sensor readings received from ubiquitous sensors. To use machine learning techniques for sensor-based activity recognition, previous approaches focused on composing a feature vector to represent sensor-reading streams received within a period of various lengths. With the constructed feature vectors, e.g., using predefined orders of moments in statistics, and their corresponding labels of activities, standard classification algorithms can be applied to train a predictive model, which will be used to make predictions online. However, we argue that in this way some important information, e.g., statistical information captured by higher-order moments, may be discarded when constructing features. Therefore, in this paper, we propose a new method, denoted by SMMAR, based on learning from distributions for sensor-based activity recognition. Specifically, we consider sensor readings received within a period as a sample, which can be represented by a feature vector of infinite dimensions in a Reproducing Kernel Hilbert Space (RKHS) using kernel embedding techniques. We then train a classifier in the RKHS. To scale-up the proposed method, we further offer an accelerated version by utilizing an explicit feature map instead of using a kernel function. We conduct experiments on four benchmark datasets to verify the effectiveness and scalability of our proposed method.

Existing techniques for 3D action recognition are sensitive to viewpoint variations because they extract features from depth images which change significantly with viewpoint. In contrast, we directly process the pointclouds and propose a new technique for action recognition which is more robust to noise, action speed and viewpoint variations. Our technique consists of a novel descriptor and keypoint detection algorithm. The proposed descriptor is extracted at a point by encoding the Histogram of Oriented Principal Components (HOPC) within an adaptive spatio-temporal support volume around that point. Based on this descriptor, we present a novel method to detect Spatio-Temporal Key-Points (STKPs) in 3D pointcloud sequences. Experimental results show that the proposed descriptor and STKP detector outperform state-of-the-art algorithms on three benchmark human activity datasets. We also introduce a new multiview public dataset and show the robustness of our proposed method to viewpoint variations.

Several techniques for human activity recognition (HAR) in a smart indoor environment have been developed and improved along with the rapid advancement of sensor technologies. However, recognizing multiple people’s activities is still challenging due to the complexity of their activities, such as parallel and collaborative activities. To address these challenges, we propose a transformer with a bidirectional gated recurrent unit (GRU) deep learning (DL) method, called TRANS-BiGRU, to efficiently learn and recognize different types of activities performed by multiple residents. We compare the proposed model with the state-of-the-art models and various DL models, such as Ensemble2LSTM (Ens2-LSTM), bidirectional GRUs (Bi-GRU), and traditional machine learning (ML) models, such as support vector machine (SVM). Our experimental results based on the center for advanced studies in adaptive system and ARAS public data sets show that our model significantly outperforms the existing models for complex activity recognition of multiple residents.

Static postural transition-based technique and efficient feature extraction for sensor-based activity recognition

Weakly-supervised sensor-based activity segmentation and recognition via learning from distributions

This chapter presents a comprehensive survey on the state of the art of various aspects of sensor-based activity recognition. It first examines the general rationale and distinctions of different sensor technologies for activity monitoring. Then we review the major approaches and methods associated with sensor-based activity modeling and recognition from which strengths and weaknesses of those approaches are analysed and highlighted. The survey makes a primary distinction between data-driven and knowledge-driven approaches, and uses this distinction to structure our survey.

Real-time pattern and activity recognition techniques have experienced a recent surge in the use of advanced statistical techniques for analyzing data. More explicitly, Machine Learning classification methods have been used in a variety of applications and for different purposes, such as recognizing different types of events from accelerometer data (e.g. in smart watches, wearables, and vibration analysis). The objective of this work is to present a novel use of accelerometer-based pattern recognition techniques to detect the urban space infrastructure characteristics based on their effect on a vehicle’s body frame dynamics. For this study, we focus on detecting speed bumps, potholes and curves based on a real-time data streaming with the objective of rendering a street digitalization to automatically update an urban space Digital Twin. The present study has achieved real-time event detection for an urban trajectory with high accuracy which yields data that can be later fused with other sensors data. This approach is effectively contributing to the infrastructure layer of a multi-layered approach of digital modelling. Furthermore, the results contribute to expand the body of knowledge of real-time accelerometer event recognition techniques.

Transformer for Skeleton-based action recognition: A review of recent advances

We have compared the performance of different machine learning techniques for human activity recognition. Experiments were made using a benchmark dataset where each subject wore a device in the pocket and another on the wrist. The dataset comprises thirteen activities, including physical activities, common postures, working activities and leisure activities. We apply a methodology known as the activity recognition chain, a sequence of steps involving preprocessing, segmentation, feature extraction and classification for traditional machine learning methods; we also tested convolutional deep learning networks that operate on raw data instead of using computed features. Results show that combination of two sensors does not necessarily result in an improved accuracy. We have determined that best results are obtained by the extremely randomized trees approach, operating on precomputed features and on data obtained from the wrist sensor. Deep learning architectures did not produce competitive results with the tested architecture.

Ambient sensor-based in-home activity recognition plays a crucial role in the design and development of a smart home to better and actively respond to population aging. From the perspective of machine learning, how to extract features from sensor data largely determines the power of a data-driven human activity recognizer. However, few studies systematically investigate how to encode streaming sensor events. To this end, we herein conduct a comparison of different feature extraction techniques for activity recognition. Specifically, we explore two types of feature representations (i.e., statistical features and structural features) and evaluate their single use and joint use. Besides, we experimentally analyze the impact of window size on prediction accuracy. Finally, we perform experiments on three public datasets with 15 different feature encodings and 6 classifiers. Results show that the joint use of different features generally obtains enhanced accuracy and that the interval 60s of window size achieves a better accuracy-speed tradeoff.

Human activity recognition (HAR) is challenging, particularly in natural settings, due to issues like confounding gestures present in different activities, diversity in performing the same activity, and the wide range of possible human activities. Acceleration and rotation rate, two of the most widely used sensing modalities for HAR, are limited in addressing these issues. Also, many solutions for wearables are focused on some particular activities, and they do not generalize to others. One challenge is to develop underlying generic techniques for activity recognition that can be used in many different wearable based applications. We present a set of general purpose techniques for activity recognition using wearables. The techniques are based on quaternions that represent the orientation of a device in three-dimensional space. The techniques can be used for different purposes like reducing computation, increasing robustness and accuracy, and better understanding movements for HAR.

Acoustic activity recognition has emerged as a foundational element for imbuing devices with context-driven capabilities, enabling richer, more assistive, and more accommodating computational experiences. Traditional approaches rely either on custom models trained in situ, or general models pre-trained on preexisting data, with each approach having accuracy and user burden implications. We present Listen Learner, a technique for activity recognition that gradually learns events specific to a deployed environment while minimizing user burden. Specifically, we built an end-to-end system for self-supervised learning of events labelled through one-shot interaction. We describe and quantify system performance 1) on preexisting audio datasets, 2) on real-world datasets we collected, and 3) through user studies which uncovered system behaviors suitable for this new type of interaction. Our results show that our system can accurately and automatically learn acoustic events across environments (e.g., 97% precision, 87% recall), while adhering to users' preferences for non-intrusive interactive behavior.

A novel hybrid deep learning approach with GWO–WOA optimization technique for human activity recognition

In the last years, techniques for activity recognition have attracted increasing attention. Among many applications, a special interest is in the pervasive e-Health domain where automatic activity recognition is used in rehabilitation systems, chronic disease management, monitoring of the elderly, as well as in personal well being applications. Research in this field has mainly adopted techniques based on supervised learning algorithms to recognize activities based on contextual conditions (e.g., location, surrounding environment, used objects) and data retrieved from body-worn sensors. Since these systems rely on a sufficiently large amount of training data which is hard to collect, scalability with respect to the number of considered activities and contextual data is a major issue. In this paper, we propose the use of ontologies and ontological reasoning combined with statistical inferencing to address this problem. Our technique relies on the use of semantic relationships that express the feasibility of performing a given activity in a given context. The proposed technique neither increases the obtrusiveness of the statistical activity recognition system, nor introduces significant computational overhead to real-time activity recognition. The results of extensive experiments with data collected from sensors worn by a group of volunteers performing activities both indoor and outdoor show the superiority of the combined technique with respect to a solely statistical approach. To the best of our knowledge, this is the first work that systematically investigates the integration of statistical and ontological reasoning for activity recognition.