Activity Recognition Research Research Articles

Multi-head CNN-based activity recognition and its application on chest-mounted sensor-belt

In recent years, a great deal of research has been done on the identification, monitoring, and classification of human activities. Human activity recognition (HAR) is a term commonly used to describe the automatic identification of physical activities. For activity recognition, there are primarily vision-based and sensor-based methods available. The computer vision-based method is generally effective in lab settings, but because of clutter, fluctuating light levels, and contrast, it may not perform well in real-world scenarios. Continuous monitoring and analysis of physiological signals obtained from heterogeneous sensors attached to an individual’s body is required to realise sensor-based HAR systems. Most of the previous research in human activity recognition (HAR) is biased along with feature engineering and pre-processing which requires a good amount of domain knowledge. Application-specific modelling and time-taking methods are involved in these approaches. In this work, the multi-head convolutional neural network-based human activity recognition framework is proposed where automatic feature extraction and classification are involved in the form of an end-to-end classification approach. Experiments of this approach are performed by taking raw wearable sensor data with few pre-processing steps and without the involvement of a handcrafted feature extraction technique. 99.23% and 93.55% accuracy are obtained on the WISDM and UCI-HAR datasets which denoted the much improvement in the assessment of HAR over other similar approaches. The model is also tested on locally collected data from a chest mounted belt with fabric sensors and an accuracy of 87.14% has been achieved on that data.

Empowering Participatory Research in Urban Health: Wearable Biometric and Environmental Sensors for Activity Recognition.

Participatory exposure research, which tracks behaviour and assesses exposure to stressors like air pollution, traditionally relies on time-activity diaries. This study introduces a novel approach, employing machine learning (ML) to empower laypersons in human activity recognition (HAR), aiming to reduce dependence on manual recording by leveraging data from wearable sensors. Recognising complex activities such as smoking and cooking presents unique challenges due to specific environmental conditions. In this research, we combined wearable environment/ambient and wrist-worn activity/biometric sensors for complex activity recognition in an urban stressor exposure study, measuring parameters like particulate matter concentrations, temperature, and humidity. Two groups, Group H (88 individuals) and Group M (18 individuals), wore the devices and manually logged their activities hourly and minutely, respectively. Prioritising accessibility and inclusivity, we selected three classification algorithms: k-nearest neighbours (IBk), decision trees (J48), and random forests (RF), based on: (1) proven efficacy in existing literature, (2) understandability and transparency for laypersons, (3) availability on user-friendly platforms like WEKA, and (4) efficiency on basic devices such as office laptops or smartphones. Accuracy improved with finer temporal resolution and detailed activity categories. However, when compared to other published human activity recognition research, our accuracy rates, particularly for less complex activities, were not as competitive. Misclassifications were higher for vague activities (resting, playing), while well-defined activities (smoking, cooking, running) had few errors. Including environmental sensor data increased accuracy for all activities, especially playing, smoking, and running. Future work should consider exploring other explainable algorithms available on diverse tools and platforms. Our findings underscore ML's potential in exposure studies, emphasising its adaptability and significance for laypersons while also highlighting areas for improvement.

Open Datasets in Human Activity Recognition Research—Issues and Challenges: A Review

Open Datasets in Human Activity Recognition Research—Issues and Challenges: A Review

Occlusion-Aware Graph Neural Networks for Skeleton Action Recognition

Due to its important applications in industry, the research of action recognition has attracted many attention. In particular, skeleton-based human action recognition methods have evolved to become more and more competitive. Nevertheless, occlusion is still a very challenging task in human action recognition as yet. Off-the-shelf works are usually based on complete skeleton data, few people consider action recognition in occlusion. For the sake of improving the recognition accuracy on occluded skeleton data, we put forward an occlusion-aware multi-stream fusion graph convolutional network (dubbed MSFGCN). Multiple streams are comprised in MSFGCN, and different occlusion cases can be disposed by different streams. Besides, in order to construct more discriminative features, multi-modal features were extracted simultaneously, such as joint coordinates, relative coordinates, small-scale temporal differences and large-scale temporal differences. In particular, it is the first time to take advantage of motion features at large and small scales at the same time, which helps to distinguish actions with different motion amplitudes. What's more, considering the different importance of different parts of the human body for different action recognition, the content adaptation operation is used to further optimize the recognition performance. Effective training strategies are also presented to further improve the performance of the model. Experimental results show that the proposed MSFGCN has a great advantage over other methods on occluded skeleton datasets. To showcase the effectiveness of different modules, extensive ablation experiments are performed on various skeleton action recognition datasets.

An activity recognition system at home based on illuminance sensors

AbstractA social issue that leads to the safety and security of people is the network that monitors the safety of people living alone in remote areas. Various research and developments of indoor human activity recognition using IoT sensor nodes have been conducted in recent years. In this research, we focus on turning on and off indoor lighting and propose an activity recognition system at home using an illuminance sensor. An IoT sensor node with a built‐in illuminance sensor is attached to the wall near the indoor lighting equipment. The living activities of people in the room are estimated from changes in the illuminance. We assume a significant change in the illuminance of the indoor lighting is observed at least once a day at the sampling interval of the IoT sensor node. In that case, it is possible to estimate the operation of the indoor lighting due to daily activities and to confirm the safety of people living alone in remote areas. The activity recognition system was evaluated for a total of 380 days, excluding the missing period. The estimation of indoor lighting manipulation by daily activities was accurate for 376 days. Precision, Recall, and F‐measure score values, which are evaluation indices for activity estimation, were 94.9%, 98.2%, and 96.5%, respectively.

Enhanced Adjacency Matrix-Based Lightweight Graph Convolution Network for Action Recognition.

Graph convolutional networks (GCNs), which extend convolutional neural networks (CNNs) to non-Euclidean structures, have been utilized to promote skeleton-based human action recognition research and have made substantial progress in doing so. However, there are still some challenges in the construction of recognition models based on GCNs. In this paper, we propose an enhanced adjacency matrix-based graph convolutional network with a combinatorial attention mechanism (CA-EAMGCN) for skeleton-based action recognition. Firstly, an enhanced adjacency matrix is constructed to expand the model's perceptive field of global node features. Secondly, a feature selection fusion module (FSFM) is designed to provide an optimal fusion ratio for multiple input features of the model. Finally, a combinatorial attention mechanism is devised. Specifically, our spatial-temporal (ST) attention module and limb attention module (LAM) are integrated into a multi-input branch and a mainstream network of the proposed model, respectively. Extensive experiments on three large-scale datasets, namely the NTU RGB+D 60, NTU RGB+D 120 and UAV-Human datasets, show that the proposed model takes into account both requirements of light weight and recognition accuracy. This demonstrates the effectiveness of our method.

Advancing human action recognition: A hybrid approach using attention-based LSTM and 3D CNN

In this paper, we propose a novel approach to video action recognition that integrates a modified and optimized 3D Convolutional Neural Network, a Long Short-Term Memory network, and attention mechanisms. This synergy enhances the overall performance, offering an advantage over existing methods in managing the intricacies of real-world scenarios. The uniqueness of our approach lies in its capacity to capture both spatial and temporal information from video sequences and the incorporation of an attention mechanism that selectively emphasizes key areas within the sequences, thereby enhancing recognition accuracy. The model is particularly tailored to handle complex scenarios, such as those with multiple actors or objects, or instances of occlusion. It effectively addresses the subjectivity and variability inherent in action annotations within datasets. We also apply an array of preprocessing techniques to further optimize model performance. Through rigorous experimental evaluations on benchmark datasets, namely UCF101 and HMDB51, we demonstrate that our proposed approach significantly outperforms existing state-of-the-art methods in action recognition. These results underscore the potential of our approach for further advancements in video action recognition research.

BK-POSE: A LIGHTWEIGHT MODEL FOR MULTI-PERSON POSE ESTIMATION IN THE WILD

Human pose estimation is an essential topic in computer vision research, which has numerous applications in various fields. In this article, we propose a lightweight deep learning architecture called BK-Pose for recognizing human keypoints from images and videos that can be executed on edge computing devices. The main contributions of our work are two folds: (1) A lightweight bottom-up deep learning model is introduced that can be deployed on edge devices and can achieve high frames-per-second (FPS) rates while maintaining acceptable accuracy in suitable environments. (2) The incorporation of a novel focal L2 loss (FL) technique allows for the effective balancing between “hard” and “easy” keypoints samples during training. The performance of the BK-Pose model is evaluated within a classroom environment for capturing students’ keypoints and demonstrate its efficacy. Our results show promise for further research in activity recognition using the proposed architecture.

Smart Suspenders With Sensors and Machine Learning for Human Activity Monitoring

Most of the Human Activity Recognition (HAR) research focuses on the use of smartphone in-built accelerometer sensors. As accelerometers are location-centric, they measure acceleration signals only at the installation points, increasing the number of sensors required to identify the whole human body activity. They have an inherent property of being noisy, thereby increasing the processing complexity and duration. This paper, for the first time, proposes a body-worn suspender integrated with a strain sensor system that captures the body movement’s periodicity, resulting in less noisy readings with non-localized measurements. The proposed smart suspender system reduces the need for localized sensors to measure complex activities at various points and lessen the pre-processing time for smoothening the noise signals. The system recognizes three simple and eleven complex human activities using machine and deep learning algorithms with best accuracy value of 97.85%. A comparison of the performance between kernel and linear discriminant analysis (KDA and LDA) for this system is made and KDA outperformed LDA across most classifiers.

OFPI: Optical Flow Pose Image for Action Recognition

Most approaches to action recognition based on pseudo-images involve encoding skeletal data into RGB-like image representations. This approach cannot fully exploit the kinematic features and structural information of human poses, and convolutional neural network (CNN) models that process pseudo-images lack a global field of view and cannot completely extract action features from pseudo-images. In this paper, we propose a novel pose-based action representation method called Optical Flow Pose Image (OFPI) in order to fully capitalize on the spatial and temporal information of skeletal data. Specifically, in the proposed method, an advanced pose estimator collects skeletal data before locating the target person and then extracts skeletal data utilizing a human tracking algorithm. The OFPI representation is obtained by aggregating these skeletal data over time. To test the superiority of OFPI and investigate the significance of the model having a global field of view, we trained a simple CNN model and a transformer-based model, respectively. Both models achieved superior outcomes. Because of the global field of view, especially in the transformer-based model, the OFPI-based representation achieved 98.3% and 94.2% accuracy on the KTH and JHMDB datasets, respectively. Compared with other advanced pose representation methods and multi-stream methods, OFPI achieved state-of-the-art performance on the JHMDB dataset, indicating the utility and potential of this algorithm for skeleton-based action recognition research.

GAIT RECOGNITION OF ME-SVM BASED ON SINGLE ACCELEROMETER

Gait recognition based on acceleration sensor signals is more suitable for complex environments than other signal-based methods, which can be used for data acquisition for gait recognition. However, most traditional methods only consider the position matching of multiple accelerometers, ignoring the comfort of the collector. In recent years, support vector machine (SVM) has been widely used in portrait recognition, language, and video processing. It has a flexible and powerful processing ability for nonlinear sequence input, and is a sparse and robust classifier. In this paper, we propose a multi-feature SVM algorithm ME-SVM, which is classified after multi-feature fusion. Usually, in the research of action recognition based on acceleration sensors, researchers classify from a single eigenvalue, but this paper optimizes this and further improves the recognition performance. Simulation and experimental results show that the algorithm has high accuracy and robustness.

ONTOLOGICAL, FULLY PROBABILISTIC KNOWLEDGE MODEL FOR HUMAN ACTIVITY RECOGNITION

Efficiency and scalability are obstacles that have not yet received a viable response from the human activity recognition research community. This paper proposes an activity recognition method. The knowledge model is in the form of ontology, the state-of-the-art in knowledge representation and reasoning. The ontology starts with probabilistic information about subjects’ low-level activities and location and then is populated with the assertion axioms learned from data or defined by the user. Unlike methods that choose only the most probable candidate from sensor readings, the proposed method keeps multiple candidates with the known degree of confidence for each one and involves them in decision making. Using this method, the system is more flexible to deal with unreliable data, readings from sensors, and the final recognition rate is improved. Besides, to resolve the scalability problem, a system is designed and implemented to do reasoning and storing in a relational database management system. Numerical evaluations and conceptual benchmarking prove the proposed system feasibility.

Exploring LoRa and Deep Learning-Based Wireless Activity Recognition

Today’s wireless activity recognition research still needs to be practical, mainly due to the limited sensing range and weak through-wall effect of the current wireless activity recognition based on Wi-Fi, RFID (Radio Frequency Identification, RFID), etc. Although some recent research has demonstrated that LoRa can be used for long-range and wide-range wireless sensing, no pertinent studies have been conducted on LoRa-based wireless activity recognition. This paper proposes applying long-range LoRa wireless communication technology to contactless wide-range wireless activity recognition. We propose LoRa and deep learning for contactless indoor activity recognition for the first time and propose a more lightweight improved TPN (Transformation Prediction Network, TPN) backbone network. At the same time, using only two features of the LoRa signal amplitude and phase as the input of the model, the experimental results demonstrate that the effect is better than using the original signal directly. The recognition accuracy reaches 97%, which also demonstrate that the LoRa wireless communication technology can be used for wide-range activity recognition, and the recognition accuracy can meet the needs of engineering applications.

New machine learning approaches for real-life human activity recognition using smartphone sensor-based data

In recent years, mainly due to the application of smartphones in this area, research in human activity recognition (HAR) has shown a continuous and steady growth. Thanks to its wide range of sensors, its size, its ease of use, its low price and its applicability in many other fields, it is a highly attractive option for researchers. However, the vast majority of studies carried out so far focus on laboratory settings, outside of a real-life environment. In this work, unlike in other papers, progress was sought on the latter point. To do so, a dataset already published for this purpose was used. This dataset was collected using the sensors of the smartphones of different individuals in their daily life, with almost total freedom. To exploit these data, numerous experiments were carried out with various machine learning techniques and each of them with different hyperparameters. These experiments proved that, in this case, tree-based models, such as Random Forest, outperform the rest. The final result shows an enormous improvement in the accuracy of the best model found to date for this purpose, from 74.39% to 92.97%.

A Petri Net Architecture for Real-Time Human Activity Recognition in Work Systems

Real-time human-centered assistance in industrial processes depends on the individual history of the work person's activities in the work system and requires adequate methods for tracking the person's actions. Most research in human activity recognition is based on recognizing actions from video data using computer vision methods. Digital equipment, standardized machine data interfaces, and smart wearable devices extend the possibilities to describe the current state of the work system. Petri nets have already been applied to human activity recognition, however, without the requirement of detecting actions in real-time. This paper proposes a Petri net architecture that enables hierarchical description-based human activity recognition in industrial work processes. We present an extension, a Partitioned Colored Petri Net, based on the colored Petri net formalism that infers activities from state transitions of the work system in real-time. In a case study, we demonstrate the Petri net's application for an error-based learning system that visualizes error consequences in augmented reality using experimentable digital twins.

Deep learning for Flight Maneuver Recognition: A survey

<abstract> <p>Deep learning for Flight Maneuver Recognition involves flight maneuver detection and recognition tasks in different areas, including pilot training, aviation safety, and autonomous air combat. As a key technology for these applications, deep learning for Flight Maneuver Recognition research is underdeveloped and limited by domain knowledge and data sources. This paper presents a comprehensive survey of all Flight Maneuver Recognition studies since the 1980s to accurately define the research and describe its significance for the first time. In an analogy to the flourishing Human Action Recognition research, we divided deep learning for Flight Maneuver Recognition into vision-based and sensor-based studies, combed through all the literature, and referred to existing reviews of Human Action Recognition to demonstrate the similarities and differences between Flight Maneuver Recognition and Human Action Recognition in terms of problem essentials, research methods, and publicly available datasets. This paper presents the dataset-The Civil Aviation Flight University of China, which was generated from real training of a fixed-wing flight at Civil Aviation Flight University of China. We used this dataset to reproduce and evaluate several important methods of Flight Maneuver Recognition and visualize the results. Based on the evaluation results, the paper discusses the advantages, disadvantages, and overall shortcomings of these methods, as well as the challenges and future directions for deep learning for Flight Maneuver Recognition.</p> </abstract>

HIT HAR: Human Image Threshing Machine for Human Activity Recognition Using Deep Learning Models.

In recent days, research in human activity recognition (HAR) has played a significant role in healthcare systems. The accurate activity classification results from the HAR enhance the performance of the healthcare system with broad applications. HAR results are useful in monitoring a person's health, and the system predicts abnormal activities based on user movements. The HAR system's abnormal activity predictions provide better healthcare monitoring and reduce users' health issues. The conventional HAR systems use wearable sensors, such as inertial measurement unit (IMU) and stretch sensors for activity recognition. These approaches show remarkable performances to the user's basic activities such as sitting, standing, and walking. However, when the user performs complex activities, such as running, jumping, and lying, the sensor-based HAR systems have a higher degree of misclassification results due to the reading errors from sensors. These sensor errors reduce the overall performance of the HAR system with the worst classification results. Similarly, radiofrequency or vision-based HAR systems are not free from classification errors when used in real time. In this paper, we address some of the existing challenges of HAR systems by proposing a human image threshing (HIT) machine-based HAR system that uses an image dataset from a smartphone camera for activity recognition. The HIT machine effectively uses a mask region-based convolutional neural network (R-CNN) for human body detection, a facial image threshing machine (FIT) for image cropping and resizing, and a deep learning model for activity classification. We demonstrated the effectiveness of our proposed HIT machine-based HAR system through extensive experiments and results. The proposed HIT machine achieved 98.53% accuracy when the ResNet architecture was used as its deep learning model.

HMM-Based Action Recognition System for Elderly Healthcare by Colorizing Depth Map.

Addressing the problems facing the elderly, whether living independently or in managed care facilities, is considered one of the most important applications for action recognition research. However, existing systems are not ready for automation, or for effective use in continuous operation. Therefore, we have developed theoretical and practical foundations for a new real-time action recognition system. This system is based on Hidden Markov Model (HMM) along with colorizing depth maps. The use of depth cameras provides privacy protection. Colorizing depth images in the hue color space enables compressing and visualizing depth data, and detecting persons. The specific detector used for person detection is You Look Only Once (YOLOv5). Appearance and motion features are extracted from depth map sequences and are represented with a Histogram of Oriented Gradients (HOG). These HOG feature vectors are transformed as the observation sequences and then fed into the HMM. Finally, the Viterbi Algorithm is applied to recognize the sequential actions. This system has been tested on real-world data featuring three participants in a care center. We tried out three combinations of HMM with classification algorithms and found that a fusion with Support Vector Machine (SVM) had the best average results, achieving an accuracy rate (84.04%).

Research on the application of body posture action feature extraction and recognition comparison

This paper mainly conducts a comparative study of body posture action feature extraction and recognition in three-dimensional (3D) space through motion capture technology and virtual reality technology. Firstly, this paper proposes a body posture feature extraction method in 3D space to obtain joint point data and conduct body posture feature extraction research. At the same time, this method is used for action recognition research, which solves the difficulty of traditional methods in 3D scene feature extraction and motion recognition. In addition, this paper proposes two action comparison methods in 3D space. One is a ‘distance threshold method’ that calculates the distance between bones in the form of joint points and bones, which can finally provide the effect of dynamic display of sliders; the other is the ‘feature plane method’ that calculates the detailed information based on different skeletal joint point planes, and finally provides a text display effect with detailed body posture difference values. Finally, combined with virtual reality technology, an application platform for body posture feature recognition and comparison is designed and implemented, which solves the problems of poor visualization effect and weak interaction of traditional methods.

Human Action Recognition Research Based on Fusion TS-CNN and LSTM Networks

Human Action Recognition Research Based on Fusion TS-CNN and LSTM Networks