Activity Recognition System Research Articles

Efficient I3D-VGG19-based architecture for human activity recognition

Systems for autonomously identifying and analyzing human activities (HAR) make use of video data captured from various devices area requires ongoing updates due to the developing technology and multidisciplinary nature of HAR. Objective of this research is to classify different activity performed by Human using various pretrained models and latest transfer learning methods. Set the hyperparameters values to get accurate classification based on different performance evaluation matrices. In this study, the VGG19 based optimized I3D architecture is proposed. The experimental findings demonstrate that use of optimized VGG19 based I3D model on the UCF-50 dataset has led to an enhancement in the performance of the Human Activity recognition system with accuracy rate of training is 98.24% and testing is 98.36%, surpassing the performance of alternative I3D model using DenseNet121 in direct comparison. This will facilitate the development of applications like Smart Environments, Elderly Care and Assistive Technologies, Healthcare and Wellness and various other domains.

Self-attention deep ConvLSTM with sparse-learned channel dependencies for wearable sensor-based human activity recognition

In this study, we propose a novel deep-learning architecture with sparse learning for human activity recognition. The proposed model contains 1D CNNs and LSTM layers with a self-attention mechanism to enhance a substantial number of time points in time-series data for human activity recognition systems. Based on the recent success of squeeze-and-excite (SE) networks, the proposed deep learning model utilizes the SE module to enhance channel-wise interdependencies, which in turn leads to a boost in performance. In addition, we utilized sparse learning to retrain only weak nodes and freeze stronger nodes in a fully connected layer prior to classification layer. Furthermore, we utilized an entropy-inspired formula to find sparsely located weaker nodes and validated our model on various datasets, including Opportunity, UCI-HAR, and WISDM. Herein, we present an extensive analysis and survey of state-of-the-art studies, in addition to our proposed research. For a fair comparison, we evaluated our deep learning architecture using various performance metrics and achieved better results; the proposed model outperformed state-of-the-art algorithms for human activity recognition.

Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity Recognition

The widespread adoption of wearable devices has led to a surge in the development of multi-device wearable human activity recognition (WHAR) systems. Nevertheless, the performance of traditional supervised learning-based methods to WHAR is limited by the challenge of collecting ample annotated wearable data. To overcome this limitation, self-supervised learning (SSL) has emerged as a promising solution by first training a competent feature extractor on a substantial quantity of unlabeled data, followed by refining a minimal classifier with a small amount of labeled data. Despite the promise of SSL in WHAR, the majority of studies have not considered missing device scenarios in multi-device WHAR. To bridge this gap, we propose a multi-device SSL WHAR method termed Spatial-Temporal Masked Autoencoder (STMAE). STMAE captures discriminative activity representations by utilizing the asymmetrical encoder-decoder structure and two-stage spatial-temporal masking strategy, which can exploit the spatial-temporal correlations in multi-device data to improve the performance of SSL WHAR, especially on missing device scenarios. Experiments on four real-world datasets demonstrate the efficacy of STMAE in various practical scenarios.

Design and Development of an Imitation Detection System for Human Action Recognition Using Deep Learning.

Human action recognition (HAR) is a rapidly growing field with numerous applications in various domains. HAR involves the development of algorithms and techniques to automatically identify and classify human actions from video data. Accurate recognition of human actions has significant implications in fields such as surveillance and sports analysis and in the health care domain. This paper presents a study on the design and development of an imitation detection system using an HAR algorithm based on deep learning. This study explores the use of deep learning models, such as a single-frame convolutional neural network (CNN) and pretrained VGG-16, for the accurate classification of human actions. The proposed models were evaluated using a benchmark dataset, KTH. The performance of these models was compared with that of classical classifiers, including K-Nearest Neighbors, Support Vector Machine, and Random Forest. The results showed that the VGG-16 model achieved higher accuracy than the single-frame CNN, with a 98% accuracy rate.

More Reliable Neighborhood Contrastive Learning for Novel Class Discovery in Sensor-Based Human Activity Recognition

Human Activity Recognition (HAR) systems have made significant progress in recognizing and classifying human activities using sensor data from a variety of sensors. Nevertheless, they have struggled to automatically discover novel activity classes within massive amounts of unlabeled sensor data without external supervision. This restricts their ability to classify new activities of unlabeled sensor data in real-world deployments where fully supervised settings are not applicable. To address this limitation, this paper presents the Novel Class Discovery (NCD) problem, which aims to classify new class activities of unlabeled sensor data by fully utilizing existing activities of labeled data. To address this problem, we propose a new end-to-end framework called More Reliable Neighborhood Contrastive Learning (MRNCL), which is a variant of the Neighborhood Contrastive Learning (NCL) framework commonly used in visual domain. Compared to NCL, our proposed MRNCL framework is more lightweight and introduces an effective similarity measure that can find more reliable k-nearest neighbors of an unlabeled query sample in the embedding space. These neighbors contribute to contrastive learning to facilitate the model. Extensive experiments on three public sensor datasets demonstrate that the proposed model outperforms existing methods in the NCD task in sensor-based HAR, as indicated by the fact that our model performs better in clustering performance of new activity class instances.

An efficient and lightweight multiperson activity recognition framework for robot-assisted healthcare applications

Aging is inevitably associated with a decline in physical abilities and can pose challenges to the social lives of elderly individuals. In long-term care facilities, group exercise is instrumental for keeping elderly residents physically and socially healthy. Accommodating these needs in elderly care can be challenging due to staff shortages and other lacking resources. A robotic exercise coach could be helpful in such contexts. Intelligent human–robot interaction requires accurate and efficient human activity recognition. Several solutions focusing on human activity recognition in healthcare robotics have been proposed. However, multiperson activity recognition remains a challenging task in case of using vision-based or wearable sensors data, and past research has mainly focused on single-person rather than multiperson or group activity recognition. Moreover, the existing state-of-the-art methods for activity recognition mainly use heavyweight Convolutional Neural Network (CNN) models to achieve good accuracy. However, these models have certain drawbacks, such as requiring significant computational resources, higher memory and storage needs, and slower inference times. Another challenge is the limited number of publicly available datasets containing few activities for physical activity recognition. In this work, we propose a lightweight, deep learning-based, multiperson activity recognition system for group exercise training of elderly persons. Considering the limited publicly available datasets, we curated a new dataset named the Routine Exercise Dataset (RED), comprising 19 routine exercise activities recommended for elderly persons. The RED dataset has 14,440 samples collected from 19 participants and is one of the most extensive datasets of its kind. We evaluated our proposed activity recognition method based on proposed feature extraction modules and a one-dimensional multilayer long short-term memory network on 16 datasets, including 10 publicly available benchmark activity recognition datasets, an RED dataset, a publicly available dataset combined with RED dataset, and four noise-corrupted RED datasets. The results indicate the efficiency of the proposed method for real-time activity recognition compared to the state-of-the-art methods. The proposed method achieved F1-scores of 98.64%, 97.95%, and 99% on large-scale datasets named UESTC RGB-D, NTU RGB+D, and RED, respectively. We also developed a Robot Operating System (ROS)-based application to deploy our proposed system in a social robot and test it in real-life scenarios.

Analysis and comparison of various deep learning models to implement suspicious activity recognition in CCTV surveillance

The paper aims to analyze and compare various deep learning (DL) algorithms in order to develop a Suspicious Activity Recognition (SAR) system for closed-circuit television (CCTV) surveillance. Automated systems for detecting and classifying suspicious activities are crucial as technology’s role in safety and security expands. This paper addresses these challenges by creating a robust SAR system using machine learning techniques. It analyzes and compares evaluation metrics such as Precision, Recall, F1 Score, and Accuracy using various deep learning methods (convolutional neural network (CNN), Long short-term memory (LSTM) – Visual Geometry Group 16 (VGG16), LSTM – ResNet50, LSTM – EfficientNetB0, LSTM – InceptionNetV3, LSTM – DenseNet121, and Long-term Recurrent Convolutional Network (LRCN)). The proposed system improves threat identification, vandalism deterrence, fight prevention, and video surveillance. It aids emergency response by accurately classifying suspicious activities from CCTV footage, reducing reliance on human security personnel and addressing limitations in manual monitoring. The objectives of the paper include analyzing existing works, extracting features from CCTV videos, training robust deep learning models, evaluating algorithms, and improving accuracy. The conclusion highlights the superior performance of the LSTM-DenseNet121 algorithm, achieving an overall accuracy of 91.17% in detecting suspicious activities. This enhances security monitoring capabilities and reduces response time. Limitations of the system include subjectivity, contextual understanding, occlusion, false alarms, and privacy concerns. Future improvements involve real-time object tracking, collaboration with law enforcement agencies, and performance optimization. Ongoing research is necessary to overcome limitations and enhance the effectiveness of CCTV surveillance.

Wavelet-domain human activity recognition utilizing convolutional neural networks

Human activity recognition (HAR) is a crucial area of research in human-computer interaction. Despite previous efforts in this field, there is still a need for more accurate and robust methods that can handle time-series data from different sensors. In this study, we propose a novel method that generates an image using wavelet transform to extract time-frequency features of the recorded signal. Our method employs convolutional neural networks (CNNs) for feature extraction and activity recognition, and a new loss function that produces denser representations for samples, improving the model’s generalization on unseen samples. To evaluate the effectiveness of our proposed method, we conducted experiments on multiple publicly available data sets. Our results demonstrate that our method outperforms previous methods in terms of activity classification accuracy. Specifically, our method achieves higher accuracy rates and demonstrates improved robustness in real-world settings. Overall, our proposed method addresses the research gap of accurate and robust activity recognition from time-series data recorded from different sensors. Our findings have the potential to improve the accuracy and robustness of human activity recognition systems in real-world applications.

RFID-Based Human Action Recognition Through Spatiotemporal Graph Convolutional Neural Network

Traditional solutions for human action recognition usually rely on sensor or video methods. However, these methods have some limitations, such as inconvenient portability, light intensity influence, privacy protection, etc. In this paper, an RFID-based non-wearable human action recognition scheme is proposed. In order to reduce the occlusion effect of the human body on the signal and increase the diversity of the reflected signal, a tags array is constructed. The data of phase and RSSI are fused as feature data to enhance the diversity of data. Furthermore, a combined processing method is proposed to eliminate thermal noise generated by the equipment and reduce the interference caused by the environment. Then, an action segmentation algorithm is designed to align the RF signals of human action. Finally, an efficient human action signal classification model is constructed using the spatiotemporal graph convolutional neural network (STGCN). Extensive experiments demonstrate that the overall accuracy rate of the system for human action recognition is 92.8%. Compared with the comparative mainstream recognition algorithms, STGCN shows better classification performance in terms of identification precision. In addition, multimodal RFID data fusion also improves the accuracy of identification.

A computer vision-approach for activity recognition and residential monitoring of elderly people

In this study, we explore a human activity recognition (HAR) system using computer vision for assisted living systems (ALS). Most existing HAR systems are implemented using wired or wireless sensor networks. These systems have limitations such as cost, power issues, weight, and the inability of the elderly to wear and carry them comfortably. These issues could be overcome by a computer vision based HAR system. But such systems require a highly memory-consuming image dataset. Training such a dataset takes a long time. The proposed computer-vision-based system overcomes the shortcomings of existing systems. The authors have used key-joint angles, distances between the key joints, and slopes between the key joints to create a numerical dataset instead of an image dataset. All these parameters in the dataset are recorded via real-time event simulation. The data set has 780,000 calculated feature values from 20,000 images. This dataset is used to train and detect five different human postures. These are sitting, standing, walking, lying, and falling. The implementation encompasses four distinct algorithms: the decision tree (DT), random forest (RF), support vector machine (SVM), and an ensemble approach. Remarkably, the ensemble technique exhibited exceptional performance metrics with 99 ​% accuracy, 98 ​% precision, 97 ​% recall, and an F1 score of 99 ​%.

A Survey on Different Deep Learning Model for Human Activity Recognition based on Application

The field of human activity recognition (HAR) seeks to identify and classify an individual's unique movements or activities. However, recognizing human activity from video is a challenging task that requires careful attention to individuals, their behaviors, and relevant body parts. Multimodal activity recognition systems are necessary for many applications, including video surveillance systems, human-computer interfaces, and robots that analyze human behavior. This study provides a comprehensive analysis of recent breakthroughs in human activity classification, including different approaches, methodologies, applications, and limitations. Additionally, the study identifies several challenges that require further investigation and improvements. The specifications for an ideal human activity recognition dataset are also discussed, along with a thorough examination of the publicly available human activity classification datasets.

Performance evaluation of deep learning techniques for human activity recognition system

Human Activity Recognition (HAR) is crucial in various applications, such as sports and surveillance. This paper focuses on the performance evaluation of a HAR system using deep learning techniques. Features will be extracted using 3DCNN, and classification will be performed using LSTM. Meanwhile, 3DCNN and RNN are two additional, well-known classification techniques that will be applied in order to compare the effectiveness of the three classifiers. The 3DCNN-LSTM approach contributes the highest overall accuracy of 86.57%, followed by 3DCNN-3DCNN and 3DCNN-RNN with the overall accuracy of 86.07% and 79.60%, respectively. Overall, this paper contributes to the field of HAR and provides valuable insights for the development of activity recognition systems.

Wi-Fi Sensing for Indoor Localization via Channel State Information: A Survey

Wireless Fidelity (Wi-Fi) sensing utilization has been widespread, especially for human behavior/activity recognition. It provides high flexibility since it does not require the person/object to carry any device known as device-free. This "passive" concept is also helpful for another application of Wi-Fi sensing, i.e., indoor localization. The "sensing" is conducted using particular parameters extracted from communication links of Wi-Fi devices, i.e., channel state information (CSI). This paper explores the recent trends in CSI-based indoor localization with Wi-Fi technology as its core, including their advantages, challenges, and future directions. We found tremendous benefits can be gained by employing Wi-Fi sensing in localization supported by its performance and integrability for other intelligent systems for activity recognition.

A novel WGF-LN based edge driven intelligence for wearable devices in human activity recognition

Human activity recognition (HAR) is one of the key applications of health monitoring that requires continuous use of wearable devices to track daily activities. The most efficient supervised machine learning (ML)-based approaches for predicting human activity are based on a continuous stream of sensor data. Sensor data analysis for human activity recognition using conventional algorithms and deep learning (DL) models shows promising results, but evaluating their ambiguity in decision-making is still challenging. In order to solve these issues, the paper proposes a novel Wasserstein gradient flow legonet WGF-LN-based human activity recognition system. At first, the input data is pre-processed. From the pre-processed data, the features are extracted using Haar Wavelet mother- Symlet wavelet coefficient scattering feature extraction (HS-WSFE). After that, the interest features are selected from the extracted features using (Binomial Distribution integrated-Golden Eagle Optimization) BD-GEO. The important features are then post-processed using the scatter plot matrix method. Obtained post-processing features are finally given into the WGF-LN for classifying human activities. From these experiments, the results can be obtained and showed the efficacy of the proposed model.

Image expression of time series data of wearable IMU sensor and fusion classification of gymnastics action

Human activity recognition (HAR) based on multimodal wearable motion sensors is a valuable technology, and gymnastics action recognition is one of its important application scenarios. However, this field faces challenges such as excessive sensor node deployment, hindering effective utilization of time-series sensor data, and low accuracy in action recognition. To address these issues, this paper introduces a gymnastics action recognition system comprising a hardware setup and an action recognition framework that combines a time series data imaging method with a novel three-channel convolutional model ViTGS. Firstly, a cost-effective single-node sensor system equipped with a 6-axis IMU is devised to efficiently capture gymnastics action data. Secondly, three one-dimensional time series data image encoding methods, Gramian Angular Field (GAF), Recurrence Plot (RP), and Markov Transition Field (MTF), are employed to capture the temporal features, phase space features, and state transition features of the data respectively. Finally, the ViTGS model is proposed, which uses the ViT_B/16 network to extract the features of each channel image, and uses Gated Fusion Network (GFN) to realize the effective fusion of different channel features and uses SVM as the recognition layer to further improve the classification accuracy. The experimental results show that using single-node (left wrist) sensor data, an accuracy of 98.56% is obtained on the acceleration dataset, and an accuracy of 99.04% is obtained on the angular velocity dataset. The method of this paper is not only applicable to gymnastics action recognition but also can be generalized to other human action recognition fields.

Scorpion-inspired bionic gait activity location and recognition smart home system

Vibration-based contactless technologies are arising for indoor person location and gait activity recognition applications, which can overcome issues such as privacy breaches and wearing discomfort with traditional methods to satisfy the critical need for health and safety monitoring in smart home. However, simultaneous location and recognition of occupant gait activity using information-starved vibration signals remains challenging due to its less characterization of event details. In nature, scorpions use a 3/1 neuron coding pattern to enrich vibration information by enhancing sensory differences from eight vibration sensilla embedded in their metatarsal ends of each walking leg, thus exhibiting high accuracy in locating moving prey. Inspired by this, we exploit a coding pattern to make vibration signal features richer for discrimination, while designing a bionic gait activity location and recognition system in which the data acquisition device is built by imitating scorpion vibration sensilla array distribution. To verify the feasibility of the system, we collect vibration signals produced by different people with varying gait activities (walking, jogging, running, and stepping) in an actual scene, and a trained hidden Markov model is employed to identify the activity of the current location target. The average estimated error and the relative estimated error of position estimation are calculated to be 0.3517 m and 8.79 %, respectively. The average recall and average precision of gait activity recognition are 95.83 % and 95.84 %, respectively. Our system is proven to simultaneously locate and recognize occupant gait activities, which can meet the needs including smart home, and health care monitoring.

In-depth analysis of design & development for sensor-based human activity recognition system

In-depth analysis of design & development for sensor-based human activity recognition system

Enabling Remote Elderly Care: Design and Implementation of a Smart Energy Data System with Activity Recognition.

Seniors face many challenges as they age, such as dementia, cognitive and memory disorders, vision and hearing impairment, among others. Although most of them would like to stay in their own homes, as they feel comfortable and safe, in some cases, older people are taken to special institutions, such as nursing homes. In order to provide serious and quality care to elderly people at home, continuous remote monitoring is perceived as a solution to keep them connected to healthcare service providers. The new trend in medical health services, in general, is to move from 'hospital-centric' services to 'home-centric' services with the aim of reducing the costs of medical treatments and improving the recovery experience of patients, among other benefits for both patients and medical centers. Smart energy data captured from electrical home appliance sensors open a new opportunity for remote healthcare monitoring, linking the patient's health-state/health-condition with routine behaviors and activities over time. It is known that deviation from the normal routine can indicate abnormal conditions such as sleep disturbance, confusion, or memory problems. This work proposes the development and deployment of a smart energy data with activity recognition (SEDAR) system that uses machine learning (ML) techniques to identify appliance usage and behavior patterns oriented to older people living alone. The proposed system opens the door to a range of applications that go beyond healthcare, such as energy management strategies, load balancing techniques, and appliance-specific optimizations. This solution impacts on the massive adoption of telehealth in third-world economies where access to smart meters is still limited.

Development and Testing of a Daily Activity Recognition System for Post-Stroke Rehabilitation.

Those who survive the initial incidence of a stroke experience impacts on daily function. As a part of the rehabilitation process, it is essential for clinicians to monitor patients' health status and recovery progress accurately and consistently; however, little is known about how patients function in their own homes. Therefore, the goal of this study was to develop, train, and test an algorithm within an ambient, in-home depth sensor system that can classify and quantify home activities of individuals post-stroke. We developed the Daily Activity Recognition and Assessment System (DARAS). A daily action logger was implemented with a Foresite Healthcare depth sensor. Daily activity data were collected from seventeen post-stroke participants' homes over three months. Given the extensive amount of data, only a portion of the participants' data was used for this specific analysis. An ensemble network for activity recognition and temporal localization was developed to detect and segment the clinically relevant actions from the recorded data. The ensemble network, which learns rich spatial-temporal features from both depth and skeletal joint data, fuses the prediction outputs from a customized 3D convolutional-de-convolutional network, customized region convolutional 3D network, and a proposed region hierarchical co-occurrence network. The per-frame precision and per-action precision were 0.819 and 0.838, respectively, on the test set. The outcomes from the DARAS can help clinicians to provide more personalized rehabilitation plans that benefit patients.

Generalisable Dialogue-based Approach for Active Learning of Activities of Daily Living

While Human Activity Recognition systems may benefit from Active Learning by allowing users to self-annotate their Activities of Daily Living (ADLs), many proposed methods for collecting such annotations are for short-term data collection campaigns for specific datasets. We present a reusable dialogue-based approach to user interaction for active learning in activity recognition systems, which utilises semantic similarity measures and a dataset of natural language descriptions of common activities (which we make publicly available). Our approach involves system-initiated dialogue, including follow-up questions to reduce ambiguity in user responses where appropriate. We apply this approach to two active learning scenarios: (i) using an existing CASAS dataset, demonstrating long-term usage; and (ii) using an online activity recognition system, which tackles the issue of online segmentation and labelling. We demonstrate our work in context, in which a natural language interface provides knowledge that can help interpret other multi-modal sensor data. We provide results highlighting the potential of our dialogue- and semantic similarity-based approach. We evaluate our work: (i) quantitatively, as an efficient way to seek users’ input for active learning of ADLs; and (ii) qualitatively, through a user study in which users were asked to compare our approach and an established method. Results show the potential of our approach as a hands-free interface for annotation of sensor data as part of an active learning system. We provide insights into the challenges of active learning for activity recognition under real-world conditions and identify potential ways to address them.