Human Activity Recognition Model Research Articles

WiLDAR: WiFi Signal-Based Lightweight Deep Learning Model for Human Activity Recognition

WiLDAR: WiFi Signal-Based Lightweight Deep Learning Model for Human Activity Recognition

I-Sample: Augment Domain Adversarial Adaptation Models for WiFi-based HAR

Recently, using deep learning to achieve WiFi-based human activity recognition (HAR) has drawn significant attention. While capable of achieving accurate identification in a single domain (i.e., training and testing in the same consistent WiFi environment), it would become extremely tough when WiFi environments change significantly. As such, domain adversarial neural networks-based approaches have been proposed to handle such diversities across domains, yet often found to share the same limitation in practice: the imbalance between high-capacity of feature extractors and data insufficiency of source domains. This article proposes i-Sample, an intermediate sample generation-based framework, striving to tackle this issue for WiFi-based HAR. i-Sample is mainly designed as two-stage training, where four data augmentation operations are proposed to train a coarse domain-invariant feature extractor in the first stage. In the second stage, we leverage the gradients of classification error to generate intermediate samples to refine the classifiers together with original samples, making i-Sample also capable to be integrated into most domain adversarial adaptation methods without neural network modification. We have implemented a prototype system to evaluate i-Sample, which shows that i-Sample can effectively augment the performance of nowadays mainstream domain adversarial adaptation models for WiFi-based HAR, especially when source domain data is insufficient.

Comparative performance of machine learning models for the classification of human gait

The efficacy of human activity recognition (HAR) models mostly relies on the characteristics derived from domain expertise. The input of the classification algorithm consists of many characteristics that are utilized to accurately and effectively classify human physical activities. In contemporary research, machine learning techniques have been increasingly employed to automatically extract characteristics from unprocessed sensory input to develop models for Human Activity Recognition (HAR) and classify various activities. The primary objective of this research is to compare and contrast several machine learning models and determine a reliable and precise classification model for classifying activities. This study does a comparison analysis in order to assess the efficacy of 10 distinct machine learning models using frequently used datasets in the field of HAR. In this work, three benchmark public human walking datasets are being used. The research is conducted based on eight evaluating parameters. Based on the study conducted, it was seen that the machine learning classification models Random Forest, Extra Tree, and Light Gradient Boosting Machine had superior performance in all the eight evaluating parameters compared to specific datasets. Consequently, it can be inferred that machine learning significantly enhances performance within the area of Human Activity Recognition (HAR). This study can be utilized to provide suitable model selection for HAR-based datasets. Furthermore, this research can be utilized to facilitate the identification of various walking patterns for bipedal robotic systems.

Leveraging Smartphone Sensor Data and Machine Learning Model for Human Activity Recognition and Fall Classification

This study investigates the application of Machine Learning (ML) algorithms to classify human physical activities and detect different types of falls using smartphone sensor data. Smartphones equipped with various sensors offer new opportunities for healthcare and human activity monitoring, which are crucial for health and well-being, aligning with the United Nations’ sustainable development goals (SDG). The study used the ”Sensor Data Collector” Android application to collect smartphone sensor data. The data collection experiments involve ten healthy participants engaged in diverse physical activities and different fall events, utilizing accelerometers and gyroscope sensors. Different types of fall detection include forward, backward, left lateral, and right lateral fall, which are important from a seriousness of health point of view. Participants placed their smartphones in the front pocket of their trousers to ensure a fixed position during the data collection process. The collected datasets are preprocessed using a moving average filter to denoise and smoothen the data. Different statistical features and extracted and ML algorithms, namely Support Vector Machine (SVM), Random Forest (RF), Naive Bayes (NB), and K-Nearest Neighbors (KNN), are employed to create robust classifiers. SVM and RF performed exceptionally well in classifying normal physical activities, achieving a 99.09% accuracy. SVM excelled in daily walking activity classification with a 99.59% accuracy, while the K-NN model performed best for crucial different types of fall classification, achieving an accuracy of 85.71%. This research emphasizes the potential of ML models for accurate physical activity and various types of fall classification, expanding the horizons of smartphone-based healthcare applications and aligning with sustainable development goals.

GLULA: Linear attention-based model for efficient human activity recognition from wearable sensors.

Body-worn sensor data is used in monitoring patient activity during rehabilitation and also can be extended to controlling rehabilitation devices based on the activity of the person. The primary focus of research has been on effectively capturing the spatiotemporal dependencies in the data collected by these sensors and efficiently classifying human activities. With the increasing complexity and size of models, there is a growing emphasis on optimizing their efficiency in terms of memory usage and inference time for real-time usage and mobile computers. While hybrid models combining convolutional and recurrent neural networks have shown strong performance compared to traditional approaches, self-attention-based networks have demonstrated even superior results. However, instead of relying on the same transformer architecture, there is an opportunity to develop a novel framework that incorporates recent advancements to enhance speed and memory efficiency, specifically tailored for human activity recognition (HAR) tasks. In line with this approach, we present GLULA, a unique architecture for HAR. GLULA combines gated convolutional networks, branched convolutions, and linear self-attention to achieve efficient and powerful solutions. To enhance the performance of our proposed architecture, we employed manifold mixup as an augmentation variant which proved beneficial in limited data settings. Extensive experiments were conducted on five benchmark datasets: PAMAP2, SKODA, OPPORTUNITY, DAPHNET, and USC-HAD. Our findings demonstrate that GLULA outperforms recent models in the literature on the latter four datasets but also exhibits the lowest parameter count and close to the fastest inference time among state-of-the-art models.

FLSys: Toward an Open Ecosystem for Federated Learning Mobile Apps

This article presents the design, implementation, and evaluation of <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FLSys</b> , a mobile-cloud federated learning (FL) system, which can be a key component for an open ecosystem of FL models and apps. FLSys is designed to work on smart phones with mobile sensing data. It balances model performance with resource consumption, tolerates communication failures, and achieves scalability. In FLSys, different DL models with different FL aggregation methods can be trained and accessed concurrently by different apps. Furthermore, FLSys provides advanced privacy preserving mechanisms and a common API for third-party app developers to access FL models. FLSys adopts a modular design and is implemented in Android and AWS cloud. We co-designed FLSys with a human activity recognition (HAR) model. HAR sensing data was collected in the wild from 100+ college students during a 4-month period. We implemented HAR-Wild, a CNN model tailored to mobile devices, with a data augmentation mechanism to mitigate the problem of non-Independent and Identically Distributed data. A sentiment analysis model is also used to demonstrate that FLSys effectively supports concurrent models. This article reports our experience and lessons learned from conducting extensive experiments using simulations, Android/Linux emulations, and Android phones that demonstrate FLSys achieves good model utility and practical system performance.

Predicting Human Activity and Behavior Patterns from Cell Phone Data Using Machine Learning

Across the globe, Smartphones enjoy a high ownership rate. A person’s day-to-day activities are incomplete without the use of smartphones. These sensor-rich devices are often repurposed to collect various forms of data from the user in the domains of 1) app usage, 2) overall phone activity, 3) location, 4) day and night-time activity, 5) communication, 6) social media behavior and so on. This information collected from sensor and log data taken from smartphones can be used to create predictive measures extending to an individual’s Big Five personality traits and examine them. Human activity recognition (HAR) as a field has always tried to improve its performance in the aspects of feature extraction and improve the accuracy of recognition and prediction by performing various machine learning techniques on the data received from accelerometer and gyroscope sensors and creating convolutional neural networks to recognize the multiple activities such as walking, jogging, walking up or down the stairs, etc. Although HAR is a recognized research field, there have been significant challenges such as a) different movements for different individuals, b) limited training data, and c) complexity of performed activities. The idea of this project is to create a HAR model with the help of Machine Learning and Deep Learning, which will be more efficient in terms of performance and accuracy than the conventional models and try to overcome the challenges. That will be applied to the Smartphone data to predict a smartphone user’s activity at one point of the day. The idea is to also get an analysis of the behavioral traits of the user in terms of online expenditure and predict the date of the next purchase. That will be done to send personalized information such as offers and announcements to the user catered according to their needs to enhance their personal experience.

MLCNNwav: Multilevel Convolutional Neural Network With Wavelet Transformations for Sensor-Based Human Activity Recognition

Human activity recognition (HAR) is a rapidly growing field of research that aims to automatically identify and classify human motions and activities from different tracking devices, such as cameras and sensors. One of the most widely used sensor modalities for HAR is the smartphone, which has various sensors, such as gyroscopes, accelerometers, and GPS, that can provide rich information about a person’s movements and actions. HAR applications are essential for the Internet of Things (IoT) and smart home industries. We used the recent advances in deep learning techniques to develop a new HAR model for wearable sensors. The proposed model, MLCNNwav, relies on residual convolutional neural networks and one-dimensional trainable discrete wavelet transform. The multilevel CNN is designed to capture global features, whereas the wavelet transformation enhances the representation and generalization by learning activity-related features. Several deep learning approaches are compared to assess the superiority of the developed model. Four public benchmarks HAR datasets were used for the evaluation. The outcomes confirmed that the developed MLCNNwav recorded high accuracy rates on all datasets.

Application of 3D recognition algorithm based on spatio-temporal graph convolutional network in basketball pose estimation

In recent years, human motion recognition in computer vision has become a hot research direction in this field. Based on 2D human motion recognition technology, real-time extraction of motion features from 2D planes is used to recognize human movements. This method can only learn the position contour and color information of the image. It cannot directly reflect the motion situation, which results in low recognition accuracy and efficiency. In response to this issue, this study proposes a combination method of motion recognition and 3D pose estimation to recognize and classify basketball movements. First, the 2D skeleton model is obtained by extracting the feature information in the video action, which is converted into a 3D model, and the model is replaced by the time-space convolutional network to establish a human action recognition model. The experiment showed that when the number of iterations reached 6, the accuracy of the spatio-temporal graph convolutional network algorithm model reached 92%. Comparing the accuracy rates of different algorithm models, the average accuracy rates of convolutional neural network, long short memory network, graph convolution, long short model of action recognition and graph convolution model of action recognition were 61.6%, 65.4%, 72.5%, 76.8% and 90.3% respectively. The results show that the proposed 3D recognition algorithm can accurately recognize different basketball movements. This study can provide reference for basketball coaches and athletes in basketball training.

A light-weight skeleton human action recognition model with knowledge distillation for edge intelligent surveillance applications

Skeleton based human action recognition has evolved as one of the most important applications in multimedia IoT system. However, it requires extensive computation resource including high performance computing unites and large memory to train a deep mode with large number of parameters, which seriously limits it effectiveness and efficiency for edge intelligence multimedia IoT applications. In this paper, a knowledge distillation based light-weight deep model is proposed for skeleton human action recognition to meet the edge multimedia IoT applications. It can get competitive recognition performance in terms of learning accuracy for combination of AI model and edge surveillance equipment. On the one hand, to achieve desirable accuracy, we propose a deep pose-transition image representation method based on two-stream spatial–temporal architecture, which can mine the hidden features of color texture images in spatial and temporal domain, and fuse them for comprehensive discrimination before final classification. On the other hand, to increase the transfer learning ability to the student model on the edge device, we use tucker decomposition to weak the teacher model during knowledge transfer learning process. Finally, in order to validate the effectiveness of our proposal, we conducted extensive experiments to evaluate the proposed approach. The experimental results demonstrate that our proposal can realize deep model miniaturization to meet the requirement of edge multimedia IoT system and achieve the competitive performance.

Semantic Loss

Deep Learning models are a standard solution for sensor-based Human Activity Recognition (HAR), but their deployment is often limited by labeled data scarcity and models' opacity. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate these issues by infusing knowledge about context information into HAR deep learning classifiers. However, existing NeSy methods for context-aware HAR require computationally expensive symbolic reasoners during classification, making them less suitable for deployment on resource-constrained devices (e.g., mobile devices). Additionally, NeSy approaches for context-aware HAR have never been evaluated on in-the-wild datasets, and their generalization capabilities in real-world scenarios are questionable. In this work, we propose a novel approach based on a semantic loss function that infuses knowledge constraints in the HAR model during the training phase, avoiding symbolic reasoning during classification. Our results on scripted and in-the-wild datasets show the impact of different semantic loss functions in outperforming a purely data-driven model. We also compare our solution with existing NeSy methods and analyze each approach's strengths and weaknesses. Our semantic loss remains the only NeSy solution that can be deployed as a single DNN without the need for symbolic reasoning modules, reaching recognition rates close (and better in some cases) to existing approaches.

Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition

Human Activity Recognition (HAR) models often suffer from performance degradation in real-world applications due to distribution shifts in activity patterns across individuals. Test-Time Adaptation (TTA) is an emerging learning paradigm that aims to utilize the test stream to adjust predictions in real-time inference, which has not been explored in HAR before. However, the high computational cost of optimization-based TTA algorithms makes it intractable to run on resource-constrained edge devices. In this paper, we propose an Optimization-Free Test-Time Adaptation (OFTTA) framework for sensor-based HAR. OFTTA adjusts the feature extractor and linear classifier simultaneously in an optimization-free manner. For the feature extractor, we propose Exponential Decay Test-time Normalization (EDTN) to replace the conventional batch normalization (CBN) layers. EDTN combines CBN and Test-time batch Normalization (TBN) to extract reliable features against domain shifts with TBN's influence decreasing exponentially in deeper layers. For the classifier, we adjust the prediction by computing the distance between the feature and the prototype, which is calculated by a maintained support set. In addition, the update of the support set is based on the pseudo label, which can benefit from reliable features extracted by EDTN. Extensive experiments on three public cross-person HAR datasets and two different TTA settings demonstrate that OFTTA outperforms the state-of-the-art TTA approaches in both classification performance and computational efficiency. Finally, we verify the superiority of our proposed OFTTA on edge devices, indicating possible deployment in real applications. Our code is available at https://github.com/Claydon-Wang/OFTTA.

Deep Convolutional Neural Network with Symbiotic Organism Search-Based Human Activity Recognition for Cognitive Health Assessment.

Cognitive assessment plays a vital role in clinical care and research fields related to cognitive aging and cognitive health. Lately, researchers have worked towards providing resolutions to measure individual cognitive health; however, it is still difficult to use those resolutions from the real world, and therefore using deep neural networks to evaluate cognitive health is becoming a hot research topic. Deep learning and human activity recognition are two domains that have received attention for the past few years. The former is for its relevance in application fields like health monitoring or ambient assisted living, and the latter is due to their excellent performance and recent achievements in various fields of application, namely, speech and image recognition. This research develops a novel Symbiotic Organism Search with a Deep Convolutional Neural Network-based Human Activity Recognition (SOSDCNN-HAR) model for Cognitive Health Assessment. The goal of the SOSDCNN-HAR model is to recognize human activities in an end-to-end way. For the noise elimination process, the presented SOSDCNN-HAR model involves the Wiener filtering (WF) technique. In addition, the presented SOSDCNN-HAR model follows a RetinaNet-based feature extractor for automated extraction of features. Moreover, the SOS procedure is exploited as a hyperparameter optimizing tool to enhance recognition efficiency. Furthermore, a gated recurrent unit (GRU) prototype can be employed as a categorizer to allot proper class labels. The performance validation of the SOSDCNN-HAR prototype is examined using a set of benchmark datasets. A far-reaching experimental examination reported the betterment of the SOSDCNN-HAR prototype over current approaches with enhanced precision of 86.51% and 89.50% on Penn Action and NW-UCLA datasets, respectively.

A multimedia art interaction model for motion recognition based on data-driven model

Ergonomics is a relatively important part of user experience in multimedia art design. The study aims to improve human–computer interaction efficiency using a data-driven neural network model combined with video data and wearable devices to achieve high-precision human motion recognition. Firstly, the human motion skeleton information is extracted based on video information through the OpenPose framework, and the motion characteristics of each joint of the human body are calculated. Then, use the inertia data of the wearable bracelet to calculate the exercise intensity. At last, the two kinds of information are sent to recurrent neural networks (RNN) together to achieve high-precision human motion recognition. The innovative contribution of the article is to establish a multimodal fusion model for human activity recognition. The experimental results show that the recognition precision of the proposed method reaches 97.85%, much higher than the backpropagation neural network (BPNN) and K-nearest neighbor (KNN), whose precision is 94.35% and 90.12%, respectively. The method’s superior performance convinces us that the model can provide strong technical support for the interaction design of multimedia art.

Towards efficient video-based action recognition: context-aware memory attention network

Abstract Given the prevalence of surveillance cameras in our daily lives, human action recognition from videos holds significant practical applications. A persistent challenge in this field is to develop more efficient models capable of real-time recognition with high accuracy for widespread implementation. In this research paper, we introduce a novel human action recognition model named Context-Aware Memory Attention Network (CAMA-Net), which eliminates the need for optical flow extraction and 3D convolution which are computationally intensive. By removing these components, CAMA-Net achieves superior efficiency compared to many existing approaches in terms of computation efficiency. A pivotal component of CAMA-Net is the Context-Aware Memory Attention Module, an attention module that computes the relevance score between key-value pairs obtained from the 2D ResNet backbone. This process establishes correspondences between video frames. To validate our method, we conduct experiments on four well-known action recognition datasets: ActivityNet, Diving48, HMDB51 and UCF101. The experimental results convincingly demonstrate the effectiveness of our proposed model, surpassing the performance of existing 2D-CNN based baseline models. Article Highlights Recent human action recognition models are not yet ready for practical applications due to high computation needs. We propose a 2D CNN-based human action recognition method to reduce the computation load. The proposed method achieves competitive performance compared to most SOTA 2D CNN-based methods on public datasets.

A Survey on Different Deep Learning Model for Human Activity Recognition based on Application

The field of human activity recognition (HAR) seeks to identify and classify an individual's unique movements or activities. However, recognizing human activity from video is a challenging task that requires careful attention to individuals, their behaviors, and relevant body parts. Multimodal activity recognition systems are necessary for many applications, including video surveillance systems, human-computer interfaces, and robots that analyze human behavior. This study provides a comprehensive analysis of recent breakthroughs in human activity classification, including different approaches, methodologies, applications, and limitations. Additionally, the study identifies several challenges that require further investigation and improvements. The specifications for an ideal human activity recognition dataset are also discussed, along with a thorough examination of the publicly available human activity classification datasets.

Discovering and Recognizing of Imbalance Human Activity in Healthcare Monitoring using Data Resampling Technique and Decision Tree Model

Human activity recognition model is vital and has been use in healthcare monitoring system. Bespoke multi-modal sensors were used such as accelerometer, gyroscope, GPS, temperature, pressure mat etc. Hence, the activities involved may varied resulted on class imbalance issue therefore, the model accuracy also degraded and may not provide the desired results in all aspects. Resampling method addressed as Synthetically Minority Oversampling Technique and Tomek Link (Smote Tomek) is proposed to balance the target classes. Moreover, many classification algorithms such as Logistic Regression, SVM and Decision Tree were selected for the experiments on two datasets namely MARBLE that was publicly available and MARDA dataset. The classification accuracy of 98.36% for the MARBLE dataset and 97.45% for the MARDA dataset. While executing the classification model for training and testing data, the time has also been calculated for time efficiency. The Decision Tree model has the fastest execution time compared with other models. The execution training time for MARBLE and MARDA were: 6.51ms, testing: 12.90ms and, training: 41.30ms, testing: 1.30ms, respectively. Subsequently, the decision tree model can be deployed in a healthcare system dashboard for effective monitoring and efficient decision making.

A smart e-health framework for monitoring the health of the elderly and disabled

The healthcare sector is experiencing a significant transformation due to the widespread adoption of IoT-based systems, especially in the care of elderly and disabled individuals who can now be monitored through portable and wearable devices. The widespread implementation of IoT-based systems in the healthcare sector holds immense potential for improving care and ensuring the well-being of vulnerable populations. In our paper, we introduce an e-health framework that utilizes real-time data from inertial, ECG, and video sensors to monitor the health and activities of these individuals. Our framework employs edge computing for efficient analysis and generates notifications based on data analysis while prioritizing privacy by activating multimedia sensors only when necessary. To achieve accurate results, we evaluated each component of our framework separately. Using the MHEALTH dataset, our proposed machine learning model achieves accuracies of 99.97% for inertial sensors’ human activity recognition (HAR) and 99.14% for ECG sensors. We also evaluated our first-level inconsistency detection-based anomaly proposal mechanism using one user’s data, demonstrating its proof-of-concept. Furthermore, our video-based HAR and fall detection module achieve an accuracy of 86.97% on the well-known DMLSmartActions dataset. We successfully deployed our proposed HAR model with inertial sensors in a controlled experimental environment, yielding promising results with an accuracy of 96% and an F1 score of 90.52%. These evaluations validate the effectiveness of our framework in monitoring the health and activities of elderly and disabled individuals.

Advanced Human Activity Recognition through Data Augmentation and Feature Concatenation of Micro-Doppler Signatures

Developing accurate classification models for radar-based Human Activity Recognition (HAR), capable of solving real-world problems, depends heavily on the amount of available data. In this paper, we propose a simple, effective, and generalizable data augmentation strategy along with preprocessing for micro-Doppler signatures to enhance recognition performance. By leveraging the decomposition properties of the Discrete Wavelet Transform (DWT), new samples are generated with distinct characteristics that do not overlap with those of the original samples. The micro-Doppler signatures are projected onto the DWT space for the decomposition process using the Haar wavelet. The returned decomposition components are used in different configurations to generate new data. Three new samples are obtained from a single spectrogram, which increases the amount of training data without creating duplicates. Next, the augmented samples are processed using the Sobel filter. This step allows each sample to be expanded into three representations, including the gradient in the x-direction (Dx), y-direction (Dy), and both x- and y-directions (Dxy). These representations are used as input for training a three-input convolutional neural network-long short-term memory support vector machine (CNN-LSTM-SVM) model. We have assessed the feasibility of our solution by evaluating it on three datasets containing micro-Doppler signatures of human activities, including Frequency Modulated Continuous Wave (FMCW) 77 GHz, FMCW 24 GHz, and Impulse Radio Ultra-Wide Band (IR-UWB) 10 GHz datasets. Several experiments have been carried out to evaluate the model's performance with the inclusion of additional samples. The model was trained from scratch only on the augmented samples and tested on the original samples. Our augmentation approach has been thoroughly evaluated using various metrics, including accuracy, precision, recall, and F1-score. The results demonstrate a substantial improvement in the recognition rate and effectively alleviate the overfitting effect. Accuracies of 96.47%, 94.27%, and 98.18% are obtained for the FMCW 77 GHz, FMCW 24 GHz, and IR- UWB 10 GHz datasets, respectively. The findings of the study demonstrate the utility of DWT to enrich micro-Doppler training samples to improve HAR performance. Furthermore, the processing step was found to be efficient in enhancing the classification accuracy, achieving 96.78%, 96.32%, and 100% for the FMCW 77 GHz, FMCW 24 GHz, and IR-UWB 10 GHz datasets, respectively.

MAWKDN: A Multimodal Fusion Wavelet Knowledge Distillation Approach Based on Cross-View Attention for Action Recognition

The recognition prerformance of existing vision-based human action recognition (HAR) methods is greatly reduced in the case of low camera resolution or occlusion. Wearable sensors can provide complementary information to alleviate this problem. It is challenging to construct a robust HAR model using multimodal wearable-sensor data. In this paper, we propose a cross-Attention-based Multimodal fusion Wavelet Knowledge Distillation Network (MAWKDN) method to guide recognition from video data by acquiring complementary information from wearable sensors and reduce the noise effects through wavelet knowledge distillation, which improves the robustness of the model. A multi-attention dilated convolution kernel residual network including dilated convolution and an attention mechanism is constructed to extract features from various sensor modalities and fuse the various modal data through the cross-view attention method to acquire additional information from different modalities. To reduce the modal differences between different modalities of the teacher and student networks and acquire similar semantic knowledge, we learn the information between different modalities by constructing a graph structure of convolutional layer features, and computing the semantic preservation loss between the teacher and student networks. To reduce the influence of noise in the input data, we construct the loss of wavelet knowledge distillation, which transforms the image through the discrete wavelet transform and only retains the low frequency features to extract the useful information. The top-1 accuracy achieved on the UTD-MHAD (99.31%), Berkeley-MHAD (99.40%) and the F1-score on the MMAct (85.26% based on cross-session) dataset prove the superior performance of MAWKDN compared with the state-of-the-art HAR methods. Moreover, we demonstrate the robustness of the MAWKDN approach on the noise-added UTD-MHAD dataset.