Human Activity Classification Research Articles

High Accuracy WiFi-Based Human Activity Classification System with Time-Frequency Diagram CNN Method for Different Places.

Older people are very likely to fall, which is a significant threat to the health. However, falls are preventable and are not necessarily an inevitable part of aging. Many different fall detection systems have been developed to help people avoid falling. However, traditional systems based on wearable devices or image recognition-based have many disadvantages, such as user-unfriendly, privacy issues. Recently, WiFi-based fall detection systems try to solve the above problems. However, there is a common problem of reduced accuracy. Since the system is trained at the original signal collecting/training place, however, the application is at a different place. The proposed solution only extracts the features of the changed signal, which is caused by a specific human action. To implement this, we used Channel State Information (CSI) to train Convolutional Neural Networks (CNNs) and further classify the action. We have designed a prototype to test the performance of our proposed method. Our simulation results show an average accuracy of same place and different place is 93.2% and 90.3%, respectively.

A deep survey on supervised learning based human detection and activity classification methods

Human detection and activity recognition is very important research area in the healthcare, video surveillance, pedestrian detection, intelligent vehicle system and home care center. Among the various human activity detection frameworks, the statistical based approach were most intensively studied and used in practice in which pattern recognition was traditionally formulated. More recently, supervised learning based techniques and methods imported from statistical learning theory have deserved increasing attention. Many new supervised learning methods such as transfer learning, multi-instance learning, and the new trends in deep learning techniques have used for the formulation of solutions to the human activity detection. This paper reviews the automatic human detection and their activity recognition in the video sequences and static images. We explain several problems of human detection and activity recognition in different steps such as processing, segmentation of human features extraction and classification. Moreover, discuss the problems in each step and provide the recent state- of-the-art methods, gaps between recent methods, technical difficulties, applications and their challenges. Several features extraction techniques and corresponding problems for human classification have been discussed in details. Special emphasis have been given on convolution neural network that solves the problem of human segmentation, efficient classification and activity recognition. The objective of this review paper is to summarize and review related of the established and recent methods used in various stages of a human detection and activity classification system and identify research topics and applications that are at the forefront of this exciting and challenging field. Further, the evaluation protocols (i.e. datasets and simulation tools) and possible solution of current limitation have been discussed briefly in this survey.

Human activity recognition using penalized support vector machines and Hidden Markov Models in multimodal systems

Human activity detection has evolved due to the advances and developments of machine learning techniques, which have enabled solutions to new challenges without ignoring prevalent difficulties that need to be addressed. One of the challenges is the learning model’s sensitivity regarding the unbalanced, atypical, and overlapping information that directly affects the performance of the model. This article evaluates a methodology for the classification of human activities that penalizes defective information. The methodology is carried out through two redundant classifiers, a penalized support vector machine that detects the sub-movements (micro-movements) and the Marvok Hidden Model that predicts the activity given the micro- movements sequence. The performance of the method was compared with state-of-the-art techniques, and the findings suggested significative advance in the detection of micro-movements compared to the data obtained with non-penalized paradigms. In this research, an adequate performance is found in the classification of primitive movements, with hit rates of 95.15% for the Kinect One®, 96.86% for the IMU sensor network, and 67.51% for the EMG sensor network.

Generalization of Channel Micro-Doppler Capacity Evaluation for Improved Finer-Grained Human Activity Classification Using MIMO UWB Radar

In this article, a channel micro-Doppler ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu \text{D}$ </tex-math></inline-formula> ) signatures’ quantitative evaluation method for multiple-input–multiple-output (MIMO) radar system, named relative signal–noise ratio in the time–frequency domain and denoted as RSNRtf, is introduced. The impact on RSNR <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tf</sub> by some key dominant factors in real practical and diverse detection environments was theoretically analyzed first. Then, this proposed method is exploited to conduct effective and superior channel selection for improved human activity classification at random human target positions. Based on numerous MIMO stepped-frequency continuous wave (SFCW) radar data from eight subjects with seven activities at nine positions in a through-wall within a view angle of ±20° and up to 6-m distance away, the influence of various primary factors on the proposed method is investigated. Investigative experiments showed that the proposed method has a wide range of applicability and a high interference tolerance even for different detection distances, angles, activities, and subjects. Classification experimental results based on typical classifiers also showed that the proposed method could effectively improve human activity classification. Moreover, we can claim that the proposed method can achieve a higher outstanding performance in the channel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu \text{D}$ </tex-math></inline-formula> capacity evaluation and enhanced classification for the MIMO-radar-based human activity classification in free space or with a wider detection field with large angle range.

BeSafe B2.0 Smart Multisensory Platform for Safety in Workplaces.

Wearable technologies are becoming a profitable means of monitoring a person’s health state, such as heart rate and physical activity. The use of the smartwatch is becoming consolidated, not only as a novelty but also as a very useful tool for daily use. In addition, other devices, such as helmets or belts, are beneficial for monitoring workers and the early detection of any anomaly. They can provide valuable information, especially in work environments, where they help reduce the rate of accidents and occupational diseases, which makes them powerful Personal Protective Equipment (PPE). The constant monitoring of the worker’s health can be done in real-time, through temperature, falls, noise, impacts, or heart rate meters, activating an audible and vibrating alarm when an anomaly is detected. The gathered information is transmitted to a server in charge of collecting and processing it. In the first place, this paper provides an exhaustive review of the state of the art on works related to electronics for human activity behavior. After that, a smart multisensory bracelet, combined with other devices, developed a control platform that can improve operators’ security in the working environment. Artificial Intelligence and the Internet of Things (AIoT) bring together the information to improve safety on construction sites, power stations, power lines, etc. Real-time and historic data is used to monitor operators’ health and a hybrid system between Gaussian Mixture Model and Human Activity Classification. That is, our contribution is also founded on the use of two machine learning models, one based on unsupervised learning and the other one supervised. Where the GMM gave us a performance of 80%, 85%, 70%, and 80% for the 4 classes classified in real time, the LSTM obtained a result under the confusion matrix of 0.769, 0.892, and 0.921 for the carrying-displacing, falls, and walking-standing activities, respectively. This information was sent in real time through the platform that has been used to analyze and process the data in an alarm system.

Study on feature processing schemes for deep‐learning‐based human activity classification using frequency‐modulated continuous‐wave radar

Radar is an attractive sensor for classifying human activity because of its invariance to the environment and its ability to operate under low lighting conditions and through obstacles. Classification for human activity finds applications in human–computer interfaces, user-intent understanding and contextual-aware smart homes. Moreover, frequency-modulated continuous-wave technology enables radar systems to retrieve both micro-Doppler and range profiles from humans, which can then be used to recognize target movements. Several feature processing schemes are studied on signatures obtained from a compact, short-range, 60-GHz, frequency-modulated continuous-wave radar for various human activities. The contribution of distinct features to the performance of classification is analysed and novel convolutional- and recurrent-neural-network architectures are consequently presented that efficiently combine range and Doppler information for classifying human activity. In particular, applying multibranch deep learning techniques to high-resolution range profiles and Doppler spectrograms achieves performance similar to that using range-Doppler map videos although it requires less computational and memory resources.

3D Human Activity Classification with 3D Zernike Moment Based Convolutional, LSTM-Deep Neural Networks

In this paper, we propose a method for classification 3D human activities using the complementarity of CNNs, LSTMs, and DNNs by combining them into one unified architecture called CLDNN. Our approach is based on the prediction of 3D Zernike Moments of some relevant joints of the human body through Kinect using the Kinect Activity Recognition Dataset. KARD includes 18 activities and each activity consists of real-world point clouds that have been carried out 3 times by 10 different subjects. We introduce the potential for the 3D Zernike Moment feature extraction approach via a 3D point cloud for human activity classification, and the ability to be trained and generalized independently from datasets using the Deep Learning methods. The experimental results obtained on datasets with the proposed system has correctly classified 96.1% of the activities. CLDNN has been shown to provide a 5% relative improvement over LSTM, the strongest of the three individual models.

Object Priors for Classifying and Localizing Unseen Actions

This work strives for the classification and localization of human actions in videos, without the need for any labeled video training examples. Where existing work relies on transferring global attribute or object information from seen to unseen action videos, we seek to classify and spatio-temporally localize unseen actions in videos from image-based object information only. We propose three spatial object priors, which encode local person and object detectors along with their spatial relations. On top we introduce three semantic object priors, which extend semantic matching through word embeddings with three simple functions that tackle semantic ambiguity, object discrimination, and object naming. A video embedding combines the spatial and semantic object priors. It enables us to introduce a new video retrieval task that retrieves action tubes in video collections based on user-specified objects, spatial relations, and object size. Experimental evaluation on five action datasets shows the importance of spatial and semantic object priors for unseen actions. We find that persons and objects have preferred spatial relations that benefit unseen action localization, while using multiple languages and simple object filtering directly improves semantic matching, leading to state-of-the-art results for both unseen action classification and localization.

Detection and Classification of Human Activity for Emergency Response in Smart Factory Shop Floor

Factory shop floor workers are exposed to threats and accidents due to their encounters with tools, equipment, and toxic materials. There are cases of occupational accidents resulting in injuries to workers and precipitating lawsuits, which on the other hand affect company’s operational cost. To ensure the safety of workers within the shop floor, there is a need for proactive activity monitoring. Such activities include detection of falling objects, abnormal vibration, and movement of humans within an acceptable area of the factory floor. Breathing sensor-based monitoring of workers in the smart factory shop floor can also be implemented. This is for the detection of human activity, especially in cases where workers are in isolation with no available emergency assistance. Internet of Things (IoT), Industrial Internet of Things (IIoT), and machine learning (ML) have enabled so many possibilities in this area. In this study, we present a simple test-bed, which is made up of a vibration sensor, a breathing and movement sensor, and a Light Detection and Ranging (LIDAR) sensor. These sensors were used to gather normal and abnormal data of human activities at the factory. We developed a dataset based on possible real-life situations and it is made up of about 10,000 data points. The data was split with a ratio of 75:25 for training and testing the model. We investigated the performance of different ML algorithms, including support vector machine (SVM), linear regression, naive Bayes (NB), K-nearest neighbor (KNN), and convolutional neural network (CNN). From our experiments, the CNN model outperformed other algorithms with an accuracy of 99.45%, 99.78%,100%, and 100%, respectively, for vibration, movement, breathing, and distance. We have also successfully developed a dataset to assist the research community in this field.

Gradient local auto-correlation features for depth human action recognition

Human action classification is a dynamic research topic in computer vision and has applications in video surveillance, human–computer interaction, and sign-language recognition. This paper aims to present an approach for the categorization of depth video oriented human action. In the approach, the enhanced motion and static history images are computed and a set of 2D auto-correlation gradient feature vectors is obtained from them to describe an action. Kernel-based Extreme Learning Machine is used with the extracted features to distinguish the diverse action types promisingly. The proposed approach is thoroughly assessed for the action datasets namely MSRAction3D, DHA, and UTD-MHAD. The approach achieves an accuracy of 97.44% for MSRAction3D, 99.13% for DHA, and 88.37% for UTD-MHAD. The experimental results and analysis demonstrate that the classification performance of the proposed method is considerable and surpasses the state-of-the-art human action classification methods. Besides, from the complexity analysis of the approach, it is turn out that our method is consistent for the real-time operation with low computational complexity.

Classification of human activity using radar and video multimodal learning

In the defence and security domain, camera systems are widely used for surveillance. The major advantage of using camera systems for surveillance is that they provide high-resolution imagery, which is easy to interpret. However, the use of camera systems and optical imagery has some drawbacks, especially for application in the military domain. In poor lighting conditions, dust or smoke the image quality degrades and, additionally, cameras cannot provide range information too. These drawbacks can be mitigated by exploiting the strengths of radar. Radar performance can be largely maintained during the night, in various weather conditions and in dust and smoke. Moreover, radar provides the distance to detected objects. Since, the strongpoints and weaknesses of radar and camera systems seem complementary, a natural question is: can radar and camera systems learn from each other? Here the potential of radar/video multimodal learning is evaluated for human activity classification. The novelty of this work is the use of radar spectrograms and related video frames for classification with a multimodal neural network. Radar spectrograms and video frames are both two-dimensional images, but the information they contain is of different nature. This approach was adopted to limit the required preprocessing load, while maintaining the complementary nature of the sensor data.

Human Activity Classification Based on Point Clouds Measured by Millimeter Wave MIMO Radar With Deep Recurrent Neural Networks

We investigate the feasibility of classifying human activities measured by a MIMO radar in the form of a point cloud. If a human subject is measured by a radar system that has a very high angular azimuth and elevation resolution, scatterers from the body can be localized. When precisely represented, individual points form a point cloud whose shape resembles that of the human subject. As the subject engages in various activities, the shapes of the point clouds change accordingly. We propose to classify human activities through recognition of point cloud variations. To construct a dataset, we used an FMCW MIMO radar to measure 19 human subjects performing 7 activities. The radar had 12 TXs and 16 RXs, producing a 33×31 virtual array with approximately 3.5 degrees of angular resolution in azimuth and elevation. To classify human activities, we used a deep recurrent neural network (DRNN) with a two-dimensional convolutional network. The convolutional filters captured point clouds' features at time instance for sequential input into the DRNN, which recognized time-varying signatures, producing a classification accuracy exceeding 97%.

Human activity classification using Decision Tree and Naïve Bayes classifiers

With rapid development in wireless sensor networks and continuous improvements in developing artificial intelligence-based scientific solutions, the concept of ambient assisted living has been encouraged and adopted. This is due to its widespread applications in smart homes and healthcare. In this regard, the concept of human activity recognition (HAR) & classification has drawn numerous researchers’ attention as it improves the quality of life. However, before using this concept in real-time scenarios, it is required to analyse its performance following activities of daily living using benchmarked data set. In this continuation, this work has adopted the activity classification algorithms to improve their accuracy further. These algorithms can be used as a benchmark to analyse others’ performance. Initially, the raw 3-axis accelerometer data is first preprocessed to remove noise and make it feasible for training and classification. For this purpose, the sliding window algorithm, linear and Gaussian filters have been applied to raw data. Then Naive Bayes (NB) and Decision Tree (DT) classification algorithms are used to classify human activities such as: sitting, standing, walking, sitting down and standing up. From results, it can be seen that maximum 89.5% and 99.9% accuracies are achieved using NB and DT classifiers with Gaussian filter. Furthermore, we have also compared the obtained results with its counterpart algorithms in order to prove its effectiveness.

Simulation framework for activity recognition and benchmarking in different radar geometries

Radar micro-Doppler signatures have been proposed for human monitoring and activity classification for surveillance and outdoor security, as well as for ambient assisted living in healthcare-related applications. A known issue is the performance reduction when the target is moving tangentially to the line of sight of the radar. Multiple techniques have been proposed to address this, such as multistatic radar and to some extent, interferometric (IF) radar. A simulator is presented to generate synthetic data representative of eight radar systems (monostatic, circular multistatic and in-line multistatic [IM] and IF) to quantify classification performances as a function of aspect angles and deployment geometries. This simulator allows an unbiased performance evaluation of different radar systems. Six human activities are considered with signatures originating from motion-captured data of 14 different subjects. The classification performances are analysed as a function of aspect angles ranging from 0° to 90° per activity and overall. It demonstrates that IF configurations are more robust than IM configurations. However, IM performs better at angles below 55° before IF configurations take over.

Lightweight Deep Learning Model in Mobile-Edge Computing for Radar-Based Human Activity Recognition

Radar-based human activity recognition (HAR) has great potential in many fields, such as surveillance, smart homes, and human-computer interaction. Complex deep neural networks have brought significant improvement in classification performance but also a surge of computational cost and the number of parameters, which makes it challenging to deploy in mobile devices. However, the existing studies in this area mainly focus on improving the classification accuracy. In this article, we propose an extremely efficient convolutional neural network (CNN) architecture named Mobile-RadarNet, which is specially designed for human activity classification based on micro-Doppler signatures. The new architecture exploits 1-D depthwise convolutions and pointwise convolutions to build lightweight CNN architecture. The experiments on a seven-class human activity data set demonstrate that the proposed Mobile-RadarNet can achieve high classification accuracy meanwhile to keep the computational complexity at an extremely low level, and thus has great potential to be deployed in the mobile devices.

Continual Learning of Micro-Doppler Signature-Based Human Activity Classification

Human activity classification based on micro-Doppler signatures measured by radar recently has been successfully applied in diverse applications due to the improvement of machine learning methods, e.g., deep convolutional neural networks (DCNNs). Despite the success, those methods encounter a common practical problem when all the radar data are not readily available before training a model but sequentially arrive as the learning continues, e.g., during surveillance or search-and-rescue operations. That is, when DCNN is naively utilized in such settings, it is well-known that catastrophic forgetting of past learned tasks occurs; hence, more robust continual learning methods should be developed. To that end, we apply several state-of-the-art methods for two practical continual learning scenarios in activity classification, i.e., when the data for a subject and an activity class arrive incrementally, respectively, and compare the competitiveness of those methods. To the best of our knowledge, this is the first comparative study of continual learning methods for the classification based on micro-Doppler signatures—we find that exemplar memory-based methods particularly become very effective for both scenarios, not only for the performance but also for the memory usage.

Monitoring Real-Time Personal Locomotion Behaviors Over Smart Indoor-Outdoor Environments Via Body-Worn Sensors

The monitoring of human physical activities using wearable sensors, such as inertial-based sensors, plays a significant role in various current and potential applications. These applications include physical health tracking, surveillance systems, and robotic assistive technologies. Despite the wide range of applications, classification and recognition of human activities remains imprecise and this may contribute to unfavorable reactions and responses. To improve the recognition of human activities, we designed a dataset in which ten participants (five male and five female) performed 11 different activities wearing three body-worn inertial sensors in different locations on the body. Our model extracts data via a hierarchical feature-based technique. These features include time, wavelet, and time-frequency domains, respectively. Stochastic gradient descent (SGD) is then introduced to optimize selective features. The selected features with optimized patterns are further processed by multi-layered kernel sliding perceptron to develop adaptive learning for the classification of physical human activities. Our proposed model was experimentally evaluated and applied on three benchmark datasets: IM-WSHA, a self-annotated dataset, PAMAP2 dataset which is comprised of daily living activities, and an HuGaDB, a dataset which contains physical activities for aging people. The experimental results show that the proposed method achieves better results and outperforms others in terms of recognition accuracy, achieving an accuracy rate of 83.18%, 94.16%, and 92.50% respectively, when IM-WSHA, PAMAP2, and HuGaDB datasets are applied.

Deep Learning Shared Bandpass Filters for Resource-Constrained Human Activity Recognition

Classification of human activities from wearable sensor data is challenged by inter-subject variance and resource-constrained platforms. We address these issues with SincEMG, a deep neural network that exploits digital signal processing concepts and transfer learning to reduce model size for activity recognition on raw sensor data. The model’s first layer decomposes signals into frequency bands using finite impulse response filters optimized directly from the data. The subsequent convolutional layers downsample across time and aggregate the first layer’s band data. Batch normalization and dropout help to regularize intermediate layer outputs. This approach reduces compute requirements by decreasing the number of learned parameters and eliminating any significant data pre-processing. In addition to these improvements, the model’s first layer learns a set of bandpass filters, which provide insight into predictive regions of the source spectrum. We evaluate SincEMG using two publicly available surface electromyography datasets. Our model uses far fewer parameters and achieves state-of-the-art results with 98.53% accuracy for 7-classes and 68.45% accuracy for 18-classes.

Industry 4.0-Oriented Deep Learning Models for Human Activity Recognition

According to the Industry 4.0 vision, humans in a smart factory, should be equipped with formidable and seamless communication capabilities and integrated into a cyber-physical system (CPS) that can be utilized to monitor and recognize human activity via artificial intelligence (e.g., deep learning). Recent advances in the accuracy of deep learning have contributed significantly to solving the human activity recognition issues but it remains necessary to develop high performance deep learning models that provide greater accuracy. In this paper, three models: long short-term memory (LSTM), convolutional neural network (CNN), and combined CNN-LSTM are proposed for classification of human activities. These models are applied to a dataset collected from 36 persons engaged in 6 classes of activities – downstairs, jogging, sitting, standing, upstairs, and walking. The proposed models are trained using TensorFlow framework with a hyper-parameter tuning method to achieve high accuracy. Experimentally, confusion matrices and receiver operating characteristic (ROC) curves are used to assess the performance of the proposed models. The results illustrate that the hybrid model CNN-LSTM provides a better performance than either LSTM or CNN in the classification of human activities. The CNN-LSTM model provides the best performance, with a testing accuracy of 97.76%, followed by the LSTM with a testing accuracy of 96.61%, while the CNN shows the least testing accuracy of 94.51%. The testing loss rates for the LSTM, CNN, and CNN-LSTM are 0.236, 0.232, and 0.167, respectively, while the precision, recall, F1-Measure, and the area under the ROC curves (AUCS) for the CNN-LSTM are 97.75%, 97.77%, 97.76%, and 100%, respectively.

Human Locomotion Activity Recognition using Spectral Analysis and Convolutional Neural Networks

To enable a collaborative robot or exoskeleton robot to better support humans more safely and efficiently, understanding human behaviours is an essential enabling technology for the machines to make decisions on optimal control strategies. This work introduces an algorithm for classification of human locomotion activities using inertial data captured with Inertial Measurement Unit (IMU) to support the control of robots, exoskeletons and many other applications. The proposed approach, to recognise human locomotion activities and gait events, includes two main steps: 1) applying spectral analysis on the inertial signals to transform the data into time-frequency representation; and 2) classify the time-frequency data of an image to be recognised using Convolutional Neural Networks (CNN). There are six activities considered in the work. The highest accuracy in classification with a sub-set of 3 classes is 99% demonstrating the promising result in being applicable for real applications.