Action Recognition Method Research Articles

Brain Activity Recognition Method Based on Attention-Based RNN Mode

Brain activity recognition based on electroencephalography (EEG) marks a major research orientation in intelligent medicine, especially in human intention prediction, human–computer control and neurological diagnosis. The literature research mainly focuses on the recognition of single-person binary brain activity, which is limited in the more extensive and complex scenarios. Therefore, brain activity recognition in multiperson and multi-objective scenarios has aroused increasingly more attention. Another challenge is the reduction of recognition accuracy caused by the interface of external noise as well as EEG’s low signal-to-noise ratio. In addition, traditional EEG feature analysis proves to be time-intensive and it relies heavily on mature experience. The paper proposes a novel EEG recognition method to address the above issues. The basic feature of EEG is first analyzed according to the band of EEG. The attention-based RNN model is then adopted to eliminate the interference to achieve the purpose of automatic recognition of the original EEG signal. Finally, we evaluate the proposed method with public and local data sets of EEG and perform lots of tests to investigate how factors affect the results of recognition. As shown by the test results, compared with some typical EEG recognition methods, the proposed method owns better recognition accuracy and suitability in multi-objective task scenarios.

Human Behavior Recognition Method based on Two-layer LSTM Network with Attention Mechanism

Aiming at the problem that the exiting human skeleton-based action recognition methods cannot fully extract the relevant information before and after the action, resulting in low utilization efficiency of skeleton points, we propose a two-layer LSTM (long short term memory) network with attention mechanism. The network has two layers, the first LSTM network is used for skeleton coding and initialization of system storage units and the second LSTM network integrates attention mechanism to further process the data of the first layer network. An algorithm is designed to assign different weights to skeleton points according to the importance of human body, which greatly increases the recognition accuracy. Action classification is accomplished by multiple support vector machines. Through training and testing, the average recognition rate of 98.5% is achieved on KTH dataset. The experimental result shows that the proposed method is effective in human behavior recognition.

A Lightweight Hierarchical Model with Frame-Level Joints Adaptive Graph Convolution for Skeleton-Based Action Recognition

In skeleton-based human action recognition methods, human behaviours can be analysed through temporal and spatial changes in the human skeleton. Skeletons are not limited by clothing changes, lighting conditions, or complex backgrounds. This recognition method is robust and has aroused great interest; however, many existing studies used deep-layer networks with large numbers of required parameters to improve the model performance and thus lost the advantage of less computation of skeleton data. It is difficult to deploy previously established models to real-life applications based on low-cost embedded devices. To obtain a model with fewer parameters and a higher accuracy, this study designed a lightweight frame-level joints adaptive graph convolutional network (FLAGCN) model to solve skeleton-based action recognition tasks. Compared with the classical 2s-AGCN model, the new model obtained a higher precision with 1/8 of the parameters and 1/9 of the floating-point operations (FLOPs). Our proposed network characterises three main improvements. First, a previous feature-fusion method replaces the multistream network and reduces the number of required parameters. Second, at the spatial level, two kinds of graph convolution methods capture different aspects of human action information. A frame-level graph convolution constructs a human topological structure for each data frame, whereas an adjacency graph convolution captures the characteristics of the adjacent joints. Third, the model proposed in this study hierarchically extracts different levels of action sequence features, making the model clear and easy to understand; further, it reduces the depth of the model and the number of parameters. A large number of experiments on the NTU RGB + D 60 and 120 data sets show that this method has the advantages of few required parameters, low computational costs, and fast speeds. It also has a simple structure and training process that make it easy to deploy in real-time recognition systems based on low-cost embedded devices.

Activity recognition using multiplex limited penetrable visibility graph

To solve the dynamic problem of different activities in human activity recognition research, an activity recognition method based on a multiplex limited penetrable visibility graph is proposed. The 21 pressure values for each sampling are mapped to nodes in the first-layer network; then the average path length of nodes in the asynchronous periodic network is obtained, and the second-layer network used to explore different activities is built. Finally, the characteristic parameters and dynamic characteristics of different activities are explored and analyzed. The experimental results demonstrate that through the joint distribution of the average clustering coefficient and the maximum degree parameter of the node, the discrimination problems of different postures can be better realized, and it has good adaptability. It provides a new approach to gait recognition research that can be used in medical clinical diagnosis, rehabilitation training, and public health.

Action recognition of construction workers under occlusion

The capability to recognize the actions of workers through videos with adequate accuracy is of great significance in construction engineering. Most of the current action recognition models, however, tend to loss their effectiveness in case of occlusion problem. In order to address this issue, a skeleton-based action recognition method is proposed in this paper. The essence of the proposed approach is to transform the occlusion problem into the process of imputing missing values of the feature matrix. The method can be implemented by following three steps. Firstly, a new encoding technique is developed to transform the recognized worker skeleton into a feature matrix. Followed by integrating an attention model into the GAN (Generative Adversarial Networks)-based data imputation model, the missing data can be effectively focused on. Finally, the Resnet (Residual Network) classification model will be introduced to determine the category of actions through the classification of imputed feature matrix. To verify the accuracy of the proposed method, a dataset containing eight actions is established based on which the method will be tested. Analytical results indicate that the method can improve the reorganization accuracy to 93.89% when there is no occlusion, and the reorganization accuracy to 86.62% when there is occlusion. At the same time, the proposed method has limitations, it will show the greatest advantage with a moderate occlusion rate.

Real-Time Action Recognition System for Elderly People Using Stereo Depth Camera.

Smart technologies are necessary for ambient assisted living (AAL) to help family members, caregivers, and health-care professionals in providing care for elderly people independently. Among these technologies, the current work is proposed as a computer vision-based solution that can monitor the elderly by recognizing actions using a stereo depth camera. In this work, we introduce a system that fuses together feature extraction methods from previous works in a novel combination of action recognition. Using depth frame sequences provided by the depth camera, the system localizes people by extracting different regions of interest (ROI) from UV-disparity maps. As for feature vectors, the spatial-temporal features of two action representation maps (depth motion appearance (DMA) and depth motion history (DMH) with a histogram of oriented gradients (HOG) descriptor) are used in combination with the distance-based features, and fused together with the automatic rounding method for action recognition of continuous long frame sequences. The experimental results are tested using random frame sequences from a dataset that was collected at an elder care center, demonstrating that the proposed system can detect various actions in real-time with reasonable recognition rates, regardless of the length of the image sequences.

Action Recognition Algorithm of Spatio–Temporal Differential LSTM Based on Feature Enhancement

The Long Short-Term Memory (LSTM) network is a classic action recognition method because of its ability to extract time information. Researchers proposed many hybrid algorithms based on LSTM for human action recognition. In this paper, an improved Spatio–Temporal Differential Long Short-Term Memory (ST-D LSTM) network is proposed, an enhanced input differential feature module and a spatial memory state differential module are added to the network. Furthermore, a transmission mode of ST-D LSTM is proposed; this mode enables ST-D LSTM units to transmit the spatial memory state horizontally. Finally, these improvements are added into classical Long-term Recurrent Convolutional Networks (LRCN) to test the new network’s performance. Experimental results show that ST-D LSTM can effectively improve the accuracy of LRCN.

SVSL: A Human Activity Recognition Method Using Soft-Voting and Self-Learning

Many smart city and society applications such as smart health (elderly care, medical applications), smart surveillance, sports, and robotics require the recognition of user activities, an important class of problems known as human activity recognition (HAR). Several issues have hindered progress in HAR research, particularly due to the emergence of fog and edge computing, which brings many new opportunities (a low latency, dynamic and real-time decision making, etc.) but comes with its challenges. This paper focuses on addressing two important research gaps in HAR research: (i) improving the HAR prediction accuracy and (ii) managing the frequent changes in the environment and data related to user activities. To address this, we propose an HAR method based on Soft-Voting and Self-Learning (SVSL). SVSL uses two strategies. First, to enhance accuracy, it combines the capabilities of Deep Learning (DL), Generalized Linear Model (GLM), Random Forest (RF), and AdaBoost classifiers using soft-voting. Second, to classify the most challenging data instances, the SVSL method is equipped with a self-training mechanism that generates training data and retrains itself. We investigate the performance of our proposed SVSL method using two publicly available datasets on six human activities related to lying, sitting, and walking positions. The first dataset consists of 562 features and the second dataset consists of five features. The data are collected using the accelerometer and gyroscope smartphone sensors. The results show that the proposed method provides 6.26%, 1.75%, 1.51%, and 4.40% better prediction accuracy (average over the two datasets) compared to GLM, DL, RF, and AdaBoost, respectively. We also analyze and compare the class-wise performance of the SVSL methods with that of DL, GLM, RF, and AdaBoost.

A Novel Hybrid Gradient-Based Optimizer and Grey Wolf Optimizer Feature Selection Method for Human Activity Recognition Using Smartphone Sensors.

Human activity recognition (HAR) plays a vital role in different real-world applications such as in tracking elderly activities for elderly care services, in assisted living environments, smart home interactions, healthcare monitoring applications, electronic games, and various human–computer interaction (HCI) applications, and is an essential part of the Internet of Healthcare Things (IoHT) services. However, the high dimensionality of the collected data from these applications has the largest influence on the quality of the HAR model. Therefore, in this paper, we propose an efficient HAR system using a lightweight feature selection (FS) method to enhance the HAR classification process. The developed FS method, called GBOGWO, aims to improve the performance of the Gradient-based optimizer (GBO) algorithm by using the operators of the grey wolf optimizer (GWO). First, GBOGWO is used to select the appropriate features; then, the support vector machine (SVM) is used to classify the activities. To assess the performance of GBOGWO, extensive experiments using well-known UCI-HAR and WISDM datasets were conducted. Overall outcomes show that GBOGWO improved the classification accuracy with an average accuracy of 98%.

Recognizing Physical Activities for Spinal Cord Injury Rehabilitation Using Wearable Sensors.

The research area of activity recognition is fast growing with diverse applications. However, advances in this field have not yet been used to monitor the rehabilitation of individuals with spinal cord injury. Noteworthily, relying on patient surveys to assess adherence can undermine the outcomes of rehabilitation. Therefore, this paper presents and implements a systematic activity recognition method to recognize physical activities applied by subjects during rehabilitation for spinal cord injury. In the method, raw sensor data are divided into fragments using a dynamic segmentation technique, providing higher recognition performance compared to the sliding window, which is a commonly used approach. To develop the method and build a predictive model, a machine learning approach was adopted. The proposed method was evaluated on a dataset obtained from a single wrist-worn accelerometer. The results demonstrated the effectiveness of the proposed method in recognizing all of the activities that were examined, and it achieved an overall accuracy of 96.86%.

Area human sensing via ambient Wi‐Fi signals

Indoor human wireless sensing is crucial for many applications such as smart homes and security monitoring. It needs to understand both the number of people and their activities, which is often difficult, especially in large spaces. This paper proposes a combined people number counting and action recognition method by using the channel state information (CSI) of Wi-Fi signal, which aims to simultaneously estimate the number of people and their actions in wireless sensing applications. First, a jump region removal algorithm is developed to calibrate the phase difference. Second, a cumulative sliding variance algorithm is designed for the detection of moving targets in the environment. Then a multi-dimensional feature is formed by the amplitude, frequency domain, and phase difference of CSI, and based on it, a two-level classifier based on the SVM algorithm is used to estimate the number of people and their actions. Experimental results show that the average accuracy of the system for the people counting is 93.5%, and the activity recognition rate of one-person and two-person is 99.6% and 94.6% respectively.

Physique-Based Human Activity Recognition Using Ensemble Learning and Smartphone Sensors

Traditional methods of Human Activity Recognition (HAR) do not take into consideration physical attributes of human subjects such as height, weight, gender etc. Thus, a particular recognition model does not perform consistently for different subjects having diverse physical attributes. In this paper, we propose a novel Physique-based HAR method that takes care of this problem and provides better accuracy in comparatively less time. Raw sensor data are acquired from inbuilt accelerometer and gyroscope sensor modules of smartphones. After pre-processing of the collected data physique-based datasets were prepared, based on the similarities of subjects' physique. Analysis is done on both of these physique-based datasets and normal traditional dataset with the help of various machine learning algorithms. The work not only identifies the suitable learning algorithms for HAR but also shows that the proposed physique-based method outperforms the traditional HAR approach for both publicly available dataset and our own generated dataset, with an accuracy of 99.88%. Individual activity-wise accuracy results are also compared with several recent benchmarks to show the efficiency of the proposed Physique-based HAR method.

Scene image and human skeleton-based dual-stream human action recognition

The dual stream-based human action recognition model offers the advantage of high recognition accuracy, but the algorithm is less robust in case of lighting changes. The human skeleton has a strong ability to express human behavior and actions; however, the scene information is ignored. Drawing on the idea of the dual-stream model, this paper proposes a human skeleton and scene image-based dual-stream model for human action recognition. The motion features are extracted through the spatio-temporal graph convolution of the human skeleton, and a scene recognition model is proposed based on the sparse frame sampling of video and video-level consensus strategy to process the scene video and gather the visual scene information. The proposed model exploits the advantages of skeleton information in motion expression and the superiority of the image in scene presentation. The scene information and spatio-temporal graph convolution-based human skeleton limbs are fused complementarily to achieve human action recognition. Compared to the conventional optical flow-based dual-stream action recognition method, this model is verified by experimenting under unstable light conditions, and the performance of human action recognition is robust and promising.

Mobile-based Activity Monitoring System for the Self-quarantine Patient

Nowadays, not all the patient can be hospitalized because of the COVID-19 pandemics. So, the self-quarantine for the patient with the various diseases will be the given solution by the hospital. It would make the hospital needs a system that can monitor the activity and the position of the patient from a distance. Nowadays, mobile phone is equipped by the sensor that can detect the user movement. Not only the user’s position, but also the user’s activity. In this paper, it will be developed an activity and position monitoring system for the self-quarantine patient that can be used in their home. The mobile activity monitoring can be achieved by activity recognition using classification method. For the needs of performance testing, we evaluate some classification method for activity recognition to compare the among classification method for the activity recognition. Some tested classification methods are Naïve Bayes, KNN, KStar and TreeJ48. Furthermore, we tested the impact of sliding windows per N samples taken to the accuracy of the activity recognition. We choose the best N sample that could give the best accuracy for activity recognition. The system not only monitor the patient’s activity, but also the patient’s position. The position monitoring can be achieved using Google Maps API. The result is Naive bayes has the accuracy of 81.25%, KNN has the accuracy of 78.125%, KStar has the accuracy of 78.125% and TreeJ48 has the accuracy of 75%. The N sample that could give the best accuracy is 6 with the accuracy of 90.15%.

A sequence models-based real-time multi-person action recognition method with monocular vision

In intelligent video surveillance under complex scenes, it is vital to identify the current actions of multi-target human bodies accurately and in real time. In this paper, a real-time multi-person action recognition method with monocular vision is proposed based on sequence models. Firstly, the key points of multi-target human body skeleton in the video are extracted by using the OpenPose algorithm. Then, the human action features are constructed, including limb direction vector and the skeleton height-width ratio. The multi-target human bodies tracking is then achieved by using the tracking algorithm. Next, the tracking results are matched with the action features, and the action recognition model is constructed, which includes the spatial branch based on Deep neural networks and the temporal branch based on Bi-directional RNN and Bi-directional long short-term memory networks. After pre-training, the model can be used to recognize the human body action from action features, and a recognition stabilizer is designed to minimize false alarms. Finally, extensive evaluations on the JHMDB dataset validate the effectiveness and the superiority of the proposed approach.

Effective action recognition with embedded key point shifts

Temporal feature extraction is an essential technique in video-based action recognition. Key points have been utilized in skeleton-based action recognition methods but they require costly key point annotation. In this paper, we propose a novel temporal feature extraction module, named Key Point Shifts Embedding Module (KPSEM), to adaptively extract channel-wise key point shifts across video frames without key point annotation. Key points are adaptively extracted as feature points with maximum feature values at split regions and key point shifts are the spatial displacements of corresponding key points. The key point shifts are encoded as the overall temporal features via linear embedding layers in a multi-set manner. Our method achieves competitive performance through embedding key point shifts with trivial computational cost, achieving the state-of-the-art performance of 78.81% on Mini-Kinetics and competitive performance on UCF101, Something-Something-v1 and HMDB51 datasets.

Wi-Sense: a passive human activity recognition system using Wi-Fi and convolutional neural network and its integration in health information systems

A human activity recognition (HAR) system acts as the backbone of many human-centric applications, such as active assisted living and in-home monitoring for elderly and physically impaired people. Although existing Wi-Fi-based human activity recognition methods report good results, their performance is affected by the changes in the ambient environment. In this work, we present Wi-Sense—a human activity recognition system that uses a convolutional neural network (CNN) to recognize human activities based on the environment-independent fingerprints extracted from the Wi-Fi channel state information (CSI). First, Wi-Sense captures the CSI by using a standard Wi-Fi network interface card. Wi-Sense applies the CSI ratio method to reduce the noise and the impact of the phase offset. In addition, it applies the principal component analysis to remove redundant information. This step not only reduces the data dimension but also removes the environmental impact. Thereafter, we compute the processed data spectrogram which reveals environment-independent time-variant micro-Doppler fingerprints of the performed activity. We use these spectrogram images to train a CNN. We evaluate our approach by using a human activity data set collected from nine volunteers in an indoor environment. Our results show that Wi-Sense can recognize these activities with an overall accuracy of 97.78%. To stress on the applicability of the proposed Wi-Sense system, we provide an overview of the standards involved in the health information systems and systematically describe how Wi-Sense HAR system can be integrated into the eHealth infrastructure.

ASNet: Auto-Augmented Siamese Neural Network for Action Recognition.

Human action recognition methods in videos based on deep convolutional neural networks usually use random cropping or its variants for data augmentation. However, this traditional data augmentation approach may generate many non-informative samples (video patches covering only a small part of the foreground or only the background) that are not related to a specific action. These samples can be regarded as noisy samples with incorrect labels, which reduces the overall action recognition performance. In this paper, we attempt to mitigate the impact of noisy samples by proposing an Auto-augmented Siamese Neural Network (ASNet). In this framework, we propose backpropagating salient patches and randomly cropped samples in the same iteration to perform gradient compensation to alleviate the adverse gradient effects of non-informative samples. Salient patches refer to the samples containing critical information for human action recognition. The generation of salient patches is formulated as a Markov decision process, and a reinforcement learning agent called SPA (Salient Patch Agent) is introduced to extract patches in a weakly supervised manner without extra labels. Extensive experiments were conducted on two well-known datasets UCF-101 and HMDB-51 to verify the effectiveness of the proposed SPA and ASNet.

Physical Activity Recognition Based on a Parallel Approach for an Ensemble of Machine Learning and Deep Learning Classifiers.

Human activity recognition (HAR) by wearable sensor devices embedded in the Internet of things (IOT) can play a significant role in remote health monitoring and emergency notification to provide healthcare of higher standards. The purpose of this study is to investigate a human activity recognition method of accrued decision accuracy and speed of execution to be applicable in healthcare. This method classifies wearable sensor acceleration time series data of human movement using an efficient classifier combination of feature engineering-based and feature learning-based data representation. Leave-one-subject-out cross-validation of the method with data acquired from 44 subjects wearing a single waist-worn accelerometer on a smart textile, and engaged in a variety of 10 activities, yielded an average recognition rate of 90%, performing significantly better than individual classifiers. The method easily accommodates functional and computational parallelization to bring execution time significantly down.

A deep learning method for video‐based action recognition

In this paper, a deep learning method for video-based action recognition is proposed. On the one hand, boundary compensation on the basis of a deep neural network is performed to achieve action proposal. Boundary compensation considering non-maximum suppression according to sliding window priority is applied to remove redundant windows. To accurately detect boundaries, a boundary compensation network is established with multiple networks to process different numbers of segments. On the other hand, action recognition based on the resultant action proposals is performed. To further utilise boundary compensation, three methods are introduced for key frame selection. Optical flow and RGB features are combined via a channel fusion to realise feature representation. A two-stream network with a spatiotemporal structure is adopted for action recognition. The proposed method is evaluated on three public datasets. The experimental results demonstrate that the proposed method achieves a superior performance to that of state-of-the-art methods.