Approach For Human Action Recognition Research Articles

A new approach for physical human activity recognition based on co-occurrence matrices.

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

Smartphone-Based Activity Recognition in a Pedestrian Navigation Context.

In smartphone-based pedestrian navigation systems, detailed knowledge about user activity and device placement is a key information. Landmarks such as staircases or elevators can help the system in determining the user position when located inside buildings, and navigation instructions can be adapted to the current context in order to provide more meaningful assistance. Typically, most human activity recognition (HAR) approaches distinguish between general activities such as walking, standing or sitting. In this work, we investigate more specific activities that are tailored towards the use-case of pedestrian navigation, including different kinds of stationary and locomotion behavior. We first collect a dataset of 28 combinations of device placements and activities, in total consisting of over 6 h of data from three sensors. We then use LSTM-based machine learning (ML) methods to successfully train hierarchical classifiers that can distinguish between these placements and activities. Test results show that the accuracy of device placement classification (97.2%) is on par with a state-of-the-art benchmark in this dataset while being less resource-intensive on mobile devices. Activity recognition performance highly depends on the classification task and ranges from 62.6% to 98.7%, once again performing close to the benchmark. Finally, we demonstrate in a case study how to apply the hierarchical classifiers to experimental and naturalistic datasets in order to analyze activity patterns during the course of a typical navigation session and to investigate the correlation between user activity and device placement, thereby gaining insights into real-world navigation behavior.

Linear dynamical systems approach for human action recognition with dual-stream deep features

Human action recognition with a dual-stream architecture using linear dynamical systems (LDSs) approach is discussed in this paper. First, a slice process is established to extract original slices from video sequences. Two slicing methods are adopted to subtract or reserve the remaining frames in the video sequences. By applying background subtraction to adjacent frames of the original slices, difference slices are also expressed. To capture the spatial component of the background and difference expressed in each slice simultaneously, a framework based on pre-trained convolutional neural networks (CNNs) is introduced for dual-stream deep feature extraction. Subsequently, LDSs are established to model the timing relationship between adjacent slices and obtain the temporal component of the background and difference features, which are expressed as linear dynamical background feature (LD-BF) and linear dynamical difference feature (LD-DF). Practical experiments were conducted to demonstrate the effectiveness and robustness of the proposed approach using different datasets. Specifically, our experiments were conducted on the UCF50, UCF101, and hmdb51 datasets. The impact of retaining various principal component analysis (PCA) feature dimensions and distinct slicing methods in terms of detail recognition were evaluated. In particular, combining LD-BF with LD-DF under appropriate feature dimensions and slicing methods further improved the accuracy for the UCF50, UCF101, and hmdb51 datasets. In addition, the computational cost of the feature extraction process was evaluated to illustrate the efficiency of the proposed approach. The experimental results show that the proposed approach is competitive with state-of-the-art approaches in the three datasets.

Multi-GAT: A Graphical Attention-Based Hierarchical Multimodal Representation Learning Approach for Human Activity Recognition

Recognizing human activities is one of the crucial capabilities that a robot needs to have to be useful around people. Although modern robots are equipped with various types of sensors, human activity recognition (HAR) still remains a challenging problem, particularly in the presence of noisy sensor data. In this work, we introduce a multimodal graphical attention-based HAR approach, called Multi-GAT, which hierarchically learns complementary multimodal features. We develop a multimodal mixture-of-experts model to disentangle and extract salient modality-specific features that enable feature interactions. Additionally, we introduce a novel message-passing based graphical attention approach to capture cross-modal relation for extracting complementary multimodal features. The experimental results on two multimodal human activity datasets suggest that Multi-GAT outperformed state-of-the-art HAR algorithms across all datasets and metrics tested. Finally, the experimental results with noisy sensor data indicate that Multi-GAT consistently outperforms all the evaluated baselines. The robust performance suggests that Multi-GAT can enable seamless human-robot collaboration in noisy human environments.

A Comparative Study of Various Human Activity Recognition Approaches

Human Activity Recognition (HAR) is a vast and exciting topic for researchers and students. HAR aims to recognize activities by observing the actions of subjects and surrounding conditions. This topic also has many significant and futuristic applications and a basis of many automated tasks like 24*7 security surveillance, healthcare, laws regulations, automatic vehicle controls, game controls by human motion detection, basically human-computer interaction. This survey paper focuses on reviewing other research papers on sensing technologies used in HAR. This paper has covered distinct research in which researchers collect data from smartphones; some use a surveillance camera system to get video clips. Most of the researchers used videos to train their systems to recognize human activities collected from YouTubes and other video sources. Several sensor-based approaches have also covered in this survey paper to study and predict human activities, such as accelerometer, gyroscope, and many more. Some of the papers also used technologies like a Convolutional neural network (CNN) with spatiotemporal three-dimensional (3D) kernels for model development and then using to integrate it with OpenCV. There are also work done for Alzheimer’s patient in the Healthcare sector, used for their better performance in day-to-day tasks. We will analyze the research using both classic and less commonly known classifiers on distinct datasets available on the UCI Machine Learning Repository. We describe each researcher’s approaches, compare the technologies used, and conclude the adequate technology for Human Activity Recognition. Every research will be discussed in detail in this survey paper to get a brief knowledge of activity recognition.

A Comparative Study of Feature Selection Approaches for Human Activity Recognition Using Multimodal Sensory Data.

Human activity recognition (HAR) aims to recognize the actions of the human body through a series of observations and environmental conditions. The analysis of human activities has drawn the attention of the research community in the last two decades due to its widespread applications, diverse nature of activities, and recording infrastructure. Lately, one of the most challenging applications in this framework is to recognize the human body actions using unobtrusive wearable motion sensors. Since the human activities of daily life (e.g., cooking, eating) comprises several repetitive and circumstantial short sequences of actions (e.g., moving arm), it is quite difficult to directly use the sensory data for recognition because the multiple sequences of the same activity data may have large diversity. However, a similarity can be observed in the temporal occurrence of the atomic actions. Therefore, this paper presents a two-level hierarchical method to recognize human activities using a set of wearable sensors. In the first step, the atomic activities are detected from the original sensory data, and their recognition scores are obtained. Secondly, the composite activities are recognized using the scores of atomic actions. We propose two different methods of feature extraction from atomic scores to recognize the composite activities, and they include handcrafted features and the features obtained using the subspace pooling technique. The proposed method is evaluated on the large publicly available CogAge dataset, which contains the instances of both atomic and composite activities. The data is recorded using three unobtrusive wearable devices: smartphone, smartwatch, and smart glasses. We also investigated the performance evaluation of different classification algorithms to recognize the composite activities. The proposed method achieved 79% and 62.8% average recognition accuracies using the handcrafted features and the features obtained using subspace pooling technique, respectively. The recognition results of the proposed technique and their comparison with the existing state-of-the-art techniques confirm its effectiveness.

Human action recognition approaches with video datasets—A survey

Human Activity Recognition has recently attracted considerable attention. This has been triggered by the rapid development of advance technologies and learning methods. Human action recognition can be actively used in a number of application domains which may positively influence various aspects of the daily life. These include, (1) preventing dangerous activities and detection of crimes such as theft, murder, and property damage, and (2) predicting pedestrian activities in traffic, among others. To better serve these applications and the like, it is essential to highlight the various aspects related to the existing methods so that their actual users could realize and identify the good performing methods that work fast and are capable of recognizing the correct activities with high accuracy. The latter scope is covered in this survey which summarizes and analyzes the methods that perform learning and analysis processes on video datasets to grasp a new perspective on human action recognition. The survey also covers the major datasets commonly used in human activity recognition research. Accordingly, this survey could be recognized as a valuable source for researchers and practitioners.

Hybrid deep learning approaches for smartphone sensor-based human activity recognition

Human Activity Recognition (HAR) has become one of the most important research fields to achieve real-time monitoring of human activities for timely decision making in various applications like fall detection, elderly care etc. Now-a-days, most people use smartphones which come with various embedded inertial sensors like accelerometer and gyroscope to monitor acceleration and angular velocity. These smartphone-based sensors have proven to be cost-effective solution in identification of activities belonging to ADL (Activities of Daily Living). Various Machine Learning, Deep learning and hybrid models have been proposed and implemented for HAR. This paper also proposes various hybrid deep learning approaches which combine Deep Neural Networks with other models like LSTM (Long Short Term Memory) Model and GRU (Gated Recurrent Unit) for effective classification of engineered features from CNN (Convolutional Neural Network) Model. A novel architecture that integrates CNN with Random Forest Classifier (DeepCNN-RF) is proposed to add randomness to the model. The proposed models have been tested on publicly available HAR Datasets like UCI HAR and WISDM Activity Recognition Datasets. Experimental results show that the hybrid models outperform the state-of-the-art data mining, machine learning techniques in UCI HAR and WISDM with an overall maximum accuracy of 97.77% and 98.2% respectively.

A Novel Multi-Stage Training Approach for Human Activity Recognition From Multimodal Wearable Sensor Data Using Deep Neural Network

Deep neural network is an effective choice to automatically recognize human actions utilizing data from various wearable sensors. These networks automate the process of feature extraction relying completely on data. However, various noises in time series data with complex inter-modal relationships among sensors make this process more complicated. In this article, we have proposed a novel multi-stage training approach that increases diversity in this feature extraction process to make accurate recognition of actions by combining varieties of features extracted from diverse perspectives. Initially, instead of using single type of transformation, numerous transformations are employed on time series data to obtain variegated representations of the features encoded in raw data. An efficient deep CNN architecture is proposed that can be individually trained to extract features from different transformed spaces. Later, these CNN feature extractors are merged into an optimal architecture finely tuned for optimizing diversified extracted features through a combined training stage or multiple sequential training stages. This approach offers the opportunity to explore the encoded features in raw sensor data utilizing multifarious observation windows with immense scope for efficient selection of features for final convergence. Extensive experimentations have been carried out in three publicly available datasets that provide outstanding performance consistently with average five-fold cross-validation accuracy of 99.29% on UCI HAR database, 99.02% on USC HAR database, and 97.21% on SKODA database outperforming other state-of-the-art approaches.

Human Action Recognition Using A Novel Deep Learning Approach

Behavior human analysis is always a significant aspect in societal communication. The human behavior analysis is developed based on few factors like human activity and action recognition. Human action recognition is an significant feature in different safety fields. The assessment of the action recognition algorithm depends on the appropriate extraction and the learning data. In the human action recognition, classification plays the major role so in order to this effectively Gated Recurrent Neural Network is used with an increased computation level. Feature extraction is one of the essential factor in human action recognition it will influence the performance and computation time of the algorithm. This paper presented an approach for human action recognition based on new mixture deep learning model. The proposed method is evaluated on the different data sets like UCF Sports, KTH and UCF101. On UCF Sports data set the proposed method has given an average of 96.8%.

Attention-based network for effective action recognition from multi-view video

A human action recognition system is affected by many challenges such as background clutter, partial occlusion, lighting, viewpoint, execution rate. Using complementary information from different views can improve view changing and occlusion problems. However, how to effectively integrate the information from multi-view images? In this paper, we propose an effective approach for multi-view human action recognition. The proposition is based on attention mechanism to pass discriminate feature between views. It is designed to form a multi-branch network whose each branch takes responsibility for extracting a view-specific feature. Furthermore, we built a cross-view attention module to enhance action recognition by transferring knowledge between views (branches). Experiments on three datasets show that the proposed solution works effectively in different scenarios. Our models have achieved the best results on two datasets (NUMA and MicaHandGesture) for both cross-subject and cross-view evaluations. On the NUMA dataset, the accuracy of our best models reach to 99.56% and 92.74% in cross-subject and cross-view evaluation scenarios respectively. And on the MicaHandGesture dataset, the accuracy are 99.06%, 91.71% in two scenarios respectively. The obtained results surpass other previous works such as Multi-Branch TSN with GRU [5] (93.81% in cross-subject evaluation, 84.4% in cross-view evaluation on the NUMA) and DA-Net [31] (92.1% for cross-subject evaluation (video-level), and 84.2% for cross-view evaluation on the NUMA dataset). We also obtained very promising results on a large-scale NTU RGB+D dataset.

Segment spatial-temporal representation and cooperative learning of convolution neural networks for multimodal-based action recognition

The multimodal based human action recognition is an attracting increasing topic since the different modalities can provide complementary information. However, it is difficult to improve the recognition performance due to the limitation of the ability to learn spatial-temporal features. In this paper, we propose a novel approach for multimodal human action recognition by learning complementary features from RGB-D sequence. Firstly, a segmented rank pooling method is proposed to compress the entire RGB-D sequence into dynamic images as inputs to the Convolutional Networks (ConvNets) for capturing spatial-temporal information. Secondly, a Segment Cooperative ConvNets (SC-ConvNets) is designed to learn the complementary features from RGB-D modalities. Different from the ConvNets-based approaches that learn multimodal features with multiple separate networks, the proposed SC-ConvNets enhance the recognition performance through joint optimization learning of single ConvNets. Then a single loss function is optimized to narrow the variance between RGB and depth modalities. In order to verify the effectiveness of the proposed method, we evaluate the SC-ConvNets on four public benchmark multimodal datasets, including NTU RGB+D 60, NTU RGB+D 120, SYSU 3D HOI, and PKU-MMD datasets. The proposed method achieves an accuracy of 89.4% and 91.2% for cross-subject and cross-view on NTU RGB+D 60, 86.9% and 87.7% for cross-subject and cross-setup on NTU RGB+D 120, 92.1% and 93.2% for cross-subject and cross-view on PKU-MMD, which are the state-of-the-art, and the accuracy of 84.2% and 82.9% for setting-1 and setting-2 on SYSU 3D HOI, which are comparable to the state-of-the-art. The experimental results demonstrate that the proposed segmented rank pooling can represent discriminative spatial-temporal information from the entire RGB and depth sequence, and the proposed SC-ConvNets can enhance recognition performance by learning complementary features from different modalities.

Multi-frequency and multi-domain human activity recognition based on SFCW radar using deep learning

Deep learning and radar make it feasible to automatically recognize human activities in various lighting conditions, even occlusion case, which significantly promotes the application of activity recognition in the fields of security surveillance, health care, and so on. In this paper, an approach for human activity recognition (HAR) using deep learning is proposed based on stepped frequency continues wave (SFCW) radar. Specifically, SFCW radar is utilized to generate two types of characteristic representation domains, namely multiple frequencies of spectrograms in time–frequency domain and range maps in range domain. On the one hand, spectrograms and range maps provide different types of features. On the other hand, multi-frequency spectrograms furnish same type of features while with different scattering properties and frequency resolutions. Then a specific deep learning network including multiple parallel deep convolutional neural networks (DCNNs) and a sparse autoencoder is designed to extract and fuse these features associated with human activities from the multi-frequency spectrograms and rang map. In particular, each DCNN is aimed at extracting the detailed micro-Doppler features from a spectrogram, while sparse autoencoder learns prime range distribution features by compressing each range map to reduce complexity and improve robustness. Experimental results verify that the proposed deep learning scheme achieves 96.42% recognition accuracy about six types of activities by incorporating three frequencies of spectrograms and range map, and surpasses two existed methods depending on single-frequency spectrogram and combination of single-frequency spectrogram and range map.

Rate-Invariant Modeling in Lie Algebra for Activity Recognition

Human activity recognition is one of the most challenging and active areas of research in the computer vision domain. However, designing automatic systems that are robust to significant variability due to object combinations and the high complexity of human motions are more challenging. In this paper, we propose to model the inter-frame rigid evolution of skeleton parts as the trajectory in the Lie group SE(3)×…×SE(3). The motion of the object is similarly modeled as an additional trajectory in the same manifold. The classification is performed based on a rate-invariant comparison of the resulting trajectories mapped to a vector space, the Lie algebra. Experimental results on three action and activity datasets show that the proposed method outperforms various state-of-the-art human activity recognition approaches.

Hybrid domain adaptation with deep network architecture for end-to-end cross-domain human activity recognition

Machine learning-based human activity recognition (HAR) as the means of human–computer interaction is important to empower the existing systems in the industry such as manufacturing and logistics to be more autonomous. However, it is often difficult to build HAR models due to the limitation of annotated samples. Domain adaptation has emerged to address such limitation (or absence) of labeled samples in the domain of interest (i.e., target domain) by using abundant amount of labeled samples in the other domain (i.e., source domain). Domain adaptation aims to solve learning problems where a source domain and a target domain are different but still related. With the use of homogeneous feature space, the existing approaches on homogeneous domain adaption is prohibitive for the real-life scenario, in which two different domains could be possibly of heterogeneous feature space. Although heterogeneous domain adaptation approaches exist, it still requires additional information (i.e., instance correspondence) which is difficult to satisfy when we deal with sensor data. Hybrid domain adaptation, a special case of heterogeneous domain adaptation where common feature between domain exists, is more realistic as the feature commonality is easier to satisfy. However, the existing approach operates using common features in the original feature space, which in fact, may still have distribution difference. In addition, the existing one requires hand-crafted feature extractions for more informative descriptors to classify human activities. In this work, we propose a deep architecture for hybrid domain adaptation, to enable end-to-end learning for human activity classification. The architecture is tested on sensor-based human activity recognition dataset. The experimental result shows that our approach yields better result compared to the existing human activity recognition approaches, both the deep and shallow approaches.

Kinematic joint descriptor and depth motion descriptor with convolutional neural networks for human action recognition

Human Action Recognition has gained a huge research interest due to its widespread applications in various fields. However, due to several challenges like noisy and occluded data, view-point variations, body sizes etc., still the action recognition remains a challenging task. Most of the existing action recognition methods focused on the single data type thereby the recognition system has limited performance. To improve the recognition performance, we have modeled a new approach for human action recognition from two different data types; they are depth images and skeleton joints. Two different descriptors are developed for action representation; they are Differential Depth Motion History Image for depth maps and Motion Kinematic Joint Descriptor for skeleton joints. To attain a discriminative feature set, we have trained three different Convolutional Neural Network Models and the results are fused for final action classification. Simulation is carried out over two public datasets and the obtained results indicate that the proposed approach outperforms state-of-art methods.

Wrapper Filter Approach for Accelerometer-Based Human Activity Recognition

With the widespread use of mobile devices all over the world, a new interesting and challenging research area known as Activity Recognition (AR) with many application domains is evolved. Basically, activity recognition aims to identify certain physical activities such as walking, jogging, sitting, standing, etc., performed daily by humans. In this paper, we investigated the effectiveness of wrapper-based feature selection approach for accelerometer-based human activity recognition. Our approach utilizes Sequential Forward Selection (SFS) technique based on three machine learning algorithms: Random Forest (RF), K-Nearest Neighbor (K-NN), and Gradient-Boosted Tree (GBT). A standard and publicly available dataset called WISDM (Wireless Sensor Data Mining), which contains accelerometer-based time series data collected from thirty-six volunteers, was used for performance evaluation of the proposed model. The experimental results showed that our GBT-based recognition model outperforms previously suggested solutions and establishing state-of-the-art performance for this dataset.

Novel tent pooling based human activity recognition approach

Nowadays, mobile devices have been widely used worldwide and they have variable sensors. By using these sensors, many applications for instance navigation, healthcare, and human activity recognition (HAR) have been developed. The primary objective of this paper is to present a multi lightweight HAR method with high prediction performance. To implement this objective, novel pooling methodology is presented and it is called as tent pooling since pooling is processed using the variable size of blocks. 2 × 2, 3 × 3, 4 × 4, 5 × 5, 6 × 6, 7 × 7 and 8 × 8 sized non-overlapping blocks are used for pooling and maximum, minimum and average pooling methods. These pooling methods have been utilized as a feature generation model. The generated features are forwarded to the ReliefF feature selection method. ReliefF is a weight-based feature selector and uses Manhattan distance based fitness function. By using relief, the weights of the features are calculated and the negative weighted features are eliminated. This operation is named positive weighted feature selection. The selected positive weighted features are classified using a cubic support vector machine (SVM). To test the proposed method, a publicly and freely published HAR dataset is used. There are 6 activities in the used dataset. Also, the results of the proposed method were compared to other state of art methods. The best accuracy rate of the proposed tent pooling based HAR method was calculated as 99.81% using the average pooling. This results clearly prove that the success of the proposed tent pooling based method. The proposed method is also simple and effective. It has the highest success rates among the selected state-art-of HAR methods.

A Machine Learning Approach for Human Activity Recognition.

Human Activity Recognition (HAR) is becoming a significant issue in modern times and directly impact the field of mobile health. Therefore, it is essential the designing of systems which are capable of recognizing properly the activities conducted by the individuals. In this work, we developed a system using the Internet of Things (IoT) and machine learning technologies in order to monitor and assist individuals in their daily life. We compared the data collected using a mobile application and a wearable device with built-in sensors (accelerometer and gyroscope) with the data of a publicly available dataset. By this way, we were able to validate our results and also investigate the functionality and applicability of the wearable device that we choose for the Human Activity Recognition problem. The classification results for the different types of activities presented using our dataset (99%) outperforms the results from the publicly database (97%).

A combined multiple action recognition and summarization for surveillance video sequences

Human action recognition and video summarization represent challenging tasks for several computer vision applications including video surveillance, criminal investigations, and sports applications. For long videos, it is difficult to search within a video for a specific action and/or person. Usually, human action recognition approaches presented in the literature deal with videos that contain only a single person, and they are able to recognize his action. This paper proposes an effective approach to multiple human action detection, recognition, and summarization. The multiple action detection extracts human bodies’ silhouette, then generates a specific sequence for each one of them using motion detection and tracking method. Each of the extracted sequences is then divided into shots that represent homogeneous actions in the sequence using the similarity between each pair frames. Using the histogram of the oriented gradient (HOG) of the Temporal Difference Map (TDMap) of the frames of each shot, we recognize the action by performing a comparison between the generated HOG and the existed HOGs in the training phase which represents all the HOGs of many actions using a set of videos for training. Also, using the TDMap images we recognize the action using a proposed CNN model. Action summarization is performed for each detected person. The efficiency of the proposed approach is shown through the obtained results for mainly multi-action detection and recognition.