Vision-based Human Action Recognition Research Articles

MAWKDN: A Multimodal Fusion Wavelet Knowledge Distillation Approach Based on Cross-View Attention for Action Recognition

The recognition prerformance of existing vision-based human action recognition (HAR) methods is greatly reduced in the case of low camera resolution or occlusion. Wearable sensors can provide complementary information to alleviate this problem. It is challenging to construct a robust HAR model using multimodal wearable-sensor data. In this paper, we propose a cross-Attention-based Multimodal fusion Wavelet Knowledge Distillation Network (MAWKDN) method to guide recognition from video data by acquiring complementary information from wearable sensors and reduce the noise effects through wavelet knowledge distillation, which improves the robustness of the model. A multi-attention dilated convolution kernel residual network including dilated convolution and an attention mechanism is constructed to extract features from various sensor modalities and fuse the various modal data through the cross-view attention method to acquire additional information from different modalities. To reduce the modal differences between different modalities of the teacher and student networks and acquire similar semantic knowledge, we learn the information between different modalities by constructing a graph structure of convolutional layer features, and computing the semantic preservation loss between the teacher and student networks. To reduce the influence of noise in the input data, we construct the loss of wavelet knowledge distillation, which transforms the image through the discrete wavelet transform and only retains the low frequency features to extract the useful information. The top-1 accuracy achieved on the UTD-MHAD (99.31%), Berkeley-MHAD (99.40%) and the F1-score on the MMAct (85.26% based on cross-session) dataset prove the superior performance of MAWKDN compared with the state-of-the-art HAR methods. Moreover, we demonstrate the robustness of the MAWKDN approach on the noise-added UTD-MHAD dataset.

An Overview of the Vision-Based Human Action Recognition Field

Artificial intelligence’s rapid advancement has enabled various applications, including intelligent video surveillance systems, assisted living, and human–computer interaction. These applications often require one core task: video-based human action recognition. Research in human video-based human action recognition is vast and ongoing, making it difficult to assess the full scope of available methods and current trends. This survey concisely explores the vision-based human action recognition field and defines core concepts, including definitions and explanations of the common challenges and most used datasets. Additionally, we provide in an easy-to-understand manner the literature approaches and their evolution over time, emphasizing intuitive notions. Finally, we explore current research directions and potential future paths. The core goal of this work is to provide future works with a shared understanding of fundamental ideas and clear intuitions about current works and find new research opportunities.

Individual Surveillance Around Parked Aircraft at Nighttime: Thermal Infrared Vision-Based Human Action Recognition

The COVID-19 crisis has led to an unusually large number of commercial aircraft being currently parked or stored. For airlines, airports, and civil aviation authorities around the world, monitoring, and protecting these parked aircraft to prevent them from causing human-made damage are becoming urgent problems that are receiving increasing attention. In this study, we use thermal infrared monitoring videos to establish a framework for individual surveillance around parked aircraft by proposing a human action recognition (HAR) algorithm. As the focus of this article, the proposed HAR algorithm seamlessly integrates a preprocessing module in which a novel data structure is constructed to introduce spatiotemporal information of the action; a convolutional neural network-based module for spatial feature extraction; a triple-layer convolutional long short-term memory network for temporal feature extraction; and two fully connected layers for classification. Moreover, because no infrared dataset is available for the HAR task on airport grounds at nighttime, we present a dataset called IIAR-30, which consists of eight action categories that frequently occur on airport grounds and 2000 video clips. The experimental results on the IIAR-30 dataset demonstrated that the recognition accuracy of the proposed method was higher than 96%. We also further evaluated the effectiveness of the proposed method by comparing it with five baselines and four other methods.

Research on the Recognition Algorithm of Basketball Technical Action Based on BP Neural System

Vision-based intelligent human action recognition is the most challenging direction in the field of computer vision in recent years. It detects human actions in video sequences, extracts action features and learns action features, and then recognizes human actions in videos. This paper is based on BP neural network’s basketball technique action recognition and experimental verification. First, design a basketball technique action recognition method based on BP neural network, analyze basketball actions, collect relevant test data, and divide the methods of basketball action recognition. Finally, analyze the action characteristics and waveform conditions of the upper- and lower-limb movements of the basketball action and analyze the key basketball action recognition data. The designed classification method realizes the effective recognition of basketball actions; then, the basketball recognition method used in this article is experimentally verified, and the feasibility and effectiveness of the recognition method selected in this article are verified by recognizing basketball technical actions, and the experimental results are carried out. Compared with other related studies, this method proposes a division of unit actions to complete the cycle division of basketball actions. The division results do not include the overlap of other actions, avoiding repeated calculations of actions and greatly reducing the amount of calculation of the system. In addition, the method for the recognition of basketball movement includes the separate recognition of upper- and lower-limb movements, comprehensive consideration of arm and leg movements, and a more comprehensive and accurate analysis of basketball movements.

Complex Network-based features extraction in RGB-D human action recognition

Analysis of human behavior through visual information has been one of the active research areas in computer vision community during the last decade. Vision-based human action recognition (HAR) is a crucial part of human behavior analysis, which is also of great demand in a wide range of applications. HAR was initially performed via images from a conventional camera; however, depth sensors have recently embedded as an additional informative resource to cameras. In this paper, we have proposed a novel approach to largely improve the performance of human action recognition using Complex Network-based feature extraction from RGB-D information. Accordingly, the constructed complex network is employed for single-person action recognition from skeletal data consisting of 3D positions of body joints. The indirect features help the model cope with the majority of challenges in action recognition. In this paper, the meta-path concept in the complex network has been presented to lessen the unusual actions structure challenges. Further, it boosts recognition performance. The extensive experimental results on two widely adopted benchmark datasets, the MSR-Action Pairs, and MSR Daily Activity3D indicate the efficiency and validity of the method.

A resource conscious human action recognition framework using 26-layered deep convolutional neural network

Vision-based human action recognition (HAR) is a hot topic of research from the decade due to a few popular applications such as visual surveillance and robotics. For correct action recognition, various local and global points are requires known as features. These features modified during the variation in human movement. But due to a bit change in several human actions, the features of these actions are mixed that degrade the recognition performance. In this article, we design a new 26-layered Convolutional Neural Network (CNN) architecture for accurate complex action recognition. The features are extracted from the global average pooling layer and fully connected (FC) layer, and fused by a proposed high entropy-based approach. Further, we propose a feature selection method name Poisson distribution along with Univariate Measures (PDaUM). Few of fused CNN features are irrelevant, and few of them are redundant that makes the incorrect prediction among complex human actions. Therefore, the proposed PDaUM based approach selects only the strongest features that later passed to the Extreme Learning Machine (ELM) and Softmax for final recognition. Four datasets are using for experimental analysis - HMDB51 (51 classes), UCF Sports (10 classes), KTH (6 classes), and Weizmann (10 classes). On these datasets, the ELM classifier gives an improved performance as compared to a Softmax classifier. The achieved accuracy on each dataset is 81.4%, 99.2%, 98.3%, and 98.7%, respectively. Comparison with existing techniques, it is shown that the proposed architecture gives better performance in terms of accuracy and testing time.

A Comprehensive Survey of Vision-Based Human Action Recognition Methods.

Although widely used in many applications, accurate and efficient human action recognition remains a challenging area of research in the field of computer vision. Most recent surveys have focused on narrow problems such as human action recognition methods using depth data, 3D-skeleton data, still image data, spatiotemporal interest point-based methods, and human walking motion recognition. However, there has been no systematic survey of human action recognition. To this end, we present a thorough review of human action recognition methods and provide a comprehensive overview of recent approaches in human action recognition research, including progress in hand-designed action features in RGB and depth data, current deep learning-based action feature representation methods, advances in human–object interaction recognition methods, and the current prominent research topic of action detection methods. Finally, we present several analysis recommendations for researchers. This survey paper provides an essential reference for those interested in further research on human action recognition.

Volumetric spatial feature representation for view-invariant human action recognition using a depth camera

The problem of viewpoint variations is a challenging issue in vision-based human action recognition. With the richer information provided by three-dimensional (3-D) point clouds thanks to the advent of 3-D depth cameras, we can effectively analyze spatial variations in human actions. In this paper, we propose a volumetric spatial feature representation (VSFR) that measures the density of 3-D point clouds for view-invariant human action recognition from depth sequence images. Using VSFR, we construct a self-similarity matrix (SSM) that can graphically represent temporal variations in the depth sequence. To obtain an SSM, we compute the squared Euclidean distance of VSFRs between a pair of frames in a video sequence. In this manner, an SSM represents the dissimilarity between a pair of frames in terms of spatial information in a video sequence captured at an arbitrary viewpoint. Furthermore, due to the use of a bag-of-features method for feature repre- sentations, the proposed method efficiently handles the variations of action speed or length. Hence, our method is robust to both variations in viewpoints and lengths of action sequences. We evaluated the proposed method by comparing with state-of-the-art methods in the literature on three public datasets of ACT4 2 , MSRAction3D, and MSRDailyActivity3D, validating the superiority of our method by achieving the highest accuracies. © 2015 Society of

A Low-Dimensional Radial Silhouette-Based Feature for Fast Human Action Recognition Fusing Multiple Views.

This paper presents a novel silhouette-based feature for vision-based human action recognition, which relies on the contour of the silhouette and a radial scheme. Its low-dimensionality and ease of extraction result in an outstanding proficiency for real-time scenarios. This feature is used in a learning algorithm that by means of model fusion of multiple camera streams builds a bag of key poses, which serves as a dictionary of known poses and allows converting the training sequences into sequences of key poses. These are used in order to perform action recognition by means of a sequence matching algorithm. Experimentation on three different datasets returns high and stable recognition rates. To the best of our knowledge, this paper presents the highest results so far on the MuHAVi-MAS dataset. Real-time suitability is given, since the method easily performs above video frequency. Therefore, the related requirements that applications as ambient-assisted living services impose are successfully fulfilled.

Human Action Recognition With Video Data: Research and Evaluation Challenges

Given a video sequence, the task of action recognition is to identify the most similar action among the action sequences learned by the system. Such human action recognition is based on evidence gathered from videos. It has wide application including surveillance, video indexing, biometrics, telehealth, and human-computer interaction. Vision-based human action recognition is affected by several challenges due to view changes, occlusion, variation in execution rate, anthropometry, camera motion, and background clutter. In this survey, we provide an overview of the existing methods based on their ability to handle these challenges as well as how these methods can be generalized and their ability to detect abnormal actions. Such systematic classification will help researchers to identify the suitable methods available to address each of the challenges faced and their limitations. In addition, we also identify the publicly available datasets and the challenges posed by them. From this survey, we draw conclusions regarding how well a challenge has been solved, and we identify potential research areas that require further work.

Realistic human action recognition by Fast HOG3D and self-organization feature map

Nowadays, local features are very popular in vision-based human action recognition, especially in "wild" or unconstrained videos. This paper proposes a novel framework that combines Fast HOG3D and self-organization feature map (SOM) network for action recognition from unconstrained videos, bypassing the demanding preprocessing such as human detection, tracking or contour extraction. The contributions of our work not only lie in creating a more compact and computational effective local feature descriptor than original HOG3D, but also lie in first successfully applying SOM to realistic action recognition task and studying its training parameters' influence. We mainly test our approach on the UCF-YouTube dataset with 11 realistic sport actions, achieving promising results that outperform local feature-based support vector machine and are comparable with bag-of-words. Experiments are also carried out on KTH and UT-Interaction datasets for comparison. Results on all the three datasets confirm that our work has comparable, if not better, performance comparing with state-of-the-art.

Adaptive Human Action Recognition With an Evolving Bag of Key Poses

Vision-based human action recognition allows to detect and understand meaningful human motion. This makes it possible to perform advanced human-computer interaction, among other applications. In dynamic environments, adaptive methods are required to support changing scenario characteristics. Specifically, in human-robot interaction, smooth interaction between humans and robots can only be performed if these are able to evolve and adapt to the changing nature of the scenarios. In this paper, an adaptive vision-based human action recognition method is proposed. By means of an evolutionary optimization method, adaptive and incremental learning of human actions is supported. Through an evolving bag of key poses, which models the learned actions over time, the current learning memory is developed to recognize increasingly more actions or actors. The evolutionary method selects the optimal subset of training instances, features and parameter values for each learning phase, and handles the evolution of the model. The experimentation shows that our proposal achieves to adapt to new actions or actors successfully, by rearranging the learned model. Stable and accurate results have been obtained on four publicly available RGB and RGB-D datasets, unveiling the method's robustness and applicability.

Optimizing human action recognition based on a cooperative coevolutionary algorithm

Vision-based human action recognition is an essential part of human behavior analysis, which is currently in great demand due to its wide area of possible applications. In this paper, an optimization of a human action recognition method based on a cooperative coevolutionary algorithm is proposed. By means of coevolution, three different populations are evolved to obtain the best performing individuals with respect to instance, feature and parameter selection. The fitness function is based on the result of the human action recognition method. Using a multi-view silhouette-based pose representation and a weighted feature fusion scheme, an efficient feature is obtained, which takes into account the multiple views and their relevance. Classification is performed by means of a bag of key poses, which represents the most characteristic pose representations, and matching of sequences of key poses. The performed experimentation indicates that not only a considerable performance gain is obtained outperforming the success rates of other state-of-the-art methods, but also the temporal and spatial performance of the algorithm is improved.

A survey of video datasets for human action and activity recognition

Vision-based human action and activity recognition has an increasing importance among the computer vision community with applications to visual surveillance, video retrieval and human–computer interaction. In recent years, more and more datasets dedicated to human action and activity recognition have been created. The use of these datasets allows us to compare different recognition systems with the same input data. The survey introduced in this paper tries to cover the lack of a complete description of the most important public datasets for video-based human activity and action recognition and to guide researchers in the election of the most suitable dataset for benchmarking their algorithms.

VISUAL-BASED HUMAN ACTION RECOGNITION ON SMART PHONES BASED ON 2D AND 3D DESCRIPTORS

The aim of this work is to present a visual-based human action recognition system which is adapted to constrained embedded devices, such as smart phones. Basically, vision-based human action recognition is a combination of feature-tracking, descriptor-extraction and subsequent classification of image representations, with a color-based identification tool to distinguish between multiple human subjects. Simple descriptors sets were evaluated to optimize recognition rate and performance and two dimensional (2D) descriptors were found to be effective. These sets installed on the latest phones can recognize human actions in videos in less than one second with a success rate of over 82%.

A survey on vision-based human action recognition

Vision-based human action recognition is the process of labeling image sequences with action labels. Robust solutions to this problem have applications in domains such as visual surveillance, video retrieval and human–computer interaction. The task is challenging due to variations in motion performance, recording settings and inter-personal differences. In this survey, we explicitly address these challenges. We provide a detailed overview of current advances in the field. Image representations and the subsequent classification process are discussed separately to focus on the novelties of recent research. Moreover, we discuss limitations of the state of the art and outline promising directions of research.

View-independent human action recognition with Volume Motion Template on single stereo camera

Vision-based human action recognition provides an advanced interface, and research in the field of human action recognition has been actively carried out. However, an environment from dynamic viewpoint, where we can be in any position, any direction, etc., must be considered in our living 3D space. In order to overcome the viewpoint dependency, we propose a Volume Motion Template (VMT) and Projected Motion Template (PMT). The proposed VMT method is an extension of the Motion History Image (MHI) method to 3D space. The PMT is generated by projecting the VMT into a 2D plane that is orthogonal to an optimal virtual viewpoint where the optimal virtual viewpoint is a viewpoint from which an action can be described in greatest detail, in 2D space. From the proposed method, any actions taken from different viewpoints can be recognized independent of the viewpoints. The experimental results demonstrate the accuracies and effectiveness of the proposed VMT method for view-independent human action recognition.