Video-based Human Action Recognition Research Articles

A survey of video-based human action recognition in team sports

Over the past few decades, numerous studies have focused on identifying and recognizing human actions using machine learning and computer vision techniques. Video-based human action recognition (HAR) aims to detect actions from video sequences automatically. This can cover simple gestures to complex actions involving multiple people interacting with objects. Actions in team sports exhibit a different nature compared to other sports, since they tend to occur at a faster pace and involve more human-human interactions. As a result, research has typically not focused on the challenges of HAR in team sports. This paper comprehensively summarises HAR-related research and applications with specific focus on team sports such as football (soccer), basketball and Australian rules football. Key datasets used for HAR-related team sports research are explored. Finally, common challenges and future work are discussed, and possible research directions identified.

Interpretability in Video-based Human Action Recognition: Saliency Maps and GradCAM in 3D Convolutional Neural Networks

Interpretability plays a vital role in understanding complex deep learning models by providing transparency and insights. It addresses the black-box nature of these models, aids in human-in-the-loop systems, enhances model development, and supports education. However, existing interpretability algorithms in computer vision primarily target images, leaving a gap for video applications. In this article, we emphasize the importance of interpretability in video-based Human Action Recognition (HAR). We extend existing 2D interpretability techniques to the 3D domain, specifically focusing on saliency maps and gradient-weighted class activation maps. The proposed interpretability system is then employed in the analysis of a well-known HAR dataset to better understand action recognition in videos.

A review of video-based human activity recognition: theory, methods and applications

A review of video-based human activity recognition: theory, methods and applications

Knowledge Distillation in Video-Based Human Action Recognition: An Intuitive Approach to Efficient and Flexible Model Training.

Training a model to recognize human actions in videos is computationally intensive. While modern strategies employ transfer learning methods to make the process more efficient, they still face challenges regarding flexibility and efficiency. Existing solutions are limited in functionality and rely heavily on pretrained architectures, which can restrict their applicability to diverse scenarios. Our work explores knowledge distillation (KD) for enhancing the training of self-supervised video models in three aspects: improving classification accuracy, accelerating model convergence, and increasing model flexibility under regular and limited-data scenarios. We tested our method on the UCF101 dataset using differently balanced proportions: 100%, 50%, 25%, and 2%. We found that using knowledge distillation to guide the model's training outperforms traditional training without affecting the classification accuracy and while reducing the convergence rate of model training in standard settings and a data-scarce environment. Additionally, knowledge distillation enables cross-architecture flexibility, allowing model customization for various applications: from resource-limited to high-performance scenarios.

Human Activity Detection Events Through Human Eye Reflection using Bystander Analyzer

ABSTRACT Automatic video-based human activity recognition has shown promise and is mostly used in video surveillance applications for diverse purposes. However, there are still substantial performance issues that make real-world implementation challenging. The main barrier to the accurate detection of human movement remains to the perspective problem, which results from the fact that video sequences are frequently shot from random camera angles. Therefore, the main focus of this study is on how human eyesight reflects in different camera views that are used to detect human activity or identify intruders. First, a gaussian mixture model that previously used the Minimal Spanning Tree for segmentation is used for pre-processing. Using the pixel-based Kruskal methodology and this method, the input data set’s minimal weight is precisely determined. Segmentation comes next. Independent discriminant features are extracted using transformation using Karhunen-Loeve expansion, which isolates human behavior based on the Person Correlation Coefficient. To effectively identify data, Deep Lens Classifier is also employed to look for any suspect human behavior. With an impressive 78.8774% accuracy, 28.6961% sensitivity, 98.50% specificity, 75.6734% precision, 48.781% recall, and 65.10% F-measure, this approach is unique in the field of detecting human activity. Finally, our proposed system’s MATLAB performance retrieves the accurate detection.

An Interpretable Deep Learning-based Feature Reduction in Video-Based Human Activity Recognition

An Interpretable Deep Learning-based Feature Reduction in Video-Based Human Activity Recognition

Audio- and Video-Based Human Activity Recognition Systems in Healthcare

Audio- and Video-Based Human Activity Recognition Systems in Healthcare

Unsupervised Human Activity Recognition Learning for Disassembly Tasks

Large volumes of used electronics are often collected in remanufacturing plants, which requires disassembly before harvesting parts for reuse. Disassembly is mainly conducted manually with low productivity. Recently, human-robot collaboration is considered as a solution. For robots to assist effectively, they should observe work environments and recognize human actions accurately. Rich activity video recording and supervised learning can be used to extract insights; however, supervised learning does not allow robots to self-accomplish the learning process. This study proposes an unsupervised learning framework for achieving video-based human activity recognition. The framework consists of two main elements: a variational autoencoder-based architecture for unlabeled data representation learning, and a hidden Markov model for activity state division. The complete explicit activity classification is validated against ground truth labels; here, we use a case study of disassembling a hard disk drive. The framework shows an average recognition accuracy of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$91.52\%$</tex-math></inline-formula> , higher than competing methods.

A smart e-health framework for monitoring the health of the elderly and disabled

The healthcare sector is experiencing a significant transformation due to the widespread adoption of IoT-based systems, especially in the care of elderly and disabled individuals who can now be monitored through portable and wearable devices. The widespread implementation of IoT-based systems in the healthcare sector holds immense potential for improving care and ensuring the well-being of vulnerable populations. In our paper, we introduce an e-health framework that utilizes real-time data from inertial, ECG, and video sensors to monitor the health and activities of these individuals. Our framework employs edge computing for efficient analysis and generates notifications based on data analysis while prioritizing privacy by activating multimedia sensors only when necessary. To achieve accurate results, we evaluated each component of our framework separately. Using the MHEALTH dataset, our proposed machine learning model achieves accuracies of 99.97% for inertial sensors’ human activity recognition (HAR) and 99.14% for ECG sensors. We also evaluated our first-level inconsistency detection-based anomaly proposal mechanism using one user’s data, demonstrating its proof-of-concept. Furthermore, our video-based HAR and fall detection module achieve an accuracy of 86.97% on the well-known DMLSmartActions dataset. We successfully deployed our proposed HAR model with inertial sensors in a controlled experimental environment, yielding promising results with an accuracy of 96% and an F1 score of 90.52%. These evaluations validate the effectiveness of our framework in monitoring the health and activities of elderly and disabled individuals.

A Hierarchical Framework for Video-Based Human Activity Recognition Using Body Part Interactions

Human Activity Recognition (HAR) is an important field with diverse applications. However, video-based HAR is challenging because of various factors, such as noise, multiple people, and obscured body parts. Moreover, it is difficult to identify similar activities within and across classes. This study presents a novel approach that utilizes body region relationships as features and a two-level hierarchical model for classification to address these challenges. The proposed system uses a Hidden Markov Model (HMM) at the first level to model human activity, and similar activities are then grouped and classified using a Support Vector Machine (SVM) at the second level. The performance of the proposed system was evaluated on four datasets, with superior results observed for the KTH and Basic Kitchen Activity (BKA) datasets. Promising results were obtained for the HMDB-51 and UCF101 datasets. Improvements of 25%, 25%, 4%, 22%, 24%, and 30% in accuracy, recall, specificity, Precision, F1-score, and MCC, respectively, are achieved for the KTH dataset. On the BKA dataset, the second level of the system shows improvements of 8.6%, 8.6%, 0.85%, 8.2%, 8.4%, and 9.5% for the same metrics compared to the first level. These findings demonstrate the potential of the proposed two-level hierarchical system for human activity recognition applications.

A multidimensional feature fusion network based on MGSE and TAAC for video-based human action recognition

With the maturity of intelligent technology such as human-computer interaction, human action recognition (HAR) technology has been widely used in virtual reality, video surveillance, and other fields. However, the current video-based HAR methods still cannot fully extract abstract action features, and there is still a lack of action collection and recognition for special personnel such as prisoners and elderly people living alone. To solve the above problems, this paper proposes a multidimensional feature fusion network, called P-MTSC3D, a parallel network based on context modeling and temporal adaptive attention module. It consists of three branches. The first branch serves as the basic network branch, which extracts basic feature information. The second branch consists of a feature pre-extraction layer and two multiscale-convolution-based global context modeling combined squeeze and excitation (MGSE) modules, which can extract spatial and channel features. The third branch consists of two temporal adaptive attention units based on convolution (TAAC) to extract temporal dimension features. In order to verify the validity of the proposed network, this paper conducts experiments on the University of Central Florida (UCF) 101 dataset and the human motion database (HMDB) 51 dataset. The recognition accuracy of the proposed P-MTSC3D network is 97.92% on the UCF101 dataset and 75.59% on the HMDB51 dataset, respectively. The FLOPs of the P-MTSC3D network is 30.85G, and the test time is 2.83 s/16 samples on the UCF101 dataset. The experimental results demonstrate that the P-MTSC3D network has better overall performance than the state-of-the-art networks. In addition, a prison action (PA) dataset is constructed in this paper to verify the application effect of the proposed network in actual scenarios.

Attentional Composition Networks for Long-Tailed Human Action Recognition

The problem of long-tailed visual recognition has been receiving increasing research attention. However, the long-tailed distribution problem remains underexplored for video-based visual recognition. To address this issue, in this article we propose a compositional learning based solution for video-based human action recognition. Our method, named Attentional Composition Networks (ACN), first learns verb-like and preposition-like components, then shuffles these components to generate samples for the tail classes in the feature space to augment the data for the tail classes. Specifically, during training, we represent each action video by a graph that captures the spatial-temporal relations (edges) among detected human/object instances (nodes). Then, ACN utilizes the position information to decompose each action into a set of verb and preposition representations using the edge features in the graph. After that, the verb and preposition features from different videos are combined via an attention structure to synthesize feature representations for tail classes. This way, we can enrich the data for the tail classes and consequently improve the action recognition for these classes. To evaluate the compositional human action recognition, we further contribute a new human action recognition dataset, namely NEU-Interaction (NEU-I). Experimental results on both Something-Something V2 and the proposed NEU-I demonstrate the effectiveness of the proposed method for long-tailed, few-shot, and zero-shot problems in human action recognition. Source code and the NEU-I dataset are available at https://github.com/YajieW99/ACN .

An Overview of the Vision-Based Human Action Recognition Field

Artificial intelligence’s rapid advancement has enabled various applications, including intelligent video surveillance systems, assisted living, and human–computer interaction. These applications often require one core task: video-based human action recognition. Research in human video-based human action recognition is vast and ongoing, making it difficult to assess the full scope of available methods and current trends. This survey concisely explores the vision-based human action recognition field and defines core concepts, including definitions and explanations of the common challenges and most used datasets. Additionally, we provide in an easy-to-understand manner the literature approaches and their evolution over time, emphasizing intuitive notions. Finally, we explore current research directions and potential future paths. The core goal of this work is to provide future works with a shared understanding of fundamental ideas and clear intuitions about current works and find new research opportunities.

A Discriminant Information Theoretic Learning Framework for Multi-modal Feature Representation

As sensory and computing technology advances, multi-modal features have been playing a central role in ubiquitously representing patterns and phenomena for effective information analysis and recognition. As a result, multi-modal feature representation is becoming a progressively significant direction of academic research and real applications. Nevertheless, numerous challenges remain ahead, especially in the joint utilization of discriminatory representations and complementary representations from multi-modal features. In this article, a discriminant information theoretic learning (DITL) framework is proposed to address these challenges. By employing this proposed framework, the discrimination and complementation within the given multi-modal features are exploited jointly, resulting in a high-quality feature representation. According to characteristics of the DITL framework, the newly generated feature representation is further optimized, leading to lower computational complexity and improved system performance. To demonstrate the effectiveness and generality of DITL, we conducted experiments on several recognition examples, including both static cases, such as handwritten digit recognition, face recognition, and object recognition, and dynamic cases, such as video-based human emotion recognition and action recognition. The results show that the proposed framework outperforms state-of-the-art algorithms.

Multi-level channel attention excitation network for human action recognition in videos

Channel attention mechanism has continuously attracted strong interests and shown great potential in enhancing the performance of deep CNNs. However, when applied to video-based human action recognition task, most existing methods generally learn channel attention at frame level, which ignores the temporal dependencies and may limit the recognition performance. In this paper, we propose a novel multi-level channel attention excitation (MCAE) module to model the temporal-related channel attention at both frame and video levels. Specifically, based on video convolutional feature maps, frame-level channel attention (FCA) is generated by exploring time-channel correlations, and video-level channel attention (VCA) is generated by aggregating global motion variations. MCAE firstly recalibrates video feature responses with frame-wise FCA, and then activates the motion-sensitive channel features with motion-aware VCA. MCAE module learns the channel discriminability from multiple levels and can act as a guidance to facilitate efficient spatiotemporal feature modeling in activated motion-sensitive channels. It can be flexibly embedded into 2D networks with very limited extra computation cost to construct MCAE-Net, which effectively enhances the spatiotemporal representation of 2D models for video action recognition task Extensive experiments on five human action datasets show that our method achieves superior or very competitive performance compared with the state-of-the-arts, which demonstrates the effectiveness of the proposed method for improving the performance of human action recognition.

Video-Based Metric Learning Framework for Basketball Skill Assessment

Video-based human action recognition has become one of the research hotspots in the field of computer vision in recent years and has been widely used in the fields of intelligent human-computer interaction and virtual reality. However, most of the current existing methods and public datasets are constructed for human daily activities, and the assessment of basketball skills is still a challenging problem. In order to solve the above issues, in this paper, the authors propose a coarse-to-fine video-based metric learning framework for basketball skills assessment. Specifically, they first use a variety of models to jointly represent the action video, and then the optimal distance metric between videos is learned based on the representation. Finally, based on the distance metric, a query video is coarsely classified to obtain the corresponding label of video action, and then the video is finely classified to judge whether the action is standardized. The experiments on a collected dataset show that the proposed framework can better identify and assess the non-standard actions of basketball.

Compressing Deep Model With Pruning and Tucker Decomposition for Smart Embedded Systems

Deep learning has been proved to be one of the most effective method in feature encoding for different intelligent applications such as video-based human action recognition. However, its nonconvex optimization mechanism leads large memory consumption, which hinders its deployment on the smart embedded systems with limited computational resources. To overcome this challenge, we propose a novel deep model compression technique for smart embedded systems, which realizes both the memory size reduction and inference complexity decrease within a small drop of accuracy. First, we propose an improved naive Bayes inference-based channel parameter pruning to obtain a sparse model with higher accuracy. Then, to improve the inference efficiency, the improved Tucker decomposition method is proposed, where an improved genetic algorithm is used to optimize the Tucker ranks. Finally, to elevate the effectiveness of our proposed method, extensive experiments are conducted. The experimental results show that our method can achieve the state-of-the-art performance compared with existing methods in terms of accuracy, parameter compression, and floating-point operations reduction.

Video Based Human Activity Recognition Surveillance System

It is argued that current state-of-the-art methods of home surveillance such as motion detection technology like CCTV intrusion alert are insufficient, in particular to cater the modern need of whole automation with vulnerabilities such as requiring human involvement. We propose an alternative system, a video-based Human Activity Recognition (HAR) approach using the combination of Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) algorithm, to dealwith the identified shortcomings. Our proposal does not require any changes to the existing home security protocols and is easily implemented using only low-cost, commercial-off-the-shelf hardware. We can easily use the traditional surveillance camera for computer vision tasks. We evaluate our approach using real-world activity data collected via video-based sensor. We quantify its effectiveness, by plotting Loss and accuracy curves. Our results show that the video-based HAR approach can provide full automation in home surveillancesystem compared to conventional CCTV motion detectors by an accuracy of more than 93%. Further system’s accuracy can be increased and we can achieve significantly better results by implementing LRCN (Long Term Recurrent Convolution Network) approach.

Advances in human action, activity and gesture recognition

A set of advanced approaches and models on human action, activity, gesture, and behavior recognition related aspects along with associated challenges are summarized in this note. Notably, Video-based human activity recognition, sensor-based activity analysis, skeleton-based activity recognition, assisted daily living, anomaly detection, facial expression and emotion analysis, gesture and sign language are highlighted in the works. This special issue also introduced six new datasets, while exploring a total of 55 different datasets in its 21 research articles. Apart from the the areas covered in this issue, research on multi-modal analysis, action quality assessment, real-time applications, and activity and behavior computing on edge devices are some of the dominant future challenges to deal with. We firmly believe that the original works and ideas presented in this special issue will serve as helpful references for the relevant research communities in the journey towards a brighter future.

Best Frame Selection to Enhance Training Step Efficiency in Video-Based Human Action Recognition

In recent years, with the growth of digital media and modern imaging equipment, the use of video processing algorithms and semantic film and image management has expanded. The usage of different video datasets in training artificial intelligence algorithms is also rapidly expanding in various fields. Due to the high volume of information in a video, its processing is still expensive for most hardware systems, mainly in terms of its required runtime and memory. Hence, the optimal selection of keyframes to minimize redundant information in video processing systems has become noteworthy in facilitating this problem. Eliminating some frames can simultaneously reduce the required computational load, hardware cost, memory and processing time of intelligent video-based systems. Based on the aforementioned reasons, this research proposes a method for selecting keyframes and adaptive cropping input video for human action recognition (HAR) systems. The proposed method combines edge detection, simple difference, adaptive thresholding and 1D and 2D average filter algorithms in a hierarchical method. Some HAR methods are trained with videos processed by the proposed method to assess its efficiency. The results demonstrate that the application of the proposed method increases the accuracy of the HAR system by up to 3% compared to random image selection and cropping methods. Additionally, for most cases, the proposed method reduces the training time of the used machine learning algorithm.