Problem Of Action Recognition Research Articles

STO-CVAE: state transition-oriented conditional variational autoencoder for data augmentation in disability classification

The class imbalance problem occurs when there is an unequal distribution of classes in a dataset and is a significant issue in various artificial intelligence applications. This study focuses on the severe multiclass imbalance problem of human activity recognition in rehabilitation exercises for people with disabilities. To overcome this problem, we present a novel human action-centric augmentation method for human skeleton-based pose estimation. This study proposes the state transition-oriented conditional variational autoencoder (STO-CVAE) to capture action patterns in repeated exercises. The proposed approach generates action samples by capturing temporal information of human skeletons to improve the identification of minority disability classes. We conducted experimental studies with a real-world dataset gathered from rehabilitation exercises and confirmed the superiority and effectiveness of the proposed method. Specifically, all investigated classifiers (i.e., random forest, support vector machine, extreme gradient boosting, light gradient boosting machine, and TabNet) trained with the proposed augmentation method outperformed the models trained without augmentation in terms of the F1-score and accuracy, with F1-score showing the most improvement. Overall, the prediction accuracy of most classes was improved; in particular, the prediction accuracy of the minority classes was greatly improved. Hence, the proposed STO-CVAE can be used to improve the accuracy of disability classification in the field of physical medicine and rehabilitation and to provide suitable personal training and rehabilitation exercise programs.

Graph-Structure-Based Multigranular Belief Fusion for Human Activity Recognition.

The belief functions (BFs) introduced by Shafer in the mid of 1970s are widely applied in information fusion to model epistemic uncertainty and to reason about uncertainty. Their success in applications is however limited because of their high-computational complexity in the fusion process, especially when the number of focal elements is large. To reduce the complexity of reasoning with BFs, we can envisage as a first method to reduce the number of focal elements involved in the fusion process to convert the original basic belief assignments (BBAs) into simpler ones, or as a second method to use a simple rule of combination with potentially a loss of the specificity and pertinence of the fusion result, or to apply both methods jointly. In this article, we focus on the first method and propose a new BBA granulation method inspired by the community clustering of nodes in graph networks. This article studies a novel efficient multigranular belief fusion (MGBF) method. Specifically, focal elements are regarded as nodes in the graph structure, and the distance between nodes will be used to discover the local community relationship of focal elements. Afterward, the nodes belonging to the decision-making community are specially selected, and then the derived multigranular sources of evidence can be efficiently combined. To evaluate the effectiveness of the proposed graph-based MGBF, we further apply this new approach to combine the outputs of convolutional neural networks + attention (CNN + Attention) in the human activity recognition (HAR) problem. The experimental results obtained with real datasets prove the potential interest and feasibility of our proposed strategy with respect to classical BF fusion methods.

Higher Order Polynomial Transformer for Fine-Grained Freezing of Gait Detection.

Freezing of Gait (FoG) is a common symptom of Parkinson's disease (PD), manifesting as a brief, episodic absence, or marked reduction in walking, despite a patient's intention to move. Clinical assessment of FoG events from manual observations by experts is both time-consuming and highly subjective. Therefore, machine learning-based FoG identification methods would be desirable. In this article, we address this task as a fine-grained human action recognition problem based on vision inputs. A novel deep learning architecture, namely, higher order polynomial transformer (HP-Transformer), is proposed to incorporate pose and appearance feature sequences to formulate fine-grained FoG patterns. In particular, a higher order self-attention mechanism is proposed based on higher order polynomials. To this end, linear, bilinear, and trilinear transformers are formulated in pursuit of discriminative fine-grained representations. These representations are treated as multiple streams and further fused by a cross-order fusion strategy for FoG detection. Comprehensive experiments on a large in-house dataset collected during clinical assessments demonstrate the effectiveness of the proposed method, and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.92 is achieved for detecting FoG.

Deep Reinforcement Learning in Human Activity Recognition: A Survey and Outlook.

Human activity recognition (HAR) is a popular research field in computer vision that has already been widely studied. However, it is still an active research field since it plays an important role in many current and emerging real-world intelligent systems, like visual surveillance and human-computer interaction. Deep reinforcement learning (DRL) has recently been used to address the activity recognition problem with various purposes, such as finding attention in video data or obtaining the best network structure. DRL-based HAR has only been around for a short time, and it is a challenging, novel field of study. Therefore, to facilitate further research in this area, we have constructed a comprehensive survey on activity recognition methods that incorporate DRL. Throughout the article, we classify these methods according to their shared objectives and delve into how they are ingeniously framed within the DRL framework. As we navigate through the survey, we conclude by shedding light on the prominent challenges and lingering questions that await the attention of future researchers, paving the way for further advancements and breakthroughs in this exciting domain.

Human Behavior Recognition Based on CNN-LSTM Hybrid and Multi-Sensing Feature Information Fusion

To address the human activity recognition problem and its application in practical situations, a CNN-LSTM hybrid neural network model capable of automatically extracting sensor data features and memorizing temporal activity data is designed and improved by integrating CNN and gated recurrent units as a variant of RNN. A multi-channel spatiotemporal fusion network-based two-person interaction behavior recognition method is proposed for two-person skeletal sequential behavior recognition. Firstly, a viewpoint invariant feature extraction method is used to extract two-player skeleton features, then a two-layer cascaded spatiotemporal fusion network model is designed, and finally, a multi-channel spatiotemporal fusion network is used to learn multiple sets of two-player skeleton features separately to obtain multi-channel fusion features, and the fusion features are used to recognize the interaction behavior, and the weights are shared among the channels. Applying the algorithm in the paper to the UCF101 dataset for experiments, the accuracy of the two-person cross-object experiment can reach 96.42% and the accuracy of the cross-view experiment can reach 97.46%. The method in the paper shows better performance in two-player interaction behavior recognition compared to typical methods in this field.

In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and Beyond

Predicting human’s gaze from egocentric videos serves as a critical role for human intention understanding in daily activities. In this paper, we present the first transformer-based model to address the challenging problem of egocentric gaze estimation. We observe that the connection between the global scene context and local visual information is vital for localizing the gaze fixation from egocentric video frames. To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel global–local correlation module to explicitly model the correlation of the global token and each local token. We validate our model on two egocentric video datasets – EGTEA Gaze + and Ego4D. Our detailed ablation studies demonstrate the benefits of our method. In addition, our approach exceeds the previous state-of-the-art model by a large margin. We also apply our model to a novel gaze saccade/fixation prediction task and the traditional action recognition problem. The consistent gains suggest the strong generalization capability of our model. We also provide additional visualizations to support our claim that global–local correlation serves a key representation for predicting gaze fixation from egocentric videos. More details can be found in our website (https://bolinlai.github.io/GLC-EgoGazeEst).

Transfer learning-based security protection framework for smart exercise monitoring

Sports exercise is very important for both physical and mental health, but improper exercise or equipment use may cause some security challenges. To address this issue, facing Internet of Things (IoT), currently, some video recognition algorithm-based smart security monitoring systems have been designed. However, existing video recognition algorithms usually assume that the data comes from the same collection device or follows the same distribution, resulting in ineffective handling of cross-camera or cross-device recognition problems. In this vein, this paper designed a transfer learning-based smart exercise monitoring security protection system, and proposed a new transfer learning-based video recognition framework, which consists of backbone network module, style transfer module, video feature aggregation module three parts and using this framework, two different models can be trained based on video face recognition dataset and video action recognition dataset, respectively, for identity recognition and action recognition. Experimental results show that the proposed framework can effectively handle video face recognition and video action recognition problems, which also demonstrates that our designed smart exercise monitoring security protection system can meet actual task requirements.

Research on the identification and integration of folk dance creation elements based on big data technology

Abstract In order to identify the elements of folk dance creation, two methods of continuous folk dance movement recognition are proposed in this paper. By considering the dance movement patterns as a whole and the templates as incomplete patterns, OE-DTW is applied to match them to segment and recognize each movement pattern one by one. To enhance the performance, a global restriction K-Repetition and endpoint detection condition are proposed to match with OE-DTW, and a penalty-based layer matching algorithm is proposed by using the layer structure feature of SegSVD so that the endpoint of the input movement pattern can be determined by the top layer matching of the latter when matching with the template pattern. Meanwhile, the similarity between two patterns based on this algorithm can also be calculated by the local results obtained from the weighted cumulative layer matching. Experiments show that the average recognition rate of OE-DTW is 0.891, SegSVD is 0.86, and CDP is 0.828. Both methods can effectively deal with the continuous action recognition problem, and they have better recognition results compared with CDP.

Human pose estimation and action recognition for fitness movements

Human pose estimation and action recognition have important applications in the areas of security, medicine and sports. In this work, we propose an improved YOLOv7-Pose algorithm to solve the problem of human posture and action recognition for fitness movements in various scenarios. For the algorithm based on YOLOv7-Pose, we add the function of classification to the original network. And we introduce Coordinate Attention to the network to improve the model’s ability to identify important features of human skeletal joints and action classifications. The improved ConvNeXt network structure is introduced to replace the CBS convolution kernel of the original model to improve the accuracy of human key point detection and action classification of the model. We optimise the spatial pyramid pool structure, which can reduce the loss function and accelerate the convergence rate of the model. We adopt EIOU as the regression function of the target detection frame to improve the accuracy of the coordinate regression. Experiments show that the improved YOLOv7-Pose has an mAP of 95.9% on a homemade test set of fitness actions, which is 5.4% higher than HRNet, with a 4.2% improvement over the original YOLOv7 algorithm, suggesting that the accuracy of action recognition and key point estimation has significantly improved.

Hybrid Learning Models for IMU-Based HAR with Feature Analysis and Data Correction.

This paper proposes a novel approach to tackle the human activity recognition (HAR) problem. Four classes of body movement datasets, namely stand-up, sit-down, run, and walk, are applied to perform HAR. Instead of using vision-based solutions, we address the HAR challenge by implementing a real-time HAR system architecture with a wearable inertial measurement unit (IMU) sensor, which aims to achieve networked sensing and data sampling of human activity, data pre-processing and feature analysis, data generation and correction, and activity classification using hybrid learning models. Referring to the experimental results, the proposed system selects the pre-trained eXtreme Gradient Boosting (XGBoost) model and the Convolutional Variational Autoencoder (CVAE) model as the classifier and generator, respectively, with 96.03% classification accuracy.

Attentional Composition Networks for Long-Tailed Human Action Recognition

The problem of long-tailed visual recognition has been receiving increasing research attention. However, the long-tailed distribution problem remains underexplored for video-based visual recognition. To address this issue, in this article we propose a compositional learning based solution for video-based human action recognition. Our method, named Attentional Composition Networks (ACN), first learns verb-like and preposition-like components, then shuffles these components to generate samples for the tail classes in the feature space to augment the data for the tail classes. Specifically, during training, we represent each action video by a graph that captures the spatial-temporal relations (edges) among detected human/object instances (nodes). Then, ACN utilizes the position information to decompose each action into a set of verb and preposition representations using the edge features in the graph. After that, the verb and preposition features from different videos are combined via an attention structure to synthesize feature representations for tail classes. This way, we can enrich the data for the tail classes and consequently improve the action recognition for these classes. To evaluate the compositional human action recognition, we further contribute a new human action recognition dataset, namely NEU-Interaction (NEU-I). Experimental results on both Something-Something V2 and the proposed NEU-I demonstrate the effectiveness of the proposed method for long-tailed, few-shot, and zero-shot problems in human action recognition. Source code and the NEU-I dataset are available at https://github.com/YajieW99/ACN .

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

Sequential attention mechanism for weakly supervised video anomaly detection

Surveillance cameras are installed across various sectors of a smart city in order to capture ongoing events for monitoring purposes. The analysis of these surveillance videos is an important research topic that involves activity recognition, object detection, anomaly recognition, and other problems. However, anomaly recognition is the most common task in a smart city, and has received significant attention with the aim of ensuring public safety and security. Many works have been published in this field, but these schemes have not been able to provide the desired detection outcomes. Mainstream anomaly recognition methods are heavily dependent on strong supervision to achieve satisfactory performance, which is time-consuming and impractical. With a particular focus on this problem, this article presents a deep convolution neural network (CNN)-based novel anomaly recognition model, in which deep features are extracted from surveillance video frames. These features are forwarded to the proposed temporal convolution network (TCN) that includes a multi-head attention module to enable it to recognise anomalies from these videos. The multi-head temporal attention mechanism enables the model to obtain more key temporal information about the complex surveillance environment. Experiments conducted on standard datasets and a comparison with state-of-the-art approaches demonstrate the effectiveness and superiority of the proposed framework, which achieves increases in accuracy of 0.9%, 1.9%, 0.65%, 0.27%, and 1.5% on the UCF-Crime2local, LAD-2000, RWF-2000, RLVS, and Crowd Violence datasets, respectively. These outcomes indicate the suitability of our method for deployment in real-time surveillance schemes.

Boosting multi‐target recognition performance with multi‐input multi‐output radar‐based angular subspace projection and multi‐view deep neural network

AbstractCurrent radio frequency (RF) classification techniques assume only one target in the field of view. Multi‐target recognition is challenging because conventional radar signal processing results in the superposition of target micro‐Doppler signatures, making it difficult to recognise multi‐target activity. This study proposes an angular subspace projection technique that generates multiple radar data cubes (RDC) conditioned on angle (RDC‐ω). This approach enables signal separation in the raw RDC, making possible the utilisation of deep neural networks taking the raw RF data as input or any other data representation in multi‐target scenarios. When targets are in closer proximity and cannot be separated by classical techniques, the proposed approach boosts the relative signal‐to‐noise ratio between targets, resulting in multi‐view spectrograms that boosts the classification accuracy when input to the proposed multi‐view DNN. Our results qualitatively and quantitatively characterise the similarity of multi‐view signatures to those acquired in a single‐target configuration. For a nine‐class activity recognition problem, 97.8% accuracy in a 3‐person scenario is achieved, while utilising DNN trained on single‐target data. We also present the results for two cases of close proximity (sign language recognition and side‐by‐side activities), where the proposed approach has boosted the performance.

Human action recognition model incorporating multiscale temporal convolutional network and spatiotemporal excitation network

Human action recognition is a research hotspot in the field of computer vision. Focusing on the problem of similar action recognition, we propose an improved two-stream adaptive graph convolutional network for skeleton-based action recognition, which incorporating a multiscale temporal convolutional network and a spatiotemporal excitation network. Using the multiscale temporal convolutional network, the temporal information can be effectively extracted by dilated convolution at different scales so as to broaden the width of the temporal network and extract more temporal features with slight difference between categories at the same time. By utilizing the spatiotemporal excitation network, the input features can be obtained through channel pooling to form single-channel features for two-dimensional convolution, by which important spatiotemporal information can be excited and the role of local nodes in similar actions can be effectively enhanced. Extensive tests and ablation studies on the three large-scale datasets, NTU-RGB+D60, NTU-RGB+D120, and Kinetics-Skeleton, were conducted. Our model outperforms the baseline by 7.2% and the state-of-the-art model by 4% in the similar action recognition on NTU-RGB+D60 dataset on average, which demonstrates the superiority of our model.

Energy efficient integrated MEMS neural network for simultaneous sensing and computing

Biological systems seamlessly combine multiple functions in lightweight and energy-efficient structures. Such capability in synthetic structures would be desirable in numerous engineering applications such as aerospace, robotics and wearable devices. Here we report an integrated silicon-based structure configured to sense, perform different classification algorithms, and produce an action signal within the same physical layer. The algorithms are coded in the mechanical responses of the sensing elements of multiple coupled micro-electro-mechanical systems (MEMS), simultaneously capturing acceleration measurements to produce an actuated signal. This all-in-one structure operates with zero circuitry and low power consumption. As a demonstration, we designed and fabricated a network of three MEMS neurons to successfully perform both simple signal classification and activity recognition problems (standing and sitting) with only 9.92 × 10−17 kWh and 17.79 × 10−19 kWh energy consumption per operation, respectively. Our approach will enable emergent technologies, such as wearable devices, to perform complex computations with power from a single battery charge.

Efficient Driver Anomaly Detection via Conditional Temporal Proposal and Classification Network

Detecting driver inattentive behaviors is crucial for driving safety in a driver monitoring system (DMS). Recent works treat driver distraction detection as a multiclass action recognition problem or a binary anomaly detection problem. The former approach aims to classify a fixed set of action classes. Although specific distraction classes can be predicted, this approach is inflexible to detect unknown driver anomalies. The latter approach mixes all distraction actions into one class: anomalous driving. Because the objective focuses on finding the difference between safe and distracted driving, this approach has better generalization in detecting unknown driver distractions. However, a detailed classification of the distraction is missing from the predictions, meaning that the downstream DMS can only treat all distractions with the same severity. In this work, we propose a two-phase anomaly proposal and classification framework [driver anomaly detection and classification network (DADCNet)] robust for open-set anomalies while maintaining high-level distraction understanding. DADCNet makes efficient allocation of multimodal and multiview inputs. The anomaly proposal network first utilizes a subset of the available modalities and views to suggest suspicious anomalous driving behavior. Then, the classification network employs more features to verify the anomaly proposal and classify the proposed distraction action. Through extensive experiments in two driver distraction datasets, our approach significantly reduces the total amount of computation during inference time while maintaining high anomaly detection sensitivity and robust performance in classifying common driver distractions.

The basic assembly of skeletal models in the fall detection problem

The paper considers the appliance of the featureless approach to the human activity recognition problem, which exclude the direct anthropomorphic and visual characteristics of human figure from further analysis and thus increase the privacy of the monitoring system. A generalized pairwise comparison function of two human skeletal models, invariant to the sensor type, is used to project the object of interest to the secondary feature space, formed by the basic assembly of skeletons. A sequence of such projections in time forms an activity map, which allows an application of deep learning methods based on convolution neural networks for activity recognition. The proper ordering of skeletal models in a basic assembly plays an important role in secondary space design. The study of ordering of the basic assembly by the shortest unclosed path algorithm and correspondent activity maps for video streams from the TST Fall Detection v2 database are presented.

A convolutional autoencoder model with weighted multi-scale attention modules for 3D skeleton-based action recognition

The 3D skeleton sequences of action can be recognized based on series of meaningful movements including changes in the direction and geometry features of the body pose. In this paper, we introduce the 3DPo-CDP descriptor, which incorporates the change direction patterns of body joints and pose features in a unified deep structure to learn more adequate features for the action recognition problem. To this end, two types of features are extracted. First, Change Direction Patterns (CDPs) are extracted by following the important points of motion trajectories where a significant change of direction has occurred using two filtering phases. The CDPs capture the global features which are invariant to noise and insignificant temporal dynamics of joints. Second, Pose Features are employed to learn the intrinsic connectivity relationships of adjacent limbs and the variance distances of body joints from representative joints to concentrate on key-frames and informative joints. The complementary features of CDPs and 3D pose, which are transformed into images, are combined in a unified representation and fed into a new convolutional autoencoder. Unlike conventional convolutional autoencoders that focus on frames, high-level discriminative features of spatiotemporal relationships of whole body joints are extracted by introducing weighted multi-scale channel and spatial attention modules. In this paper, we show that adjacent and non-adjacent neighbors can be effectively used to compute different weights for extracting cross-interaction channels and multi-scale spatial relationships of the current pixel. The extracted features are combined with the wavelet representation of statistical body information and then classified with a multi-class SVM classifier. The experimental results demonstrate the effectiveness of the augmented 3DPo-CDP descriptor using an attentional convolution autoencoder structure on five challenging 3D action recognition datasets.

Wrapper-based deep feature optimization for activity recognition in the wearable sensor networks of healthcare systems

The Human Activity Recognition (HAR) problem leverages pattern recognition to classify physical human activities as they are captured by several sensor modalities. Remote monitoring of an individual’s activities has gained importance due to the reduction in travel and physical activities during the pandemic. Research on HAR enables one person to either remotely monitor or recognize another person’s activity via the ubiquitous mobile device or by using sensor-based Internet of Things (IoT). Our proposed work focuses on the accurate classification of daily human activities from both accelerometer and gyroscope sensor data after converting into spectrogram images. The feature extraction process follows by leveraging the pre-trained weights of two popular and efficient transfer learning convolutional neural network models. Finally, a wrapper-based feature selection method has been employed for selecting the optimal feature subset that both reduces the training time and improves the final classification performance. The proposed HAR model has been tested on the three benchmark datasets namely, HARTH, KU-HAR and HuGaDB and has achieved 88.89%, 97.97% and 93.82% respectively on these datasets. It is to be noted that the proposed HAR model achieves an improvement of about 21%, 20% and 6% in the overall classification accuracies while utilizing only 52%, 45% and 60% of the original feature set for HuGaDB, KU-HAR and HARTH datasets respectively. This proves the effectiveness of our proposed wrapper-based feature selection HAR methodology.