Action Recognition Task Research Articles

Egocentric Action Recognition by Automatic Relation Modeling.

Egocentric videos, which record the daily activities of individuals from a first-person point of view, have attracted increasing attention during recent years because of their growing use in many popular applications, including life logging, health monitoring and virtual reality. As a fundamental problem in egocentric vision, one of the tasks of egocentric action recognition aims to recognize the actions of the camera wearers from egocentric videos. In egocentric action recognition, relation modeling is important, because the interactions between the camera wearer and the recorded persons or objects form complex relations in egocentric videos. However, only a few of existing methods model the relations between the camera wearer and the interacting persons for egocentric action recognition, and moreover they require prior knowledge or auxiliary data to localize the interacting persons. In this work, we consider modeling the relations in a weakly supervised manner, i.e., without using annotations or prior knowledge about the interacting persons or objects, for egocentric action recognition. We form a weakly supervised framework by unifying automatic interactor localization and explicit relation modeling for the purpose of automatic relation modeling. First, we learn to automatically localize the interactors, i.e., the body parts of the camera wearer and the persons or objects that the camera wearer interacts with, by learning a series of keypoints directly from video data to localize the action-relevant regions with only action labels and some constraints on these keypoints. Second, more importantly, to explicitly model the relations between the interactors, we develop an ego-relational LSTM (long short-term memory) network with several candidate connections to model the complex relations in egocentric videos, such as the temporal, interactive, and contextual relations. In particular, to reduce human efforts and manual interventions needed to construct an optimal ego-relational LSTM structure, we search for the optimal connections by employing a differentiable network architecture search mechanism, which automatically constructs the ego-relational LSTM network to explicitly model different relations for egocentric action recognition. We conduct extensive experiments on egocentric video datasets to illustrate the effectiveness of our method.

A Multitask Deep Learning Approach for Sensor-Based Human Activity Recognition and Segmentation

Deep learning for sensor-based human activity recognition (HAR) has been a focus of research in recent years. Sensor data stream segmentation is a core element in HAR, which has currently been treated as an independent preprocessing task, usually with a fixed-size window. This has led to two critical problems, namely the multi-class window problem caused by possible multiple activities within a fixed-size window and the fluctuation of prediction results due to noisy data and over-segmentation. To address these research challenges, in this paper, we conceive a novel Multi-Task deep learning approach to segmenting and recognizing human activity simultaneously. Specifically, we propose a multi-scale window method based on feature sequence generation to overcome the multi-class window problem. We develop a novel boundary offset prediction algorithm to adjust a windows boundary to tackle the over-segmentation issue. In addition, We design a multi-task framework to streamline and optimize the activity recognition and segmentation tasks simultaneously. We conduct extensive experiments on eight benchmark datasets to evaluate the proposed framework and associated methods. Initial results show that our approach outperforms the performance of current state-of-the-art HAR methods.

Temporal–Spatial Dynamic Convolutional Neural Network for Human Activity Recognition Using Wearable Sensors

The Sensor-based Human Activity Recognition (SHAR) task seeks to recognize signals collected by various sensors embedded in intelligent devices to assist people in their daily lives. Motivated by the success of deep learning, many researchers are studying combining deep learning with SHAR. The key to implementing sensor-based human activity recognition with deep learning lies in facilitating model performance and maintaining efficiency when the model is performed on resource-constrained devices. To address this challenge, we present an effective sensor signal representation method, termed the Temporal-Spatial Dynamic Convolutional (TS-DyConv) network, to recognize human activity. TS-DyConv aims to dynamically learn the convolutional kernels weighted with attention generated along the temporal and spatial kernel spaces. This way, the TS-DyConv can diversify the kernel to enhance the sensor signal’s recognition capabilities without raising complexity and maintaining efficiency. Extensive experiments conducted on three benchmark SHAR datasets, e.g., OPPORTUNITY, PAMAP2, and USC-HAD, demonstrate the superiority of our method over the deep learning baselines and existing SHAR works.

LgNet: A Local-Global Network for Action Recognition and Beyond

This work addresses the task of action recognition in video sequences. In real world applications, this task is quite challenging due to the complex background of video content, the similarities between different types of actions, the dependence on a large amount of annotated data, and so on. Most of the existing methods fail to distinguish similar actions with the same static appearance and motion pattern. We attempt to address this issue from the perspective of a local-global view, considering videos as combinations of a set of action units (local semantic information) and their relations along temporal dimension (global relation information). To achieve this end, we propose a novel Local-global Networks (LgNet) to enhance recognition of similar action. Besides, we propose an end-to-end training method to decrease the reliance on annotated data. It combines self-supervised learning and supervised learning, which not only enables the model to learn video representations from a large number unannotated data but also avoids subsequent finetuning. The proposed training method can be flexibly equipped to a wide array of vision tasks. Experiments on several benchmark datasets show that our proposed model and training method achieve state-of-the-art performance.

Skeleton-CutMix: Mixing Up Skeleton With Probabilistic Bone Exchange for Supervised Domain Adaptation.

We present Skeleton-CutMix, a simple and effective skeleton augmentation framework for supervised domain adaptation and show its advantage in skeleton-based action recognition tasks. Existing approaches usually perform domain adaptation for action recognition with elaborate loss functions that aim to achieve domain alignment. However, they fail to capture the intrinsic characteristics of skeleton representation. Benefiting from the well-defined correspondence between bones of a pair of skeletons, we instead mitigate domain shift by fabricating skeleton data in a mixed domain, which mixes up bones from the source domain and the target domain. The fabricated skeletons in the mixed domain can be used to augment training data and train a more general and robust model for action recognition. Specifically, we hallucinate new skeletons by using pairs of skeletons from the source and target domains; a new skeleton is generated by exchanging some bones from the skeleton in the source domain with corresponding bones from the skeleton in the target domain, which resembles a cut-and-mix operation. When exchanging bones from different domains, we introduce a class-specific bone sampling strategy so that bones that are more important for an action class are exchanged with higher probability when generating augmentation samples for that class. We show experimentally that the simple bone exchange strategy for augmentation is efficient and effective and that distinctive motion features are preserved while mixing both action and style across domains. We validate our method in cross-dataset and cross-age settings on NTU-60 and ETRI-Activity3D datasets with an average gain of over 3% in terms of action recognition accuracy, and demonstrate its superior performance over previous domain adaptation approaches as well as other skeleton augmentation strategies.

HUMAN ACTION RECOGNITION: IMPROVING THE ACCURACY OF DEEP CONV-LSTM ARCHITECTURE THROUGH NOISE CLEANING PRIOR TO KEY FRAMES SELECTION

Recognizing human actions has numerous practical applications that can address issues and enhance the effectiveness and living standards across various domains. For solving the human action recognition task from videos, the Deep Conv-LSTM model architecture has been used on the UCF101 dataset. To preprocess the frames, a method has been proposed that integrates two algorithms: Noise Cleaning and Dissimilarity-Based Key Frame Selec¬t¬i¬o¬n (KFS). By employing these algorithms cohesively and effectively, image quality has been enhanced and unwanted data have been eliminated. The Uniform Frame Selection, Dissimilarity-Based KFS, and the proposed algorithm are evaluated, and their performances are compared based on the accuracy and data size reduction. The results show that the Dissimilarity-Based KFS algorithm outperforms the Uniform Frame Selection algorithm in accuracy by 2%, and the proposed method shows a 3% and 5% improvement in accuracy compared to the first two algorithms, respectively. Furthermore, the proposed algorithm reduces the data size by 26%, making it computationally efficient.

Exploring the Impact of Synthetic Data on Human Activity Recognition Tasks

This paper investigates the potential of synthetic data in enhancing the performance of Machine Learning classifiers. Our focus lies primarily on the Human Activity Recognition task, Specifically through wearable device sensors. We analyze the performance of three Generative Adversarial Networks (GANs) and a Diffuse model, considering fidelity, diversity, and generalization metrics. In addition, we assess the relationship between the addition of synthetic samples to the training data and the impact of imbalanced classes on the generative model in the production of synthetic samples. We also introduce a novel GAN designed to generate synthetic samples of time series data from wearable devices. After conducting nearly 400 experiments, our results suggest that synthetic samples can significantly improve the performance of Machine Learning models when real data are scarce. More importantly, our findings underline that data quality precedes the quantity of synthetic data added to training samples.

Toward Deep Transfer Learning for Realistic Activity Recognition in Videos

Today, videos have become popular on the internet and specified in social network and media platforms such as Youbue, Ticktok, and Vimeo. Video understanding has attracted much attention in the research community in recent years. Automatically recognizing human activity in wild videos is a trending research topic with a wide range of applications in advertising, smarthome, and surveillance camera systems. Deep convolutional neural networks have become a new de facto visual recognition problem. It achieved much success in image recognition problems that are leveraged from the ImageNet dataset. Many researchers have applied CNNs to the video domain, but the results in realistic video still have many challenges, and the recognition rate is not as expected. Because the realistic activity in video is extremely small, we cannot train a large deep convolutional network to achieve good performance. In this work, we answer the question “how could we transfer the visual features from the ImageNet dataset into video for activity recognition tasks?”. We propose an approach based on the pretrained models from ImageNet for activity recognition in video that is based on transforming video into an image map for temporal information and a frame-based description for spatial information. We test our proposed approach on three datasets, UCF11, UCF50 and UCF101, and it achieves an accuracy of over 95% for the backbone of ResNet and over 88% for the backbone of MobileNet. The experimental results show that our proposed method is robust and efficient in wild video datasets.

ActivBench: Leveraging Human Activity Inference from Smartphone Sensors for Human Computer Interactions

Human activity recognition (HAR) from smartphone sensors has gained significant attention due to its potential to enhance user experience (UX) and human computer interaction (HCI) in various domains, HAR can enable personalized, context-aware, and adaptive interfaces that improve accessibility and promote health and wellness in various applications such as healthcare, smart homes, fitness tracking, and context-aware systems. However, evaluating the performance of different machine learning (ML) algorithms on activity recognition tasks remains challenging, primarily due to the lack of standardized benchmark datasets and evaluation protocols. In this paper, we presented ActivBench, an end-to-end computational intelligence benchmark designed to facilitate the evaluation and comparison of ML algorithms for human activity inference from smartphone sensors. We addressed the challenges in benchmarking activity recognition systems by providing a unified evaluation protocol and standardized performance metrics. Through extensive experiments using various state-of-the-art algorithms, we demonstrated the effectiveness of ActivBench in assessing the strengths and limitations of different approaches. The benchmark results provide valuable insights into the strengths and limitations of different algorithms, facilitating the development of robust and accurate activity recognition systems that can enhance human computer interaction in various applications. ActivBench is serving as a valuable resource for researchers and practitioners in human activity recognition and human-computer interaction, enabling fair comparisons and fostering advancements in the field. It also serves as a catalyst for advancements in the field, enabling the exploration of novel algorithms, feature engineering techniques, and sensor modalities.

MAVEN: A Memory Augmented Recurrent Approach for Multimodal Fusion

Multisensory systems provide complementary information that aids many machine learning approaches in perceiving the environment comprehensively. These systems consist of heterogeneous modalities, which have disparate characteristics and feature distributions. Thus, extracting, aligning, and fusing complementary representations from heterogeneous modalities (e.g., visual, skeleton, and physical sensors) remains challenging. To address these challenges, we have used the insights from several neuroscience studies of animal multisensory systems to develop MAVEN, a memory-augmented recurrent approach for multimodal fusion. MAVEN generates unimodal memory banks comprised of spatial-temporal features and uses our proposed recurrent representation alignment approach to align and refine unimodal representations iteratively. MAVEN then utilizes a multimodal variational attention-based fusion approach to produce a robust multimodal representation from the aligned unimodal features. Our extensive experimental evaluations on three multimodal datasets suggest that MAVEN outperforms state-of-the-art multimodal learning approaches in the challenging human activity recognition task across all evaluation conditions (cross-subject, leave-one-subject-out, and cross-session). Additionally, our extensive ablation studies suggest that MAVEN significantly outperforms the feed-forward fusion-based learning models (p<0.05). Finally, the robust performance of MAVEN in extracting complementary multimodal representation from occluded and noisy data suggests its applicability on real-world datasets.

Amplifying action-context greater: image segmentation-guided intraoperative active bleeding detection

ABSTRACT The intraoperative active bleeding (iAB) detection model can be used for image-guided surgery and as a significant statistical index in predicting patient outcomes after surgery. However, detecting iAB is difficult due to the similarity between active and non-active bleeding or active bleeding in a small area. Using the spatial and temporal characteristics of the iAB area within frames simultaneously can overcome this. We propose a novel training method that can adequately fuse image segmentation and temporal action localisation models for effective iAB detection. The proposed active bleeding detection model has the following supervision process: First, annotate temporal localisation information for active bleeding that is relatively easy to annotate. Next, in the active bleeding section, where temporal localisation is annotated, spatial localisation information for selected frames is annotated and used as auxiliary information for active bleeding detection. We constructed a cross-validation set of 40 robotic subtotal gastrectomies and verified the ability of an active bleeding model guided by image segmentation information to bring improvements to the active bleeding recognition task. In addition, we applied performance evaluation for outcome analysis by measuring errors in iAB duration and counting in surgical videos for each algorithm. 1

A Hybrid Attention-Based Deep Neural Network for Simultaneous Multi-Sensor Pruning and Human Activity Recognition

With the popularity and development of Internet of Things (IoT) technology, human activity recognition using IoT devices such as wearable sensors can be implemented for various applications. Due to the complexity of activity recognition, multiple homogeneous or heterogeneous sensors are used to obtain excessive information in most wearable activity recognition systems. However, the increased number of sensors and the way of multichannel signal data bring huge challenges to human activity recognition tasks. How to select suitable sensor channels to balance the computational complexity and recognition accuracy has become a major issue. In this article, we extend the sparse group Lasso mechanism to human activity recognition tasks and propose a hybrid attention-based multi-sensor pruning and feature selection deep neural network, called HAP-DNN. This architecture is able to further perform feature selection on the basis of sensor pruning. HAP-DNN consists of three detachable modules: 1) a feature compression & reconstruction module for sensor feature information fusion and restoration; 2) a feature weight calculation module for calculating sensor channel weights and feature weights; and 3) a learning module for classification, which can be regarded as a filter feature selection method. Four public activity recognition data sets are used to verify our proposed architecture, and the experimental results show that HAP-DNN achieves the best classification performance with the least number of retained feature channels.

Multi-classification of Human Action Based on ResNet

With the development of machine learning and other related technologies, more and more excellent research on human action recognition and classification has been proposed, which significantly promotes the application of this technology in the actual situation. This paper mainly focuses on the characteristics of ResNet18, ResNet50, ResNet101, and ResNet152 in 15 human action recognition and classification tasks respectively. First, adjust and enhance the sample images of all training sets and test sets, and adjust the overall parameters according to the characteristics of the input image, so that all images can be normalized and high quality can be guaranteed at the same time; Then use the above four ResNet models to study and test with an epoch of 200, and explore the characteristics of the four models in this project by comparing the accuracy, loss, confusion matrix graphic, F1-score, and the number of epoch experienced in reaching the steady state of accuracy. The results show that in the overall effect, with the increase of the number of layers of the ResNet models, the accuracy changes in a positive correlation, and the epoch number experienced by the accuracy reaching the steady state changes in a negative correlation.

3D-Pruning: A Model Compression Framework for Efficient 3D Action Recognition

The existing end-to-end optimized 3D action recognition methods often suffer from high computational costs. Observing that different frames and different points in point cloud sequences often have different importance values for the 3D action recognition task, in this work, we propose a fully automatic model compression framework called 3D-Pruning (3DP) for efficient 3D action recognition. After performing model compression by using our 3DP framework, the compressed model can process different frames and different points in each frame by using different computational complexities based on their importance values, in which both the importance value and computational complexity for each frame/point can be automatically learned. Extensive experiments on five benchmark datasets demonstrate the effectiveness of our 3DP framework for model compression.

Multiscale echo self-attention memory network for multivariate time series classification

Recently, ESN has been applied to time series classification own to its high-dimensional random projection ability and training efficiency characteristic. The major drawback of applying ESN to time series classification is that ESN cannot capture long-term dependency information well. Therefore, the Multiscale Echo Self-Attention Memory Network (MESAMN) is proposed to address this issue. Specifically, the MESAMN consists of a memory encoder and a memory learner. In the memory encoder, multiple differently initialized ESNs are utilized for high-dimensional projection which is then followed by a self-attention mechanism to capture the long-term dependent features. A multiscale convolutional neural network is developed as the memory learner to learn local features using features extracted by the memory encoder. Experimental results show that the proposed MESAMN yields better performance on 18 multivariate time series classification tasks as well as three 3D skeleton-based action recognition tasks compared to existing models. Furthermore, the capacity for capturing long-term dependencies of the MESAMN is verified empirically.

DANet: Semi-supervised differentiated auxiliaries guided network for video action recognition

Video Action Recognition (ViAR) aims to identify the category of the human action observed in a given video. With the advent of Deep Learning (DL) techniques, noticeable performance breakthroughs have been achieved in this study. However, the success of most existing DL-based ViAR methods heavily relies on the existence of a large amount of annotated data, i.e., videos with corresponding action categories. In practice, obtaining such a desired number of annotations is often difficult due to expensive labeling costs, which may lead to significant performance degradation for these methods. To address this issue, we propose an end-to-end semi-supervised Differentiated Auxiliary guided Network (DANet) to best use a few annotated videos. Except for the common supervised learning on a few annotated videos, the DANet also involves the knowledge of multiple pre-trained auxiliary networks to optimize the ViAR network in a self-supervised way on the unannotated data by removing the annotations. Considering the tight connection between video action recognition and classical static image-based visual tasks, the abundant knowledge from the pre-trained static image-based models can be used for training the ViAR model. Specifically, the DANet is a two-branch architecture, which includes a target branch of the ViAR network, and an auxiliary branch of multiple auxiliary networks (i.e., referring to diverse off-the-shelf models of relevant image tasks). Given a limited number of annotated videos, we train the target ViAR network end-to-end in a semi-supervised way, namely, with both the supervised cross-entropy loss on annotated videos, and the per-auxiliary weighted self-supervised contrastive losses on the same videos but without using annotations. Besides, we further explore different weighted guidance of the auxiliary networks to the ViAR network to better reflect different relationships between the image-based models and the ViAR model. Finally, we conduct extensive experiments on several popular action recognition benchmarks in comparison with existing state-of-the-art methods, and the experimental results demonstrate the superiority of DANet over most of the compared methods. In particular, the DANet obviously suppresses state-of-the-art ViAR methods even with very fewer annotated videos.

Action Recognition Framework in Traffic Scene for Autonomous Driving System

For the autonomous driving system, accurately recognizing the actions of different roles in the traffic scene is the prerequisite for realizing this kind of human-vehicle information interaction. In this paper, we propose a complete framework based on 3D human pose estimation to recognize the actions of different roles on the road. The main objects recognized include traffic police, cyclists, and some passersby in need. We perform action recognition based on a dynamic adaptive graph convolutional network, which can realize the action recognition of objects based on 3D human pose. In addition to the action recognition module, we have optimized both the object detection module and the human pose estimation module in the framework so that the framework can handle multiple objects at the same time, which can be closer to the real traffic scene. To realize complex and changeable human action recognition, we built a multi-view camera system to collect responsible 3D human pose datasets containing traffic police gestures, cyclist gestures, and pedestrians’ body movements. In the experiments, compared to other state-of-the-art researches, the proposed framework can achieve comparable results with the same dataset. Satisfactory performance has also been obtained on the real data we collected, which can handle a variety of different action recognition tasks at the same time.

Action recognition based on discrete cosine transform by optical pixel-wise encoding

The framework provides a novel pipeline for action recognition. The action recognition task classifies the action label of the scene. High-speed cameras are commonly used to generate high frame-rate videos for capturing sufficient motion information. However, the data volume would be the bottleneck of the system. With the insight that the discrete cosine transform (DCT) of video signals reveals the motion information remarkably, instead of obtaining video data as with traditional cameras, the proposed method directly captures a DCT spectrum of video in a single shot through optical pixel-wise encoding. Considering that video signals are sparsely distributed in the DCT domain, a learning-based frequency selector is designed for pruning the trivial frequency channels of the spectrum. An opto-electronic neural network is designed for action recognition from a single coded spectrum. The optical encoder generates the DCT spectrum, and the rest of the network jointly optimizes the frequency selector and classification model simultaneously. Compared to conventional video-based action recognition methods, the proposed method achieves higher accuracy with less data, less communication bandwidth, and less computational burden. Both simulations and experiments demonstrate that the proposed method has superior action recognition performance. To the best of our knowledge, this is the first work investigating action recognition in the DCT domain.

A Perception-Action Assessment of the Functionality of Peripheral Vision in Expert and Novice Australian Footballers.

While widely acknowledged as being important for team-sport performance, the contribution of peripheral vision is poorly understood. This study aimed to better understand the role of far peripheral vision in team sport by exploring how domain experts and novices used far peripheral vision to support decision making and action control. Expert (n = 25) and novice (n = 23) Australian football players completed a perception-only task to assess the extent of their peripheral field. Next, they completed two sport-specific variations (response and recognition) of a "no-look" pass task that required passing a ball to a teammate who appeared in their far peripheral field. In the perception-only task, novices outperformed experts. However, in the sport-specific action response and recognition tasks, experts demonstrated superior performance as they responded to the stimulus farther from central vision and more accurately. Results demonstrate expertise effects for the use of far peripheral vision in sport.

Triple Cross-Domain Attention on Human Activity Recognition Using Wearable Sensors

Efficiently identifying activities of daily living (ADL) provides very important contextual information that is able to improve the effectiveness of various sports tracking and healthcare applications. Recently, attention mechanism that selectively focuses on time series signals has been widely adopted in sensor based human activity recognition (HAR), which can enhance interesting target activity and ignore irrelevant background activity. Several attention mechanisms have been investigated, which achieve remarkable performance in HAR scenario. Despite their success, prior these attention methods ignore the cross-interaction between different dimensions. In the paper, in order to avoid above shortcoming, we present a triplet cross-dimension attention for sensor-based activity recognition task, where three attention branches are built to capture the cross-interaction between sensor dimension, temporal dimension and channel dimension. The effectiveness of triplet attention method is validated through extensive experiments on four public HAR dataset namely UCI-HAR, PAMAP2, WISDM and UNIMIB-SHAR as well as the weakly labeled HAR dataset. Extensive experiments show consistent improvements in classification performance with various backbone models such as plain CNN and ResNet, demonstrating a good generality ability of the triplet attention. Visualization analysis is provided to support our conclusion, and actual implementation is evaluated on a Raspberry Pi platform.