Network For Action Recognition Research Articles

Towards an Action Recognition Framework for Endovascular Surgery.

Objective knowledge about instrument manoeuvres in endovascular surgery is essential for evaluating surgical skills and developing advanced technologies for cathlab routines. To the recent day, endovascular navigation has been exclusively assessed in laboratory scenarios. By contrast, information contained in available fluoroscopy data from clinical cases has been disregarded. In this work, we pioneer a learning-based framework for motion activity recognition in fluoroscopy sequences. The architecture is composed of two networks for instrument segmentation and action recognition. In this preliminary study, we demonstrate feasibility of recognising instrument manoeuvres automatically in our ex vivo datasets.Clinical relevance-The proposed framework contributes to image-based and automated assessment of endovascular tasks. This facilitates robotic control development, surgical education, and smart clinical documentation.

Multiple temporal scale aggregation graph convolutional network for skeleton-based action recognition

Skeleton-based human action recognition is attracting increasing attention and is widely applied in virtual reality, human–computer interaction system and other cases. Nevertheless, the performance of recent methods on actions with similar appearance features is still barely satisfactory due to their inherent weakness in modeling the discriminative temporal dynamics. Besides, previous methods have limitation in mining the global information of action. To this end, a multiple temporal scale aggregation graph convolutional network is proposed. Firstly, taking advantage of the varying temporal resolutions offered by different layers in the graph convolutional network, we develop a multiple temporal scale aggregation module to extract discriminative temporal feature. Secondly, a new skeleton feature representation method termed as relative joint across frames is proposed, which provides more global structure clue than the absolute coordinates. Furthermore, we propose a five-stream structure that comprehensively models complementary features and eventually achieves a significant performance boost. In the empirical experiments, our method shows an improvement of 2.38% and 4.08% compared to the baseline method on the cross-subject evaluation benchmark of NTU-RGB+D 60 and NTU-RGB+D 120, respectively.

Enhanced Adjacency Matrix-Based Lightweight Graph Convolution Network for Action Recognition.

Graph convolutional networks (GCNs), which extend convolutional neural networks (CNNs) to non-Euclidean structures, have been utilized to promote skeleton-based human action recognition research and have made substantial progress in doing so. However, there are still some challenges in the construction of recognition models based on GCNs. In this paper, we propose an enhanced adjacency matrix-based graph convolutional network with a combinatorial attention mechanism (CA-EAMGCN) for skeleton-based action recognition. Firstly, an enhanced adjacency matrix is constructed to expand the model's perceptive field of global node features. Secondly, a feature selection fusion module (FSFM) is designed to provide an optimal fusion ratio for multiple input features of the model. Finally, a combinatorial attention mechanism is devised. Specifically, our spatial-temporal (ST) attention module and limb attention module (LAM) are integrated into a multi-input branch and a mainstream network of the proposed model, respectively. Extensive experiments on three large-scale datasets, namely the NTU RGB+D 60, NTU RGB+D 120 and UAV-Human datasets, show that the proposed model takes into account both requirements of light weight and recognition accuracy. This demonstrates the effectiveness of our method.

Transforming spatio-temporal self-attention using action embedding for skeleton-based action recognition

Over the past few years, skeleton-based action recognition has attracted great success because the skeleton data is immune to illumination variation, view-point variation, background clutter, scaling, and camera motion. However, effective modeling of the latent information of skeleton data is still a challenging problem. Therefore, in this paper, we propose a novel idea of action embedding with a self-attention Transformer network for skeleton-based action recognition. Our proposed technology mainly comprises of two modules as, (i) action embedding and (ii) self-attention Transformer. The action embedding encodes the relationship between corresponding body joints (e.g., joints of both hands move together for performing clapping action) and thus captures the spatial features of joints. Meanwhile, temporal features and dependencies of body joints are modeled using Transformer architecture. Our method works in a single-stream (end-to-end) fashion, where multiple-layer perceptron (MLP) is used for classification. We carry out an ablation study and evaluate the performance of our model on a small-scale SYSU-3D dataset and large-scale NTU-RGB+D and NTU-RGB+D 120 datasets where the results establish that our method performs better than other state-of-the-art architectures.

Fast 3D-graph convolutional networks for skeleton-based action recognition

Research on human action recognition based on skeletons has received much attention. But most of the research focuses on improving the model’s generalization ability, while ignoring significant efficiency issues. This leads to developing heavy models with poor scalability and cost-effectiveness in practical use. This paper, we investigate the under-studied but practically critical recognition model efficiency problem. To this end, we present a new Fast Recognition Distillation (FRD) model learning strategy. Specifically, FRD trains a lightweight recognition neural network structure that can be quickly executed at a low computational cost. It can be achieved by effectively disseminating the identification probability information of the teacher network to the lightweight network. We call the probability information of the teacher network as soft-target, and FRD can learn more potential information from soft-target. In addition, we also used a particular loss function for soft-target. Through the FRD network, while basically maintaining the recognition accuracy, we minimized the network structure. Extensive experiments on the two large-scale datasets, NTU-RGBD and Kinetics-Skeleton, demonstrate that our model (FRD) is more lightweight and refined than others. Therefore, our model FRD is efficient.

Adaptive Multi-Scale Difference Graph Convolution Network for Skeleton-Based Action Recognition

Graph convolutional networks (GCNs) have obtained remarkable performance in skeleton-based action recognition. However, previous approaches fail to capture the implicit correlations between joints and handle actions across varying time intervals. To address these problems, we propose an adaptive multi-scale difference graph convolution Network (AMD-GCN), which comprises an adaptive spatial graph convolution module (ASGC) and a multi-scale temporal difference convolution module (MTDC). The first module is capable of acquiring data-dependent and channel-wise graphs that are adaptable to both samples and channels. The second module employs the multi-scale approach to model temporal information across a range of time scales. Additionally, the MTDC incorporates an attention-enhanced module and difference convolution to accentuate significant channels and enhance temporal features, respectively. Finally, we propose a multi-stream framework for integrating diverse skeletal modalities to achieve superior performance. Our AMD-GCN approach was extensively tested and proven to outperform the current state-of-the-art methods on three widely recognized benchmarks: the NTU-RGB+D, NTU-RGB+D 120, and Kinetics Skeleton datasets.

Two-Streams: Dark and Light Networks with Graph Convolution for Action Recognition from Dark Videos (Student Abstract)

In this article, we propose a two-stream action recognition technique for recognizing human actions from dark videos. The proposed action recognition network consists of an image enhancement network with Self-Calibrated Illumination (SCI) module, followed by a two-stream action recognition network. We have used R(2+1)D as a feature extractor for both streams with shared weights. Graph Convolutional Network (GCN), a temporal graph encoder is utilized to enhance the obtained features which are then further fed to a classification head to recognize the actions in a video. The experimental results are presented on the recent benchmark ``ARID" dark-video database.

Spatial–Temporal gated graph attention network for skeleton-based action recognition

Spatial–Temporal gated graph attention network for skeleton-based action recognition

Viewpoint guided multi-stream neural network for skeleton action recognition

Viewpoint guided multi-stream neural network for skeleton action recognition

Multi-stream Global–Local Motion Fusion Network for skeleton-based action recognition

Skeleton-based action recognition is widely used in varied areas such as human–machine interaction and virtual reality. Benefit from the powerful expression ability to depict structural data, graph convolutional networks (GCNs) have been developed to address this task by modeling the human body skeletons as spatial–temporal graphs. However, most existing GCN-based methods usually ignore the diversity of the motion information between channels of the input feature. And how to enhance the ability to capture the long-term global correlations in spatial and temporal dimensions is also a fundamental challenge. In this work, we propose a novel multi-stream framework Global–Local Motion Fusion Network (GLMFN), which integrates the global and local motion information of spatial–temporal dimensions. Specifically, we design a grouping graph convolution module to enforce the ability to aggregate local spatial motion information. Besides, to learn richer semantic features, we propose two modules based on the self-attention operator: a spatial self-attention module and a temporal self-attention module. The former is responsible for extracting spatial long-term motion relationships, while the latter aims to capture temporal long-term motion relationships. Moreover, we present a multi-stream fusion strategy with a series of treatments for body joints to achieve a better recognition effect. To validate the efficacy and efficiency of the proposed model, we perform exhaustive experiments on the NTU-RGBD dataset and NTU-RGBD-120 dataset, and our method achieves the state-of-the-art performance on both datasets.

TFC-GCN: Lightweight Temporal Feature Cross-Extraction Graph Convolutional Network for Skeleton-Based Action Recognition

For skeleton-based action recognition, graph convolutional networks (GCN) have absolute advantages. Existing state-of-the-art (SOTA) methods tended to focus on extracting and identifying features from all bones and joints. However, they ignored many new input features which could be discovered. Moreover, many GCN-based action recognition models did not pay sufficient attention to the extraction of temporal features. In addition, most models had swollen structures due to too many parameters. In order to solve the problems mentioned above, a temporal feature cross-extraction graph convolutional network (TFC-GCN) is proposed, which has a small number of parameters. Firstly, we propose the feature extraction strategy of the relative displacements of joints, which is fitted for the relative displacement between its previous and subsequent frames. Then, TFC-GCN uses a temporal feature cross-extraction block with gated information filtering to excavate high-level representations for human actions. Finally, we propose a stitching spatial–temporal attention (SST-Att) block for different joints to be given different weights so as to obtain favorable results for classification. FLOPs and the number of parameters of TFC-GCN reach 1.90 G and 0.18 M, respectively. The superiority has been verified on three large-scale public datasets, namely NTU RGB + D60, NTU RGB + D120 and UAV-Human.

MuAt-Va: Multi-Attention and Video-Auxiliary Network for Device-Free Action Recognition

With the growing popularity of internet of things (IoT) systems, device-free action recognition begins to attract extensive attentions due to its friendly feasibility in broad applications such as human-computer interaction and smart elderly care. Considering abundant information in the vision modality, existing methods adopt the cross-model methods for performance enhancement. However, the dependency of synchronous multimodal data in the collection and recognition stage brings into the vision weaknesses such as sensitivity to occlusion and privacy invasion. In this paper, we integrate multi-attention structure and auxiliary video information into a novel end-to-end deep learning framework named MuAt-Va, in which video soft labels learned in advance are utilized to teach the multi-attention WiFi feature training process without vision information involved during test. Specifically, in order to enlarge the application scope and reduce the data cost, we beforehand acquire videos under a satisfactory condition only once, and then leverage teacher-student mechanism to guide the WiFi stream. Instead of straightforwardly concatenating multi-antenna channel state information (CSI) from homogeneous wireless signals as previous works, we design a CSI subcarrier-wise, temporal-wise and view-wise attention module to assign different weights on the basis of data characteristics for the sensing task. Our experiments with multiple subjects data in two scenes demonstrate that MuAt-Va can accurately recognize human actions with more superior performances.

Multi-stream ternary enhanced graph convolutional network for skeleton-based action recognition

Multi-stream ternary enhanced graph convolutional network for skeleton-based action recognition

DL-PCN: Differential learning and parallel convolutional network for action recognition

Graph Convolution Network (GCN) algorithms have greatly improved the accuracy of skeleton-based human action recognition. GCN can utilize the spatial information between skeletal joints in subsequent frames better than other deep learning algorithms, which is beneficial for achieving high accuracy. However, the traditional GCN algorithms consume lots of computation for the stack of multiple primary GCN layers. Aiming at solving the problem, we introduce a lightweight network, a Differential Learning and Parallel Convolutional Networks (DL-PCN), whose key modules are Differential Learning (DLM) and the Parallel Convolutional Network (PCN). DLM features a feedforward connection, which carries the error information of GCN modules with the same structure, where GCN and CNN modules directly extract the original information from the input data, making the spatiotemporal information extracted by these modules more complete than that of GCN and CNN tandem structure. PCN comprises GCN and Convolution Neural Network (CNN) in parallel. Our network achieves comparable performance on the NTU RGB+D 60 dataset, the NTU RGB+D 120 dataset and the Northwestern-UCLA dataset while considering both accuracy and calculation parameters.

Action Recognition Network Based on Local Spatiotemporal Features and Global Temporal Excitation

Temporal modeling is a key problem in action recognition, and it remains difficult to accurately model temporal information of videos. In this paper, we present a local spatiotemporal extraction module (LSTE) and a channel time excitation module (CTE), which are specially designed to accurately model temporal information in video sequences. The LSTE module first obtains difference features by computing the pixel-wise differences between adjacent frames within each video segment and then obtains local motion features by stressing the effect of the feature channels sensitive to difference information. The local motion features are merged with the spatial features to represent local spatiotemporal features of each segment. The CTE module adaptively excites time-sensitive channels by modeling the interdependencies of channels in terms of time to enhance the global temporal information. Further, the above two modules are embedded into the existing 2DCNN baseline methods to build an action recognition network based on local spatiotemporal features and global temporal excitation (LSCT). We conduct experiments on the temporal-dependent Something-Something V1 and V2 datasets. We compare the recognition results with those obtained by the current methods, which proves the effectiveness of our methods.

Fine-Tuned Temporal Dense Sampling with 1D Convolutional Neural Network for Human Action Recognition.

Human action recognition is a constantly evolving field that is driven by numerous applications. In recent years, significant progress has been made in this area due to the development of advanced representation learning techniques. Despite this progress, human action recognition still poses significant challenges, particularly due to the unpredictable variations in the visual appearance of an image sequence. To address these challenges, we propose the fine-tuned temporal dense sampling with 1D convolutional neural network (FTDS-1DConvNet). Our method involves the use of temporal segmentation and temporal dense sampling, which help to capture the most important features of a human action video. First, the human action video is partitioned into segments through temporal segmentation. Each segment is then processed through a fine-tuned Inception-ResNet-V2 model, where max pooling is performed along the temporal axis to encode the most significant features as a fixed-length representation. This representation is then fed into a 1DConvNet for further representation learning and classification. The experiments on UCF101 and HMDB51 demonstrate that the proposed FTDS-1DConvNet outperforms the state-of-the-art methods, with a classification accuracy of 88.43% on the UCF101 dataset and 56.23% on the HMDB51 dataset.

Spatial–Temporal Self-Attention Enhanced Graph Convolutional Networks for Fitness Yoga Action Recognition

Fitness yoga is now a popular form of national fitness and sportive physical therapy. At present, Microsoft Kinect, a depth sensor, and other applications are widely used to monitor and guide yoga performance, but they are inconvenient to use and still a little expensive. To solve these problems, we propose spatial–temporal self-attention enhanced graph convolutional networks (STSAE-GCNs) that can analyze RGB yoga video data captured by cameras or smartphones. In the STSAE-GCN, we build a spatial–temporal self-attention module (STSAM), which can effectively enhance the spatial–temporal expression ability of the model and improve the performance of the proposed model. The STSAM has the characteristics of plug-and-play so that it can be applied in other skeleton-based action recognition methods and improve their performance. To prove the effectiveness of the proposed model in recognizing fitness yoga actions, we collected 960 fitness yoga action video clips in 10 action classes and built the dataset Yoga10. The recognition accuracy of the model on Yoga10 achieves 93.83%, outperforming the state-of-the-art methods, which proves that this model can better recognize fitness yoga actions and help students learn fitness yoga independently.

Multi-scale spatial–temporal convolutional neural network for skeleton-based action recognition

Multi-scale spatial–temporal convolutional neural network for skeleton-based action recognition

Skeleton-based multi-stream adaptive-attentional sub-graph convolution network for action recognition

Skeleton-based multi-stream adaptive-attentional sub-graph convolution network for action recognition

A Lightweight Attentional Shift Graph Convolutional Network for Skeleton-Based Action Recognition

In the field of skeleton-based human behavior recognition, graph convolutional neural networks have made remarkable achievements. However, high precision networks are often accompanied by numerous parameters and computational cost, and their application in mobile devices has considerable limitations. Aiming at the problem of excessive spatiotemporal complexity of high-accuracy methods, this paper further analyzes the lightweight human action recognition model and proposes a lightweight architecture attentional shift graph convolutional network. There are three main improvements in this model. Firstly, shift convolution is a lightweight convolution method that can be combined with graph convolution to effectively reduce its complexity. At the same time, a shallow architecture for multi-stream early fusion is designed to reduce the network scale by merging multi-stream networks and reducing the number of network layers. In addition, the efficient channel attention module is introduced into the model to capture the underlying characteristic information in the channel domain. Experiments are conducted on the three existing skeleton datasets, NTU RGB+D, NTU-120 RGB+D, and Northwestern-UCLA. Results demonstrate that the proposed model is not only competitive in accuracy, but also outperforms current mainstream methods in parameter count and computational cost, and supports running in some devices with limited computing and storage resources.