Action Recognition In Videos Research Articles

Performance Art Video Action Management Oriented to 6G Wireless Transmission Technology

With the development of 5G communication networks, the development prospects, capability requirements and key technologies of the next-generation communication system 6G have become new hotspots. This article expands the coverage of the communication system through 6G, realizes the wireless transmission technology, builds the ideal of an intelligent mobile society everywhere, and analyzes the development trend of 6G technology, the challenges it faces, and the possible wireless transmission technology. Wireless transmission technology is widely used in various fields of production and life. Its various technical requirements and terminal energy limitations restrict the application and development of communications. This article focuses on the high-efficiency optimization scheme that combines 6G wireless transmission technology to improve the utilization efficiency of network resources. In addition, this thesis also discusses video action management in public performance art. Based on public performance art, the research on video action recognition can be summarized into three levels from simple to complex according to the content of the action, namely mobile vision, action vision and action vision. Video action vision is currently being studied. In the recognition process, most of the action frame information in the video is repetitive or has a low correlation with action recognition, which not only increases the complexity of calculation, but also affects the accuracy of action recognition. In the process of feature selection and understanding, the main methods include characterization fusion and research based on gesture features. Based on the introduction of 6G wireless transmission technology, this article introduces the technological development of 6G network, which promotes the rapid development of public performance art video action management.

See, move and hear: a local-to-global multi-modal interaction network for video action recognition

See, move and hear: a local-to-global multi-modal interaction network for video action recognition

Aeriform in-action: A novel dataset for human action recognition in aerial videos

Human actions being diverse in nature cannot be generalized, thus making it quite difficult to train a machine to recognize such diversified actions. This challenge is further compounded by the lack of availability of datasets for aerial surveillance, as collecting and annotating a large dataset is a formidable task. This paper aims to solve the problem of data scarcity by introducing a new dataset, Aeriform in-action for recognizing human actions from aerial videos. The proposed dataset consists of 32 high-resolution videos containing 13 action classes with 55,477 frames (without augmentation) and almost 400,000 annotations. It includes complex and aggressive actions such as kicking and punching, as well as drone signaling actions like waving and handshaking. The dataset also includes human-object interactions like carrying and reading. In addition to the dataset, this paper also presents a two-step deep learning framework for recognizing human actions based on the integration of human detection and action recognition module. The action recognition module adopts a modified version of the ResNeXt101 architecture (M-ResNext101) to recognize human actions in aerial videos. The performance of the proposed M-ResNext101 model is compared with 13 other deep learning models, and it outperforms all of them with an accuracy of 76.44% on the test data. The proposed dataset for human action recognition in aerial videos is available on https://surbhi-31.github.io/Aeriform-in-action/.

Video‐based real‐time assessment and diagnosis of autism spectrum disorder using deep neural networks

AbstractHuman action recognition (HAR) in untrimmed videos can make insightful predictions of human behaviour. Previous work on HAR‐included models trained on spatial and temporal annotations and could classify limited actions from trimmed videos. These methods reported limitations such as (1) performance degradation due to the lack of precision temporal regions proposal and (2) poor adaptability of the models in the clinical domain because of unrelated actions of interest. We propose an innovative method that could analyse untrimmed behavioural videos to recommend actions of interest leading to diagnostic and functional assessments for children with Autism Spectrum Disorder (ASD). Our method entails end‐to‐end behaviour action recognition (BAR) pipeline, including child detection, temporal action localization, and actions of interest identification and classification. The model trained on the data of 400 ASD children and 125 with other developmental delays (ODD) accurately identified ASD, ODD, and Neurotypical children with 79.7%, 77.2%, and 80.8% accuracy, respectively. The model's performance on an independent benchmark Self‐Stimulatory Behaviour Dataset (SSBD) reported top‐1 accuracy of 78.57% for combined localization with action recognition, significantly higher than the earlier reported outcomes.

Human Action Recognition

human-to-human interaction and interpersonal relations Human activity recognition plays a significant role. Because to identity of a person, their personality, and psychological state it provides information, it is difficult to extract. another person's activities is one of the main subjects of study of the scientific areas of computer vision The human ability to recognize and soft computing learning. robotics for human behavior characterization, require a multiple activity recognition system, many applications, including video surveillance systems, human computer interaction, and human activity recognition is an important research direction In video analysis, a large number of papers have been published on human activity recognition in video and image sequences In the past, including methods, systems, and quantitative evaluation of the performance of human activity recognition In this paper, we provide a comprehensive survey of the recent development of the techniques, for the video improve classification, interpretation, and performance. The experimental results show that our method can significantly.

Relational Action Bank with Semantic–Visual Attention for Few-Shot Action Recognition

Recently, few-shot learning has attracted significant attention in the field of video action recognition, owing to its data-efficient learning paradigm. Despite the encouraging progress, identifying ways to further improve the few-shot learning performance by exploring additional or auxiliary information for video action recognition remains an ongoing challenge. To address this problem, in this paper we make the first attempt to propose a relational action bank with semantic–visual attention for few-shot action recognition. Specifically, we introduce a relational action bank as the auxiliary library to assist the network in understanding the actions in novel classes. Meanwhile, the semantic–visual attention is devised to adaptively capture the connections to the foregone actions via both semantic correlation and visual similarity. We extensively evaluate our approach via two backbone models (ResNet-50 and C3D) on HMDB and Kinetics datasets, and demonstrate that the proposed model can obtain significantly better performance compared against state-of-the-art methods. Notably, our results demonstrate an average improvement of about 6.2% when compared to the second-best method on the Kinetics dataset.

SOR-TC: Self-attentive octave ResNet with temporal consistency for compressed video action recognition

Modeling and recognizing video activities from videos are key parts of many promising applications such as visual surveillance, human–computer interaction, and video summarization. However, current approaches mainly suffer from two issues: (a) Short-term local and global spatial features are not well represented. The spatial redundancy and dependency have not been well considered on CNN-based action recognition, which may result in a further increase in both memory and computation cost. (b) Long-term temporal consistency is not well captured. The action consistency across multiple clips has been ignored in video-level action recognition approaches. To address these two issues, we propose a Self-Attentive Octave ResNet with Temporal Consistency (SOR-TC) for compressed video action recognition to better capture the short-term and long-term features in video and improve the efficiency and effectiveness of action recognition. In addition, this paper introduces a consistency hypothesis that adjacent clips should predict similar actions. So the consistency loss function is designed to learn the correlation of clips. Finally, extensive experimental results on two benchmark datasets HMDB-51 and UCF-101 verify the effectiveness of our proposed method.

Semi-Supervised Action Recognition From Temporal Augmentation Using Curriculum Learning

Semi-supervised learning for video action recognition is a very challenging research area. Existing state-of-the-art methods perform data augmentation on the temporality of actions, which are combined with the mainstream consistency-based semi-supervised learning framework FixMatch for action recognition. However, these approaches have the following limitations: (1) data augmentation based on video clips lacks coarse-grained and fine-grained representations of actions in temporal sequences, and the models have difficulty understanding synonymous representations of actions in different motion phases. (2) Pseudo labeling selection based on the constant thresholds lacks a “make-up curriculum” for difficult actions, that results in the low utilization of unlabeled data corresponding to difficult actions. To address the above shortcomings, we propose a semi-supervised action recognition via the temporal augmentation using curriculum learning (TACL) algorithm. Compared to previous works, TACL explores different representations of the same semantics of actions in temporal sequences for video and uses the idea of curriculum learning (CL) to reduce the difficulty of the model training process. First, for different action expressions with the same semantics, we designed the temporal action augmentation (TAA) for videos to obtain coarse-grained and fine-grained action expressions based on constant-velocity and hetero-velocity methods, respectively. Second, we construct a temporal signal to constrain the model such that fine-grained action expressions containing different movement phases have the same prediction results, and achieve action consistency learning (ACL) by combining the label and pseudo-label signals. Finally, we propose action curriculum pseudo labeling (ACPL), a loosely and strictly parallel dynamic threshold evaluation algorithm for selecting and labeling unlabeled data. We evaluate TACL on three standard public datasets: UCF101, HMDB51, and Kinetics. The combined experiments show that TACL significantly improves the accuracy of models trained on a small amount of labeled data and better evaluates the learning effects for different actions.

Vision-Based HAR in UAV Videos Using Histograms and Deep Learning Techniques.

Activity recognition in unmanned aerial vehicle (UAV) surveillance is addressed in various computer vision applications such as image retrieval, pose estimation, object detection, object detection in videos, object detection in still images, object detection in video frames, face recognition, and video action recognition. In the UAV-based surveillance technology, video segments captured from aerial vehicles make it challenging to recognize and distinguish human behavior. In this research, to recognize a single and multi-human activity using aerial data, a hybrid model of histogram of oriented gradient (HOG), mask-regional convolutional neural network (Mask-RCNN), and bidirectional long short-term memory (Bi-LSTM) is employed. The HOG algorithm extracts patterns, Mask-RCNN extracts feature maps from the raw aerial image data, and the Bi-LSTM network exploits the temporal relationship between the frames for the underlying action in the scene. This Bi-LSTM network reduces the error rate to the greatest extent due to its bidirectional process. This novel architecture generates enhanced segmentation by utilizing the histogram gradient-based instance segmentation and improves the accuracy of classifying human activities using the Bi-LSTM approach. Experimental outcomes demonstrate that the proposed model outperforms the other state-of-the-art models and has achieved 99.25% accuracy on the YouTube-Aerial dataset.

On Modality Bias Recognition and Reduction

Making each modality in multi-modal data contribute is of vital importance to learning a versatile multi-modal model. Existing methods, however, are often dominated by one or few of modalities during model training, resulting in sub-optimal performance. In this article, we refer to this problem as modality bias and attempt to study it in the context of multi-modal classification systematically and comprehensively. After stepping into several empirical analyses, we recognize that one modality affects the model prediction more just because this modality has a spurious correlation with instance labels. To primarily facilitate the evaluation on the modality bias problem, we construct two datasets, respectively, for the colored digit recognition and video action recognition tasks in line with the Out-of-Distribution (OoD) protocol. Collaborating with the benchmarks in the visual question answering task, we empirically justify the performance degradation of the existing methods on these OoD datasets, which serves as evidence to justify the modality bias learning. In addition, to overcome this problem, we propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned according to the training set statistics. Thereafter, we apply this method on 10 baselines in total to test its effectiveness. From the results on four datasets regarding the above three tasks, our method yields remarkable performance improvements compared with the baselines, demonstrating its superiority on reducing the modality bias problem.

Development of a Novel Lightweight CNN Model for Classification of Human Actions in UAV-Captured Videos

There has been increased attention paid to autonomous unmanned aerial vehicles (UAVs) recently because of their usage in several fields. Human action recognition (HAR) in UAV videos plays an important role in various real-life applications. Although HAR using UAV frames has not received much attention from researchers to date, it is still a significant area that needs further study because of its relevance for the development of efficient algorithms for autonomous drone surveillance. Current deep-learning models for HAR have limitations, such as large weight parameters and slow inference speeds, which make them unsuitable for practical applications that require fast and accurate detection of unusual human actions. In response to this problem, this paper presents a new deep-learning model based on depthwise separable convolutions that has been designed to be lightweight. Other parts of the HarNet model comprised convolutional, rectified linear unit, dropout, pooling, padding, and dense blocks. The effectiveness of the model has been tested using the publicly available UCF-ARG dataset. The proposed model, called HarNet, has enhanced the rate of successful classification. Each unit of frame data was pre-processed one by one by different computer vision methods before it was incorporated into the HarNet model. The proposed model, which has a compact architecture with just 2.2 million parameters, obtained a 96.15% success rate in classification, outperforming the MobileNet, Xception, DenseNet201, Inception-ResNetV2, VGG-16, and VGG-19 models on the same dataset. The proposed model had numerous key advantages, including low complexity, a small number of parameters, and high classification performance. The outcomes of this paper showed that the model’s performance was superior to that of other models that used the UCF-ARG dataset.

Multi-level channel attention excitation network for human action recognition in videos

Channel attention mechanism has continuously attracted strong interests and shown great potential in enhancing the performance of deep CNNs. However, when applied to video-based human action recognition task, most existing methods generally learn channel attention at frame level, which ignores the temporal dependencies and may limit the recognition performance. In this paper, we propose a novel multi-level channel attention excitation (MCAE) module to model the temporal-related channel attention at both frame and video levels. Specifically, based on video convolutional feature maps, frame-level channel attention (FCA) is generated by exploring time-channel correlations, and video-level channel attention (VCA) is generated by aggregating global motion variations. MCAE firstly recalibrates video feature responses with frame-wise FCA, and then activates the motion-sensitive channel features with motion-aware VCA. MCAE module learns the channel discriminability from multiple levels and can act as a guidance to facilitate efficient spatiotemporal feature modeling in activated motion-sensitive channels. It can be flexibly embedded into 2D networks with very limited extra computation cost to construct MCAE-Net, which effectively enhances the spatiotemporal representation of 2D models for video action recognition task Extensive experiments on five human action datasets show that our method achieves superior or very competitive performance compared with the state-of-the-arts, which demonstrates the effectiveness of the proposed method for improving the performance of human action recognition.

Two-Level Attention Module Based on Spurious-3D Residual Networks for Human Action Recognition.

In recent years, deep learning techniques have excelled in video action recognition. However, currently commonly used video action recognition models minimize the importance of different video frames and spatial regions within some specific frames when performing action recognition, which makes it difficult for the models to adequately extract spatiotemporal features from the video data. In this paper, an action recognition method based on improved residual convolutional neural networks (CNNs) for video frames and spatial attention modules is proposed to address this problem. The network can guide what and where to emphasize or suppress with essentially little computational cost using the video frame attention module and the spatial attention module. It also employs a two-level attention module to emphasize feature information along the temporal and spatial dimensions, respectively, highlighting the more important frames in the overall video sequence and the more important spatial regions in some specific frames. Specifically, we create the video frame and spatial attention map by successively adding the video frame attention module and the spatial attention module to aggregate the spatial and temporal dimensions of the intermediate feature maps of the CNNs to obtain different feature descriptors, thus directing the network to focus more on important video frames and more contributing spatial regions. The experimental results further show that the network performs well on the UCF-101 and HMDB-51 datasets.

WLiT: Windows and Linear Transformer for Video Action Recognition.

The emergence of Transformer has led to the rapid development of video understanding, but it also brings the problem of high computational complexity. Previously, there were methods to divide the feature maps into windows along the spatiotemporal dimensions and then calculate the attention. There are also methods to perform down-sampling during attention computation to reduce the spatiotemporal resolution of features. Although the complexity is effectively reduced, there is still room for further optimization. Thus, we present the Windows and Linear Transformer (WLiT) for efficient video action recognition, by combining Spatial-Windows attention with Linear attention. We first divide the feature maps into multiple windows along the spatial dimensions and calculate the attention separately inside the windows. Therefore, our model further reduces the computational complexity compared with previous methods. However, the perceptual field of Spatial-Windows attention is small, and global spatiotemporal information cannot be obtained. To address this problem, we then calculate Linear attention along the channel dimension so that the model can capture complete spatiotemporal information. Our method achieves better recognition accuracy with less computational complexity through this mechanism. We conduct extensive experiments on four public datasets, namely Something-Something V2 (SSV2), Kinetics400 (K400), UCF101, and HMDB51. On the SSV2 dataset, our method reduces the computational complexity by 28% and improves the recognition accuracy by 1.6% compared to the State-Of-The-Art (SOTA) method. On the K400 and two other datasets, our method achieves SOTA-level accuracy while reducing the complexity by about 49%.

Heterogeneous recurrent spiking neural network for spatio-temporal classification.

Spiking Neural Networks are often touted as brain-inspired learning models for the third wave of Artificial Intelligence. Although recent SNNs trained with supervised backpropagation show classification accuracy comparable to deep networks, the performance of unsupervised learning-based SNNs remains much lower. This paper presents a heterogeneous recurrent spiking neural network (HRSNN) with unsupervised learning for spatio-temporal classification of video activity recognition tasks on RGB (KTH, UCF11, UCF101) and event-based datasets (DVS128 Gesture). We observed an accuracy of 94.32% for the KTH dataset, 79.58% and 77.53% for the UCF11 and UCF101 datasets, respectively, and an accuracy of 96.54% on the event-based DVS Gesture dataset using the novel unsupervised HRSNN model. The key novelty of the HRSNN is that the recurrent layer in HRSNN consists of heterogeneous neurons with varying firing/relaxation dynamics, and they are trained via heterogeneous spike-time-dependent-plasticity (STDP) with varying learning dynamics for each synapse. We show that this novel combination of heterogeneity in architecture and learning method outperforms current homogeneous spiking neural networks. We further show that HRSNN can achieve similar performance to state-of-the-art backpropagation trained supervised SNN, but with less computation (fewer neurons and sparse connection) and less training data.

Microcomb-based integrated photonic processing unit

The emergence of parallel convolution-operation technology has substantially powered the complexity and functionality of optical neural networks (ONN) by harnessing the dimension of optical wavelength. However, this advanced architecture faces remarkable challenges in high-level integration and on-chip operation. In this work, convolution based on time-wavelength plane stretching approach is implemented on a microcomb-driven chip-based photonic processing unit (PPU). To support the operation of this processing unit, we develop a dedicated control and operation protocol, leading to a record high weight precision of 9 bits. Moreover, the compact architecture and high data loading speed enable a preeminent photonic-core compute density of over 1 trillion of operations per second per square millimeter (TOPS mm−2). Two proof-of-concept experiments are demonstrated, including image edge detection and handwritten digit recognition, showing comparable processing capability compared to that of a digital computer. Due to the advanced performance and the great scalability, this parallel photonic processing unit can potentially revolutionize sophisticated artificial intelligence tasks including autonomous driving, video action recognition and image reconstruction.

Toward Deep Transfer Learning for Realistic Activity Recognition in Videos

Today, videos have become popular on the internet and specified in social network and media platforms such as Youbue, Ticktok, and Vimeo. Video understanding has attracted much attention in the research community in recent years. Automatically recognizing human activity in wild videos is a trending research topic with a wide range of applications in advertising, smarthome, and surveillance camera systems. Deep convolutional neural networks have become a new de facto visual recognition problem. It achieved much success in image recognition problems that are leveraged from the ImageNet dataset. Many researchers have applied CNNs to the video domain, but the results in realistic video still have many challenges, and the recognition rate is not as expected. Because the realistic activity in video is extremely small, we cannot train a large deep convolutional network to achieve good performance. In this work, we answer the question “how could we transfer the visual features from the ImageNet dataset into video for activity recognition tasks?”. We propose an approach based on the pretrained models from ImageNet for activity recognition in video that is based on transforming video into an image map for temporal information and a frame-based description for spatial information. We test our proposed approach on three datasets, UCF11, UCF50 and UCF101, and it achieves an accuracy of over 95% for the backbone of ResNet and over 88% for the backbone of MobileNet. The experimental results show that our proposed method is robust and efficient in wild video datasets.

DTCM: Joint Optimization of Dark Enhancement and Action Recognition in Videos.

Recognizing human actions in dark videos is a useful yet challenging visual task in reality. Existing augmentation-based methods separate action recognition and dark enhancement in a two-stage pipeline, which leads to inconsistently learning of temporal representation for action recognition. To address this issue, we propose a novel end-to-end framework termed Dark Temporal Consistency Model (DTCM), which is able to jointly optimize dark enhancement and action recognition, and force the temporal consistency to guide downstream dark feature learning. Specifically, DTCM cascades the action classification head with the dark augmentation network to perform dark video action recognition in a one-stage pipeline. Our explored spatio-temporal consistency loss, which utilizes the RGB-Difference of dark video frames to encourage temporal coherence of the enhanced video frames, is effective for boosting spatio-temporal representation learning. Extensive experiments demonstrated that our DTCM has remarkable performance: 1) Competitive accuracy, which outperforms the state-of-the-arts on the ARID dataset by 2.32% and the UAVHuman-Fisheye dataset by 4.19% in accuracy, respectively; 2) High efficiency, which surpasses the current most advanced method (Chen et al., 2021) with only 6.4% GFLOPs and 71.3% number of parameters; 3) Strong generalization, which can be used in various action recognition methods (e.g., TSM, I3D, 3D-ResNext-101, Video-Swin) to promote their performance significantly.

ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos

ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos

Multimodal Fusion for Audio-Image and Video Action Recognition

Multimodal Fusion for Audio-Image and Video Action Recognition