Video Recognition Research Articles

Imperceptible Adversarial Attack With Multigranular Spatiotemporal Attention for Video Action Recognition

In recent years, the application of video Internet of Things (IoT) in various cities and public places has brought unprecedented opportunities to the security field and achieved great success. However, the latest research shows that video recognition models are also vulnerable to adversarial examples, but adversarial examples based on physical attacks are easily detected by humans, making it difficult to pass human review. To address this problem, in this paper, we propose to introduce a novel Multi-granular Spatio-temporal Attention Network (MSANet), which can attack the video action recognition models imperceptibly. Specifically, to exploit video motion information more effectively and to reduce the detectability of attack perturbations, we design a multiplexed spatio-temporal attention module to select and enhance spatial regions and temporal frames at coarse-grained and fine-grained levels respectively, thus maintaining a certain degree of smoothness while reducing the perturbation size and avoiding attacking overfitting. In addition, our proposed MSANet achieves imperceptible perturbations to video sequences through alternate iterative optimization combined with the PGD attack mechanism. The extended experimental results on two different models (e.g., TDN and TSM) and two widely-used datasets (HMDB-51 hm and UCF-101 ucf), compared to the state-of-the-art model, demonstrate the effectiveness of our devised video action recognition attack approach.

Behavior Recognition of Squid Jigger Based on Deep Learning

In recent years, with the development of pelagic fishing, the working environment and monitoring of crew (squid jigger) members have become increasingly important. However, traditional methods of pelagic human observers suffer from high costs, low coverage, poor timeliness, and susceptibility to subjective factors. In contrast, the Electronic Monitoring System (EMS) has advantages such as continuous operation under various weather conditions; more objective, transparent, and efficient data; and less interference with fishing operations. This paper shows how the 3DCNN model, LSTM+ResNet model, and TimeSformer model are applied to video-classification tasks, and for the first time, they are applied to an EMS. In addition, this paper tests and compares the application effects of the three models on video classification, and discusses the advantages and challenges of using them for video recognition. Through experiments, we obtained the accuracy and relevant indicators of video recognition using different models. The research results show that when NUM_FRAMES is set to 8, the LSTM+ResNet-50 model has the best performance, with an accuracy of 88.47%, an F1 score of 0.8881, and an map score of 0.8133. Analyzing the EMS for pelagic fishing can improve China’s performance level and management efficiency in pelagic fishing, and promote the development of the fishery knowledge service system and smart fishery engineering.

HUMAN FACE RECOGNITION ON IMAGE VIDEO CONFERENCE APPLICATION USING SIAMESE NETWORK WITH SKIP CONNECTION SMALLER VGG MODEL

Attendance recording is needed to find out someone's attendance at a meeting or meeting. These meetings are sometimes conducted online through the video conferencing application. Recording attendance at online meetings is using an online form that is distributed via chat. There are several problems such as chats piling up and meeting participants arriving late so they cannot access the form link. Therefore, facial recognition can be used to record attendance using screenshots as an attendance record with the aim of helping to facilitate attendance recording through video conferencing applications using computer vision technology. This study proposes a method of using the Siamese network with the Smaller VGG skip connection model to improve human face recognition in video conferencing application images. Has validation accuracy results in the training phase of 98%, precision of 98%, and recall of 98%. For the similarity phase where the model is applied to the Siamese network, the accuracy is 95%, the precision is 53%, and the recall is 78%. Then the model is applied to the pipeline system with the YOLO-face model to classify the results of face detection from Yolo with the faces in the database so that the model does not need to be retrained if there are new faces, it only needs to add facial images to the database to be compared with the query image..

Transfer learning-based security protection framework for smart exercise monitoring

Sports exercise is very important for both physical and mental health, but improper exercise or equipment use may cause some security challenges. To address this issue, facing Internet of Things (IoT), currently, some video recognition algorithm-based smart security monitoring systems have been designed. However, existing video recognition algorithms usually assume that the data comes from the same collection device or follows the same distribution, resulting in ineffective handling of cross-camera or cross-device recognition problems. In this vein, this paper designed a transfer learning-based smart exercise monitoring security protection system, and proposed a new transfer learning-based video recognition framework, which consists of backbone network module, style transfer module, video feature aggregation module three parts and using this framework, two different models can be trained based on video face recognition dataset and video action recognition dataset, respectively, for identity recognition and action recognition. Experimental results show that the proposed framework can effectively handle video face recognition and video action recognition problems, which also demonstrates that our designed smart exercise monitoring security protection system can meet actual task requirements.

Emotion and Gesture Guided Action Recognition in Videos Using Supervised Deep Networks

Emotion and Gesture Guided Action Recognition in Videos Using Supervised Deep Networks

Exploring Spatio–Temporal Graph Convolution for Video-Based Human–Object Interaction Recognition

Video-based human-object interaction recognition is a challenging task since the state of objects as well as their correlations change constantly in the video. Existing methods mainly use 3DCNN or use separate components ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., GCN + RNN) to model the spatial correlation or the temporal correlation respectively, but ignore modeling spatio-temporal correlations simultaneously and long-term temporal dynamics of objects. In this paper, we propose a novel model, named Spatio-Temporal Interaction Graph Parsing Networks (STIGPN), for human-object interaction recognition in videos. STIGPN captures both spatial and temporal correlations simultaneously and thus can capture intra-frame and inter-frame dependencies efficiently and effectively. To model long-term temporal dynamics of objects, we introduce spatio-temporal feature enhancement, which can improve the detection of the salient human-object interaction pairs. We explore three types of spatio-temporal graph convolutions to simultaneously capture the spatio-temporal correlations and assess their effectiveness as the basic building block of STIGPN. Extensive experiments on CAD-120, Something-Else and Charades datasets show that our proposed solution leads to competitive results compared with the state-of-the-art methods. Code for STIGPN is available at: https://github.com/NingWang2049/STIGPN2.

GESTURE RECOGNITION OF MACHINE LEARNING AND CONVOLUTIONAL NEURAL NETWORK METHODS FOR KAZAKH SIGN LANGUAGE

Recently, there has been a growing interest in machine learning and neural networks among the public, largely due to advancements in technology which have led to improved methods of computer recognition of objects, sounds, texts, and other data types. As a result, human-computer interactions are becoming more natural and comprehensible to the average person. The progress in computer vision has enabled the use of increasingly sophisticated models for object recognition in images and videos, which can also be applied to recognize hand gestures. In this research, popular hand gesture recognition models, such as the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Support Vector Machine (SVM) were examined. These models vary in their approaches, processing time, and training data size. The important feature of this research work is the use of various machine learning algorithms and methods such as CNN, LSTM, and SVM. Experiments showed different results when training neural networks for sign language recognition in the Kazakh sign language based on the dactyl alphabet. This article provides a detailed description of each method, their respective purposes, and effectiveness in terms of performance and training. Numerous experimental results were recorded in a table, demonstrating the accuracy of recognizing each gesture. Additionally, specific hand gestures were isolated for testing in front of the camera to recognize the gesture and display the result on the screen. An important feature was the use of mathematical formulas and functions to explain the working principle of the machine learning algorithm, as well as the logical scheme and structure of the LSTM algorithm.

EMO-MoviNet: Enhancing Action Recognition in Videos with EvoNorm, Mish Activation, and Optimal Frame Selection for Efficient Mobile Deployment.

The primary goal of this study is to develop a deep neural network for action recognition that enhances accuracy and minimizes computational costs. In this regard, we propose a modified EMO-MoviNet-A2* architecture that integrates Evolving Normalization (EvoNorm), Mish activation, and optimal frame selection to improve the accuracy and efficiency of action recognition tasks in videos. The asterisk notation indicates that this model also incorporates the stream buffer concept. The Mobile Video Network (MoviNet) is a member of the memory-efficient architectures discovered through Neural Architecture Search (NAS), which balances accuracy and efficiency by integrating spatial, temporal, and spatio-temporal operations. Our research implements the MoviNet model on the UCF101 and HMDB51 datasets, pre-trained on the kinetics dataset. Upon implementation on the UCF101 dataset, a generalization gap was observed, with the model performing better on the training set than on the testing set. To address this issue, we replaced batch normalization with EvoNorm, which unifies normalization and activation functions. Another area that required improvement was key-frame selection. We also developed a novel technique called Optimal Frame Selection (OFS) to identify key-frames within videos more effectively than random or densely frame selection methods. Combining OFS with Mish nonlinearity resulted in a 0.8-1% improvement in accuracy in our UCF101 20-classes experiment. The EMO-MoviNet-A2* model consumes 86% fewer FLOPs and approximately 90% fewer parameters on the UCF101 dataset, with a trade-off of 1-2% accuracy. Additionally, it achieves 5-7% higher accuracy on the HMDB51 dataset while requiring seven times fewer FLOPs and ten times fewer parameters compared to the reference model, Motion-Augmented RGB Stream (MARS).

HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition

Videos are multimodal in nature. Conventional video recognition pipelines typically fuse multimodal features for improved performance. However, this is not only computationally expensive but also neglects the fact that different videos rely on different modalities for predictions. This article introduces Hierarchical and Conditional Modality Selection (HCMS), a simple yet efficient multimodal learning framework for efficient video recognition. HCMS operates on a low-cost modality, i.e., audio clues, by default, and dynamically decides on-the-fly whether to use computationally expensive modalities, including appearance and motion clues, on a per-input basis. This is achieved by the collaboration of three LSTMs that are organized in a hierarchical manner. In particular, LSTMs that operate on high-cost modalities contain a gating module, which takes as inputs lower-level features and historical information to adaptively determine whether to activate its corresponding modality; otherwise, it simply reuses historical information. We conduct extensive experiments on two large-scale video benchmarks, FCVID and ActivityNet, and the results demonstrate the proposed approach can effectively explore multimodal information for improved classification performance while requiring much less computation.

TSNet: Token Sparsification for Efficient Video Transformer

In the domain of video recognition, video transformers have demonstrated remarkable performance, albeit at significant computational cost. This paper introduces TSNet, an innovative approach for dynamically selecting informative tokens from given video samples. The proposed method involves a lightweight prediction module that assigns importance scores to each token in the video. Tokens with top scores are then utilized for self-attention computation. We apply the Gumbel-softmax technique to sample from the output of the prediction module, enabling end-to-end optimization of the prediction module. We aim to extend our method on hierarchical vision transformers rather than single-scale vision transformers. We use a simple linear module to project the pruned tokens, and the projected result is then concatenated with the output of the self-attention network to maintain the same number of tokens while capturing interactions with the selected tokens. Since feedforward networks (FFNs) contribute significant computation, we also propose linear projection for the pruned tokens to accelerate the model, and the existing FFN layer progresses the selected tokens. Finally, in order to ensure that the structure of the output remains unchanged, the two groups of tokens are reassembled based on their spatial positions in the original feature map. The experiments conducted primarily focus on the Kinetics-400 dataset using UniFormer, a hierarchical video transformer backbone that incorporates convolution in its self-attention block. Our model demonstrates comparable results to the original model while reducing computation by over 13%. Notably, by hierarchically pruning 70% of input tokens, our approach significantly decreases 55.5% of the FLOPs, while the decline in accuracy is confined to 2%. Additional testing of wide applicability and adaptability with other transformers such as the Video Swin Transformer was also performed and indicated its progressive potentials in video recognition benchmarks. By implementing our token sparsification framework, video vision transformers can achieve a remarkable balance between enhanced computational speed and a slight reduction in accuracy.

Temporal segment dropout for human action video recognition

Temporal information is important for human action video recognition. With the widely used spatio-temporal neural networks, researchers have found that the learned high-level features preserve overfitted spatial information and limited temporal information, leading to inferior performance. This is because existing networks lack efficient regularization for the temporal structure. To learn more robust temporal features, we propose a temporal regularization method named Temporal Segment Dropout (TSD). TSD drops the most salient spatial features in order to enhance the temporal features in a clip of temporal segments. Without learning from complex examples, TSD can be easily deployed in existing networks. In the experiment, TSD is extensively evaluated on benchmark action recognition datasets, which brings consistent improvements over the baselines, especially for the action-centric classes.

Thermal Image Processing for Respiratory Estimation from Cubical Data with Expandable Depth.

As healthcare costs continue to rise, finding affordable and non-invasive ways to monitor vital signs is increasingly important. One of the key metrics for assessing overall health and identifying potential issues early on is respiratory rate (RR). Most of the existing methods require multiple steps that consist of image and signal processing. This might be difficult to deploy on edge devices that often do not have specialized digital signal processors (DSP). Therefore, the goal of this study is to develop a single neural network realizing the entire process of RR estimation in a single forward pass. The proposed solution builds on recent advances in video recognition, capturing both spatial and temporal information in a multi-path network. Both paths process the data at different sampling rates to capture rapid and slow changes that are associated with differences in the temperature of the nostril area during the breathing episodes. The preliminary results show that the introduced end-to-end solution achieves better performance compared to state-of-the-art methods, without requiring additional pre/post-processing steps and signal-processing techniques. In addition, the presented results demonstrate its robustness on low-resolution thermal video sequences that are often used at the embedded edge due to the size and power constraints of such systems. Taking that into account, the proposed approach has the potential for efficient and convenient respiratory rate estimation across various markets in solutions deployed locally, close to end users.

Sneaky emotions: impact of data partitions in affective computing experiments with brain-computer interfacing

Brain-Computer Interfacing (BCI) has shown promise in Machine Learning (ML) for emotion recognition. Unfortunately, how data are partitioned in training/test splits is often overlooked, which makes it difficult to attribute research findings to actual modeling improvements or to partitioning issues. We introduce the “data transfer rate” construct (i.e., how much data of the test samples are seen during training) and use it to examine data partitioning effects under several conditions. As a use case, we consider emotion recognition in videos using electroencephalogram (EEG) signals. Three data splits are considered, each representing a relevant BCI task: subject-independent (affective decoding), video-independent (affective annotation), and time-based (feature extraction). Model performance may change significantly (ranging e.g. from 50% to 90%) depending on how data is partitioned, in classification accuracy. This was evidenced in all experimental conditions tested. Our results show that (1) for affective decoding, it is hard to achieve performance above the baseline case (random classification) unless some data of the test subjects are considered in the training partition; (2) for affective annotation, having data from the same subject in training and test partitions, even though they correspond to different videos, also increases performance; and (3) later signal segments are generally more discriminative, but it is the number of segments (data points) what matters the most. Our findings not only have implications in how brain data are managed, but also in how experimental conditions and results are reported.

Research on Evaluation Indicator of Ice Rink and Curling Stone Motion for the 2022 Beijing Winter Olympic Games Based on Video Recognition Method

During curling sports, the movement of the stone is affected by the quality of the ice. Therefore, the delivery team led by the ice maker hopes that the quality of the ice surface will be stable and that the athletes will always ‘read the ice’ and pay attention to the small changes in the ice surface. This phenomenon is the charm of curling. Many friction models have been proposed to describe the regularity of the curling motion. In the curling competitions of the 2022 Beijing Winter Olympic Games, the 2021 World Wheelchair Curling Championships, and the warm-up competition before, the research team installed a video image capture system in the arena to capture and record the data of the curling motion by using the depth neural network and object tracking algorithm. Further motion data research verifies the relationship between the friction coefficient and the speed. The quality control parameter of ice rink α is proposed, which is related to the influencing factors of the ice surface temperature, the ice hardness, the size of the pebble point, and the width of the curling friction band. The quality of the curling ice rink can be evaluated accurately and comprehensively by using parameter α. Based on the relationship between the friction coefficient and the speed, a physical model of horizontal sliding of the curling stone is established, which agrees well with the results of data obtained from video acquisition. Therefore, the movement distance along the rink can be accurately predicted. This paper analyzes the relationship between the long-time (the time it takes for the curling stone to travel between the two hog lines) and the stop position and that between the long-time and the split-time (the time it takes for the curling stone to travel from the back line to the hog line). Based on this result, a ruler can be established to assist athletes in estimating the sliding distance of the stone before curling throwing. This research also studies the relationship between three factors (the sliding speed in the x-direction, the angular speed, and a tiny lateral deflection speed in the y-direction) and the deviation of the stone. At the same time, there are also some interesting phenomena of the lateral deflection of the stone, such as the relationship between the lateral deflection angle tanθ and the initial lateral speed. As a result, the prediction of the curling stone’s exact final location can be realized. In summary, this article proposes an indicator for evaluating the quality of ice rinks and a physical model of curling based on the curling friction model, which is validated by data obtained from a video capture system of the 2022 Beijing Winter Olympics. The results described above have been applied in the post-match operation of the National Aquatics Center to guide the production of Olympic-grade ice surfaces and to guide athletes to “read ice” accurately during training.

Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos.

Adversarial robustness assessment for video recognition models has raised concerns owing to their wide applications on safety-critical tasks. Compared with images, videos have much high dimension, which brings huge computational costs when generating adversarial videos. This is especially serious for the query-based black-box attacks where gradient estimation for the threat models is usually utilized, and high dimensions will lead to a large number of queries. To mitigate this issue, we propose to simultaneously eliminate the temporal and spatial redundancy within the video to achieve an effective and efficient gradient estimation on the reduced searching space, and thus query number could decrease. To implement this idea, we design the novel Adversarial spatial-temporal Focus (AstFocus) attack on videos, which performs attacks on the simultaneously focused key frames and key regions from the inter-frames and intra-frames in the video. AstFocus attack is based on the cooperative Multi-Agent Reinforcement Learning (MARL) framework. One agent is responsible for selecting key frames, and another agent is responsible for selecting key regions. These two agents are jointly trained by the common rewards received from the black-box threat models to perform a cooperative prediction. By continuously querying, the reduced searching space composed of key frames and key regions is becoming precise, and the whole query number becomes less than that on the original video. Extensive experiments on four mainstream video recognition models and three widely used action recognition datasets demonstrate that the proposed AstFocus attack outperforms the SOTA methods, which is prevenient in fooling rate, query number, time, and perturbation magnitude at the same time.

Gate-Shift-Fuse for Video Action Recognition.

Convolutional Neural Networks are the de facto models for image recognition. However 3D CNNs, the straight forward extension of 2D CNNs for video recognition, have not achieved the same success on standard action recognition benchmarks. One of the main reasons for this reduced performance of 3D CNNs is the increased computational complexity requiring large scale annotated datasets to train them in scale. 3D kernel factorization approaches have been proposed to reduce the complexity of 3D CNNs. Existing kernel factorization approaches follow hand-designed and hard-wired techniques. In this paper we propose Gate-Shift-Fuse (GSF), a novel spatio-temporal feature extraction module which controls interactions in spatio-temporal decomposition and learns to adaptively route features through time and combine them in a data dependent manner. GSF leverages grouped spatial gating to decompose input tensor and channel weighting to fuse the decomposed tensors. GSF can be inserted into existing 2D CNNs to convert them into an efficient and high performing spatio-temporal feature extractor, with negligible parameter and compute overhead. We perform an extensive analysis of GSF using two popular 2D CNN families and achieve state-of-the-art or competitive performance on five standard action recognition benchmarks.

Action class relation detection and classification across multiple video datasets

The Meta Video Dataset (MetaVD) provides annotated relations between action classes in major datasets for human action recognition in videos. Although these annotated relations enable dataset augmentation, it is only applicable to those covered by MetaVD. For an external dataset to enjoy the same benefit, the relations between its action classes and those in MetaVD need to be determined. To address this issue, we consider two new machine learning tasks: action class relation detection and classification. We propose a unified model to predict relations between action classes, using language and visual information associated with classes. Experimental results show that (i) pre-trained recent neural network models for texts and videos contribute to high predictive performance, (ii) the relation prediction based on action label texts is more accurate than based on videos, and (iii) a blending approach that combines predictions by both modalities can further improve the predictive performance in some cases.

A heterogeneous two-stream network for human action recognition

The most widely used two-stream architectures and building blocks for human action recognition in videos generally consist of 2D or 3D convolution neural networks. 3D convolution can abstract motion messages between video frames, which is essential for video classification. 3D convolution neural networks usually obtain good performance compared with 2D cases, however it also increases computational cost. In this paper, we propose a heterogeneous two-stream architecture which incorporates two convolutional networks. One uses a mixed convolution network (MCN), which combines some 3D convolutions in the middle of 2D convolutions to train RGB frames, another one adopts BN-Inception network to train Optical Flow frames. Considering the redundancy of neighborhood video frames, we adopt a sparse sampling strategy to decrease the computational cost. Our architecture is trained and evaluated on the standard video actions benchmarks of HMDB51 and UCF101. Experimental results show our approach obtains the state-of-the-art performance on the datasets of HMDB51 (73.04%) and UCF101 (95.27%).

A Jeap-BiLSTM Neural Network for Action Recognition

Human action recognition in videos is an important task in computer vision with applications in fields such as surveillance, human–computer interaction, and sports analysis. However, it is a challenging task due to the complex background changes and redundancy of long-term video information. In this paper, we propose a novel bi-directional long short-term memory method with attention pooling based on joint motion and difference entropy (JEAP-BiLSTM) to address these challenges. To obtain discriminative features, we introduce a joint entropy map that measures both the entropy of motion and the entropy of change. The Bi-LSTM method is then applied to capture visual and temporal associations in both forward and backward directions, enabling efficient capture of long-term temporal correlation. Furthermore, attention pooling is used to highlight the region of interest and to mitigate the effects of background changes in video information. Experiments on the UCF101 and HMDB51 datasets demonstrate that the proposed JEAP-BiLSTM method achieves recognition rates of 96.4% and 75.2%, respectively, outperforming existing methods. Our proposed method makes significant contributions to the field of human action recognition by effectively capturing both spatial and temporal patterns in videos, addressing background changes, and achieving state-of-the-art performance.

DeepVQL: Deep Video Queries on PostgreSQL

The recent development of mobile and camera devices has led to the generation, sharing, and usage of massive amounts of video data. As a result, deep learning technology has gained attention as an alternative for video recognition and situation judgment. Recently, new systems supporting SQL-like declarative query languages have emerged, focusing on developing their own systems to support new queries combined with deep learning that are not supported by existing systems. The proposed DeepVQL system in this paper is implemented by expanding the PostgreSQL system. DeepVQL supports video database functions and provides various user-defined functions for object detection, object tracking, and video analytics queries. The advantage of this system is its ability to utilize queries with specific spatial regions or temporal durations as conditions for analyzing moving objects in traffic videos.