Features Of Video Frames Research Articles

Automated detection of steps in videos of strabismus surgery using deep learning

BackgroundLearning to perform strabismus surgery is an essential aspect of ophthalmologists’ surgical training. Automated classification strategy for surgical steps can improve the effectiveness of training curricula and the efficient evaluation of residents’ performance. To this end, we aimed to develop and validate a deep learning (DL) model for automated detecting strabismus surgery steps in the videos.MethodsIn this study, we gathered 479 strabismus surgery videos from Shanghai Children’s Hospital, affiliated to Shanghai Jiao Tong University School of Medicine, spanning July 2017 to October 2021. The videos were manually cut into 3345 clips of the eight strabismus surgical steps based on the International Council of Ophthalmology’s Ophthalmology Surgical Competency Assessment Rubrics (ICO-OSCAR: strabismus). The videos dataset was randomly split by eye-level into a training (60%), validation (20%) and testing dataset (20%). We evaluated two hybrid DL algorithms: a Recurrent Neural Network (RNN) based and a Transformer-based model. The evaluation metrics included: accuracy, area under the receiver operating characteristic curve, precision, recall and F1-score.ResultsDL models identified the steps in video clips of strabismus surgery achieved macro-average AUC of 1.00 (95% CI 1.00–1.00) with Transformer-based model and 0.98 (95% CI 0.97-1.00) with RNN-based model, respectively. The Transformer-based model yielded a higher accuracy compared with RNN-based models (0.96 vs. 0.83, p < 0.001). In detecting different steps of strabismus surgery, the predictive ability of the Transformer-based model was better than that of the RNN. Precision ranged between 0.90 and 1 for the Transformer-based model and 0.75 to 0.94 for the RNN-based model. The f1-score ranged between 0.93 and 1 for the Transformer-based model and 0.78 to 0.92 for the RNN-based model.ConclusionThe DL models can automate identify video steps of strabismus surgery with high accuracy and Transformer-based algorithms show excellent performance when modeling spatiotemporal features of video frames.

Cross-stage recurrent feature sharing network for video dehazing

Hazy weather conditions feed serious haze effects, such as color and visibility degradations in the captured videos. Extensive research on video dehazing has been carried out. However, they lack to balance the spatial and high-level contextual information during restoration. Being a complex task in this paper, we propose a cross-stage recurrent feature sharing network to balance the competing objective. In the proposed network, a cross-stage feature merging module operates on learned features of the previous video frame using the proposed encoder decoder based generator architecture. Learned features fed into the same encoder decoder framework for the current video frame feature learning. This module benefit the network while restoring the current video frame. It shares the feature information between two consecutive frames as they have a very minute information change. Meanwhile, the design of our encoder decoder generator architecture is based on a multi-receptive resolution module, feature fusion module, and pixel-wise spatial attention module. These modules mainly focus on acquiring broad contextual information, fusing intermediate-stage features to increase the learning ability of the network, and capturing robust features while discarding unnecessary features. Experimental evaluation results and ablation study show that the proposed network is superior to existing state-of-the-art methods for video dehazing.

Facial expression recognition based on adaptation of the classifier to videos of the user

In this paper, an approach that can significantly increase the accuracy of facial emotion recognition by adapting the model to the emotions of a particular user (e.g., smartphone owner) is considered. At the first stage, a neural network model, which was previously trained to recognize facial expressions in static photos, is used to extract visual features of faces in each frame. Next, the face features of video frames are aggregated into a single descriptor for a short video fragment. After that a neural network classifier is trained. At the second stage, it is proposed that adaptation (fine-tuning) to this classifier should be performed using a small set of video data with the facial expressions of a particular user. After emotion classification, the user can adjust the predicted emotions to further improve the accuracy of a personal model. As part of an experimental study for the RAVDESS dataset, it has been shown that the approach with model adaptation to a specific user can significantly (up to 20 – 50 %) improve the accuracy of facial expression recognition in the video.

Gaze prediction based on long short-term memory convolution with associated features of video frames

Gaze prediction is a key issue for visual perception research. It can be used to infer important regions in videos to reduce the amount of computation in learning and inference of various analysis tasks. Vanilla methods for dynamic video unable to extract valid features, and the motion information among dynamic video frames are ignored, which lead to poor prediction results. We propose a gaze prediction based on LSTM convolution with associated features of video frames (LSTM-CVFAF). Firstly, by adding learnable central prior knowledge, the proposed method can effectively and accurately extract the spatial information of each frame. Secondly, the LSTM is deployed to get temporal motion gaze features. Finally, the spatial and temporal motion information is fused to generate the gaze prediction maps of the dynamic video. Compared with the state-of-art models on DHF1K dataset, the CC, AUC-j, sAUC, NSS are separately increased by 5.1%, 0.6%, 38.2% and 0.5%.

Sequential Order-Aware Coding-Based Robust Subspace Clustering for Human Action Recognition in Untrimmed Videos.

Human action recognition (HAR) is one of most important tasks in video analysis. Since video clips distributed on networks are usually untrimmed, it is required to accurately segment a given untrimmed video into a set of action segments for HAR. As an unsupervised temporal segmentation technology, subspace clustering learns the codes from each video to construct an affinity graph, and then cuts the affinity graph to cluster the video into a set of action segments. However, most of the existing subspace clustering schemes not only ignore the sequential information of frames in code learning, but also the negative effects of noises when cutting the affinity graph, which lead to inferior performance. To address these issues, we propose a sequential order-aware coding-based robust subspace clustering (SOAC-RSC) scheme for HAR. By feeding the motion features of video frames into multi-layer neural networks, two expressive code matrices are learned in a sequential order-aware manner from unconstrained and constrained videos, respectively, to construct the corresponding affinity graphs. Then, with the consideration of the existence of noise effects, a simple yet robust cutting algorithm is proposed to cut the constructed affinity graphs to accurately obtain the action segments for HAR. The extensive experiments demonstrate the proposed SOAC-RSC scheme achieves the state-of-the-art performance on the datasets of Keck Gesture and Weizmann, and provides competitive performance on the other 6 public datasets such as UCF101 and URADL for HAR task, compared to the recent related approaches.

A hybrid-attention-ConvLSTM-based deep learning architecture to extract modal frequencies from limited data using transfer learning

This paper leverages each pixel of a picture acquired from a video camera, in which structural dynamic information is contained, in order to decompose spatiotemporal information from such a non-contact virtual sensor array in the same way as traditional accelerometers to extract structural modal frequencies. Attention-based deep neural network architecture is proposed in this work to better visualize the dynamic properties of structures in the existence of noise with a high resolution. The work combines CNNs and Recurrent Neural Networks (RNNs) to predict modal frequencies of structures from a series of consecutive images. High discriminative features of video frames are firstly extracted using the CNN, and then Conv-Long Short-Term Memory (ConvLSTM) is applied to further process the extracted features to capture the temporal dynamics in videos. The attention mechanisms are embedded in the network to ensure the model learns to focus selectively on those frames containing system dynamics. In particular, the proposed computer vision-based deep learning model takes the video of a vibrating structure as the input and successfully estimates the modal frequencies. Transfer learning is applied to cohere the knowledge learned from publicly available datasets to a much more sophisticated structure and estimate the resonant frequencies. The proposed algorithm optimizes the filter design for video processing in a fully automated way without any human intervention and can generalize and transfer that learned information to more complex structures. The model is trained using publicly available generic baseline data (Dataset A) consisting of several simple beam structures with different material properties and sizes and transferred the learned knowledge to unseen data (Dataset B) consisting of an independent turbine blade. It is concluded that the newly proposed method is more autonomous, accurate, and capable of generalizing the model to a new independent dataset using a transfer learning strategy, and the most advantage of the proposed approach is that the trained deep learning architecture has the capability of estimating the resonant frequencies for independent structures and extending the resonant frequency estimations to higher modes.

A multi-flexible video summarization scheme using property-constraint decision tree

Video summarization (VS) technology presents a multi-tendency development. However, there are still certain challenges in shortening a long video into several diverse and concise versions, such as generating various versions and variable lengths of summaries depending on user requirements. In this context, it is critical to extract the spatial–temporal features of video frames. To address these issues, this paper proposes a multi-flexible video summarization scheme using a property-constraint decision.Due to the difficulty of directly transforming user requirements into summary algorithm factors, this paper employs property-constraints as a bridge between user requirements and the video summary algorithm. The specific implementation method is to aggregate and analyze existing research results on summary-oriented properties. It begins with the establishment of a property-constraints library, followed by the translating and mapping of user requirements into VS property-constraints, and ultimately constructing a property-constraints tree for flexible decision versions of VS. In addition, a hybrid cascade bidirectional network (CB-ConvLSTM) based on the Convolutional LSTM Network (ConvLSTM) is designed to extract the spatial–temporal features of the video. On this basis, the VS generator is configured. The goal of this scheme, which combines the VS property constraints and CB-ConvLSTM, is to “analyze once, satisfy multiple factors, generate multiple levels”. Verification experiments and a comparative analysis are conducted on benchmark datasets to evaluate the algorithm in this paper. The results indicate that the proposed algorithm is highly rational, effective, and usable.

Region Dual Attention-Based Video Emotion Recognition.

To solve the emotional differences between different regions of the video frame and make use of the interrelationship between different regions, a region dual attention-based video emotion recognition method (RDAM) is proposed. RDAM takes as input video frame sequences and learns a discriminatory video emotion representation that can make full use of the emotional differences of different regions and the interrelationship between regions. Specifically, we construct two parallel attention modules: one is the regional location attention module, which generates a weight value for each feature region to identify the relative importance of different regions. Based on the weight, the emotion feature that can perceive the emotional sensitive region is generated. The other is the regional relationship attention module, which generates a region relation matrix that represents the interrelationship of different regions of a video frame. Based on the region relation matrix, the emotion feature that can perceive interrelationship between different regions is generated. The outputs of these two attention modules are fused to produce the emotional features of video frames. Then, the features of video frame sequences are fused by attention-based fusion network, and the final emotion feature of the video is produced. The experimental results on the video emotion recognition data sets show that the proposed method outperforms the other related works.

Detecting abnormality with separated foreground and background: Mutual Generative Adversarial Networks for video abnormal event detection

As one of the most important tasks in intelligent video analysis, video abnormal event detection has been extensively studied. Prior arts have made a great process in designing frameworks to capture spatio-temporal features of video frames. However, video frames usually consist of various objects. It is challenging to grasp the nuances of anomalies against noisy backgrounds. To tackle the bottleneck, we propose a novel Foreground–Background Separation Mutual Generative Adversarial Network (FSM-GAN) framework. The FSM-GAN permits the separation of video frames into the foreground and background. The separated foreground and background are utilized as the input of mutual generative adversarial networks, which transform raw-pixel images in optical-flow representations and vice versa. In the networks, the background is regarded as known conditions and the model focuses on learning the high-level spatio-temporal foreground features to represent the event with the given conditions during the mutual adversarial training. In the test stage, these high-level features instead of low-level visual primitives are utilized to measure the abnormality in the semantic level. Compared with state-of-the-art methods and other abnormal event detection approaches, the proposed framework demonstrates its effectiveness and reliability across various scenes and events.

Zero-watermarking Algorithm for Audio and Video Matching Verification

&lt;abstract&gt; &lt;p&gt;For the needs of tamper-proof detection and copyright identification of audio and video matching, this paper proposes a zero-watermark algorithm that can be used for audio and video matching verification. The algorithm segments audio and video in smaller time units, generates a video frame feature matrix based on NSCT, DCT, and SVD, and generates a sound watermark based on methods such as DWT and K-means. The zero watermark combines video, audio and copyright information. The experimental results show that the zero watermark generated by this algorithm can not only realize highly accurate matching detection and positioning of audio and video, but also well resist common single attack and combination attacks such as noise, scaling, rotation, frame attack and format conversion, which has good robustness.&lt;/p&gt; &lt;/abstract&gt;

Key Clips and Key Frames Extraction of Videos Based on Deep Learning

The surveillance camera network covering the city, while protecting the safety of people’s lives and property, generate a large amount of surveillance video data every day, but few videos contain useful information. Surveillance cameras in sparsely crowded areas may capture most of the video in the background. A large number of surveillance videos bring greater storage pressure, and also increase the difficulty of the staff’s work. In this paper, we use the auto-encoder network to extract features of video frames. By comparing the feature differences between video frames, we can automatically select key video clips and key frame images that contain useful information to slim down the surveillance video. Through experiments on actual cameras, we found that this method can achieve the purpose of reducing the pressure of storage and traversal.

Video emotion recognition based on Convolutional Neural Networks

The existing video sentiment analysis methods only obtain features from the spatial and temporal signals of the video for sentiment classification, and cannot solve the difficulty of not knowing which emotion contributes the most to the entire video sentiment analysis in the video sentiment analysis. To solve this problem, a neural network with video frame weight vector is proposed. First, the video frame feature is obtained through the reel neural network, and then the weight vector layer is used to calculate the weight of the feature, and finally the frame feature with weight is put into the LSTM Training to obtain a video sentiment analysis model. We verified on the BAUM-1s data set. The results show that this method is better than existing methods in accuracy.

An Efficient CNN and BI-LSTM Model for Abnormal Event Detection in Video Surveillance

Abnormal event detection plays a complex and important task in intelligent transportation and Computer vision. At present, the existing abnormal event detection models based on deep learning mainly focus on data represented by a vectorial form, which pay little attention to the impact of the internal structure characteristics of feature vector. In order to address the above issues, we propose an abnormal event detection hybrid modulation method via feature extraction calibrating classification in video surveillance scenes in this paper. Our main contribution is to calibrate the classification of an abnormal event by constructing feature expectation subgraphs. First, we employ convolutional neural network and long short-term memory models to extract the spatiotemporal features of video frame, and then construct the feature expectation subgraph for each key frame of every video, which could be used to capture the internal sequential and topological relational characteristics of structured feature vector. This model included a pre-trained CNN for extracting spatio-temporal features from each individual frame selected from a series of frames, which is then passed to multi-layer Bi-directional Long Short-term Memory (BD-LSTM) that possesses the capability of accurately classifying the abnormal events in complex surveillance scenes of the road. Finally, the experiments and performance of frame level and pixel level values on a common UMN dataset are estimated and evaluated.

Translating video into language by enhancing visual and language representations

It is a fundamental task of translating videos into natural language automatically by computer. At present, the models for video description based on deep learning have made a great breakthrough. However, the static information loss is serious during encoding stage for motion feature of videos, and the linguistic feature from LSTM network lack personalized expression, leading to inappropriate words and poor semantics in generation sentences. In this work, a model with enhanced features of visual and language is proposed to address the challenges. First, static features of video frames from the first LSTM layer are incorporated, then fed into another LSTM layer according by frame sequence. Second, the feature of word is combined with the output of LSTM network for predicted probability of candidate word on each time step. The experimental results demonstrate effectiveness of the proposed approach with competitive performance compared with other state-of-the-art methods on various metrics.

Abnormal Event Detection via Feature Expectation Subgraph Calibrating Classification in Video Surveillance Scenes

At present, the existing abnormal event detection models based on deep learning mainly focus on data represented by a vectorial form, which pay little attention to the impact of the internal structure characteristics of feature vector. In addition, a single classifier is difficult to ensure the accuracy of classification. In order to address the above issues, we propose an abnormal event detection hybrid modulation method via feature expectation subgraph calibrating classification in video surveillance scenes in this paper. Our main contribution is to calibrate the classification of a single classifier by constructing feature expectation subgraphs. First, we employ convolutional neural network and long short-term memory models to extract the spatiotemporal features of video frame, and then construct the feature expectation subgraph for each key frame of every video, which could be used to capture the internal sequential and topological relational characteristics of structured feature vector. Second, we project expectation subgraphs on the sparse vector to combine with a support vector classifier to calibrate the results of a linear support vector classifier. Finally, the experiments on a common dataset named UCSDped1 and a coal mining video dataset in comparison with some existing works demonstrate that the performance of the proposed method is better than several the state-of-the-art approaches.

Key Algorithm for Human Motion Recognition in Virtual Reality Video Sequences Based on Hidden Markov Model

This paper provides an in-depth discussion of human motion recognition in Virtual Reality (VR) video sequences through hidden Markov models, which are four steps from VR video acquisition and pre-processing, foreground detection, extraction of human feature parameters, and hidden Markov model human motion recognition. A hybrid Gaussian model was used to build a background model in real-time based on changes in VR video information, and the image was subtracted by the background differential method. The optical flow method was used for foreground detection of the target, and the effects of sparse and dense optical flow were compared to obtain the motion characteristics and optical flow information of the target human body, respectively. Features were extracted for human motion, in terms of common geometric features of the body and optical flow information, respectively. In terms of common geometric information, the width-to-height ratio, perimeter-to-area ratio, center of mass, eccentricity, and feature angle were extracted, respectively. For the optical flow information, optical flow descriptors were constructed using a grid-based approach. And feature fusion was performed for the above two parameters by the k-means method to construct the word pocket model. The hidden Markov model parameters were obtained by using the hidden Markov model for the recognition of human motion and training the human feature parameters for each of the four motions. The recognition of the four common human body movements was realized by the forward-backward algorithm. The test results show that the motion recognition method in this paper has high recognition performance and good anti-interference performance. The time-sequence pooling is used to sort the effective video frame feature sequences to obtain the feature vectors that can represent the dynamic changes of video time sequence; finally, the time-sequence feature vectors are used to train the support vector machine for classification recognition. The recognition accuracy is 65.2% and 89.4% for the HMDB51 and UCF101 datasets, respectively.

Joint Dynamic Pose Image and Space Time Reversal for Human Action Recognition from Videos

Human action recognition aims to classify a given video according to which type of action it contains. Disturbance brought by clutter background and unrelated motions makes the task challenging for video frame-based methods. To solve this problem, this paper takes advantage of pose estimation to enhance the performances of video frame features. First, we present a pose feature called dynamic pose image (DPI), which describes human action as the aggregation of a sequence of joint estimation maps. Different from traditional pose features using sole joints, DPI suffers less from disturbance and provides richer information about human body shape and movements. Second, we present attention-based dynamic texture images (att-DTIs) as pose-guided video frame feature. Specifically, a video is treated as a space-time volume, and DTIs are obtained by observing the volume from different views. To alleviate the effect of disturbance on DTIs, we accumulate joint estimation maps as attention map, and extend DTIs to attention-based DTIs (att-DTIs). Finally, we fuse DPI and att-DTIs with multi-stream deep neural networks and late fusion scheme for action recognition. Experiments on NTU RGB+D, UTD-MHAD, and Penn-Action datasets show the effectiveness of DPI and att-DTIs, as well as the complementary property between them.

Spatial-temporal pyramid based Convolutional Neural Network for action recognition

Convolutional Neural Networks (CNNs) usually use top-level appearance features of video frames for action recognition. However, these methods discard the implicit complementary advantages across different-scale appearance representations which are effective for object detection, instance segmentation and person re-identification. In this paper, a new spatial pyramid module is proposed to take full use of inherent multi-scale information of CNNs with nearly cost-free by which a bottom-up architecture with lateral connections is constructed for combining high-, mid-, low-level representations of CNNs into a hierarchical frame-level feature elaborately. Additionally, temporal relations at appropriate timescale are contributed to the identification of an action. To this end, we also propose a new temporal pyramid module in which frame-level features belonged to one video are reused by various timescale pooling approaches to get different time-grained features of snippets efficiently. Followed by snippet-relation reasoning, different timescale temporal relations are derived and accumulated for the comprehensive prediction. Unifying the proposed spatial and temporal pyramid modules, a novel network, Spatial-Temporal Pyramid Network (S-TPNet), is proposed to extract spatial-temporal pyramid features for action recognition in videos. Unlike previous models which boost performance at the cost of computation, S-TPNet can be trained in an end-to-end fashion with great efficiency. Extensive experiments on Kinetics, UCF101, and HMDB51 demonstrate that S-TPNet displays significant performance improvements compared with existing frameworks and obtains competitive performance with the state-of-the-arts.11The code is available at https://www.github.com/ZhenxingZheng/S-TPNet.

An attention mechanism based convolutional LSTM network for video action recognition

As an important issue in video classification, human action recognition is becoming a hot topic in computer vision. The ways of effectively representing the spatial static and temporal dynamic information of videos are important problems in video action recognition. This paper proposes an attention mechanism based convolutional LSTM action recognition algorithm to improve the accuracy of recognition by extracting the salient regions of actions in videos effectively. First, GoogleNet is used to extract the features of video frames. Then, those feature maps are processed by the spatial transformer network for the attention. Finally the sequential information of the features is modeled via the convolutional LSTM to classify the action in the original video. To accelerate the training speed, we adopt the analysis of temporal coherence to reduce the redundant features extracted by GoogleNet with trivial accuracy loss. In comparison with the state-of-the-art algorithms for video action recognition, competitive results are achieved on three widely-used datasets, UCF-11, HMDB-51 and UCF-101. Moreover, by using the analysis of temporal coherence, desirable results are obtained while the training time is reduced.

Event detection in soccer videos using unsupervised learning of Spatio-temporal features based on pooled spatial pyramid model

Most existing researches for semantic analysis of soccer videos benefit from special approaches to bridge the semantic gap between low-level features and high-level events using a hierarchical structure. In this paper, we propose a novel data-driven model for automatic recognition of important events in soccer broadcast videos based on the analysis of spatio-temporal local features of video frames. Our presented algorithm explores the local visual content of video frames by focusing on spatial and temporal learned features in a low-dimensional transformed sparse space. The proposed algorithm, without using mid-level futures, dynamically extracts the most informative semantic concepts/features and improves the generality of the system. The dictionary learning process plays an important role in sparse coding and sparse representation-based event classification. In this paper, we present a novel dictionary learning method, which calculates several category-specific dictionaries by training the detected shots of various view categories. In order to evaluate the feasibility and effectiveness of the proposed algorithm, an extensive experimental investigation is conducted for the analysis, detection, and classification of soccer events on a large collection of video data. Experimental results indicate that our approach outperforms the state-of-the-art methods and demonstrate the effectiveness of the proposed approach.