Frames Of Video Sequence Research Articles

Fast Dual Discrete and Complexwavelet Based Video Magnification for 3D Facial Video Identification

Magnifying micro motion of videos that are undetectable by humans has recently been popular in many applications. This is due to its impact in numerous applications. In this paper, we explore this technique in 3D facial video identification, where we try to distinguish between real 3D facial objects in videos and 2D images of faces in a video frame sequence, and utilize this in biometric identification. We present a fast 2D Dual Discrete Wavelet Transform 2D-DWT based video magnification technique that detects micro movements by magnifying the phase differences between subsequent video frame's wavelet coefficients, at different sub bands. Next, in order to overcome shortcoming of 2D-DWT systems, 2D Dual Complex Wavelet Transform 2D-CWT has also been employed to estimate phase changes between subsequent video frames at different spatial locations of Complex Wavelets sub-bands. This latter presented CWT Technique uses the Radon Transform to detect any periodic motion in the video frames. Several simulation results are given to show that our proposed hybrid technique achieves comparable and sometimes superior performance with far less complexity when compared with recent literature in micro motion magnification, such as steerable pyramids STR and Riesz Transform RT based steerable pyramids RT-STR. Both DWT and CWT techniques are combined for 3D facial video identification. The attached videos demonstrate the superior video quality obtained by the proposed technique.

Big Data and Deep Learning‐Based Video Classification Model for Sports

Information technologies such as deep learning, big data, cloud computing, and the Internet of Things provide key technical tools to drive the rapid development of integrated manufacturing. In recent years, breakthroughs have been made in big data analysis using deep learning. The research on the sports video high‐precision classification model in this paper, more specifically, is the automatic understanding of human movements in free gymnastics videos. This paper will combine knowledge related to big data‐based computer vision and deep learning to achieve intelligent labeling and representation of specific human movements present in video sequences. This paper mainly implements an automatic narrative based on long‐ and short‐term memory networks to achieve the classification of sports videos. In the classical video description model S2VT, long‐ and short‐term memory networks are used to learn the mapping relationship between word sequences and video frame sequences. In this paper, we introduce an attention mechanism to highlight the importance of keyframes that determine freestyle gymnastic movements. In this paper, a dataset of freestyle gymnastics breakdown movements for professional events is built. Experiments are conducted on the data and the self‐constructed dataset, and the planned sampling method is applied to eliminate the differences between the training decoder and the prediction decoder. The experimental results show that the improved method in this paper can improve the accuracy of sports video classification. The video classification model based on big data and deep learning is to provide users with a better user experience and improve the accuracy of video classification. Also, in the experiments of this paper, the effect of extracting features for the classification of different lifting sports models is compared, and the effect of feature extraction network on the automatic description of free gymnastic movements is analyzed.

Subsampled turbulence removal network

We present a deep-learning approach to restore a sequence of turbulence-distorted video frames from turbulent deformations and space-time varying blurs. Instead of requiring a massive training sample size in deep networks, we purpose a training strategy that is based on a new data augmentation method to model turbulence from a relatively small dataset. Then we introduce a subsampled method to enhance the restoration performance of the presented GAN model. The contributions of the paper is threefold: first, we introduce a simple but effective data augmentation algorithm to model the turbulence in real life for training in the deep network; Second, we firstly purpose the Wasserstein GAN combined with $\ell_1$ cost for successful restoration of turbulence-corrupted video sequence; Third, we combine the subsampling algorithm to filter out strongly corrupted frames to generate a video sequence with better quality.

Deep Edge Computing for Videos

This paper provides a modular architecture with deep neural networks as a solution for real-time video analytics in an edge-computing environment. The modular architecture consists of two networks of Front-CNN (Convolutional Neural Network) and Back-CNN, where we adopt Shallow 3D CNN (S3D) as the Front-CNN and a pre-trained 2D CNN as the Back-CNN. The S3D (i.e., the Front CNN) is in charge of condensing a sequence of video frames into a feature map with three channels. That is, the S3D takes a set of sequential frames in the video shot as input and yields a learned 3 channel feature map (3CFM) as output. Since the 3CFM is compatible with the three-channel RGB color image format, we can use the output of the S3D (i.e., the 3CFM) as the input to a pre-trained 2D CNN of the Back-CNN for the transfer-learning. This serial connection of Front-CNN and Back-CNN architecture is end-to-end trainable to learn both spatial and temporal information of videos. Experimental results on the public datasets of UCF-Crime and UR-Fall Detection show that the proposed S3D-2DCNN model outperforms the existing methods and achieves state-of-the-art performance. Moreover, since our Front-CNN and Back-CNN modules have a shallow S3D and a light-weighted 2D CNN, respectively, it is suitable for real-time video recognition in edge-computing environments. We have implemented our CNN model on NVIDIA Jetson Nano Developer as an edge-computing device to show its real-time execution.

Block Matching Algorithms for the Estimation of Motion in Image Sequences: Analysis

Several video coding standards and techniques have been introduced for multimedia applications, particularly the h.26x series for video processing. These standards employ motion estimation processing to reduce the amount of data that is required to store or transmit the video. The motion estimation process is an inextricable part of the video coding as it removes the temporal redundancy between successive frames of video sequences. This paper is about these motion estimation algorithms, their search procedures, complexity, advantages, and limitations. A survey of motion estimation algorithms including full search, many fast, and fast full search block-based algorithms has been presented. An evaluation of up-to-date motion estimation algorithms, based on several empirical results on several test video sequences, is presented as well.

Background modeling from video sequences via online motion-aware RPCA

Background modeling of video frame sequences is a prerequisite for computer vision applications. Robust principal component analysis(RPCA), which aims to recover low rank matrix in applications of data mining and machine learning, has shown improved background modeling performance. Unfortunately, The traditional RPCA method considers the batch recovery of low rank matrix of all samples, which leads to higher storage cost. This paper proposes a novel online motion-aware RPCA algorithm, named OM-RPCAT, which adopt truncated nuclear norm regularization as an approximation method for of low rank constraint. And then, Two methods are employed to obtain the motion estimation matrix, the optical flow and the frame selection, which are merged into the data items to separate the foreground and background. Finally, an efficient alternating optimization algorithm is designed in an online manner. Experimental evaluations of challenging sequences demonstrate promising results over state-of-the-art methods in online application.

Complete Video-Level Representations for Action Recognition

In most of the existing work for activity recognition, 3D ConvNets show promising performance for learning spatiotemporal features of videos. However, most methods sample fixed-length frames from the original video, which are cropped to a fixed size and fed into the model for training. In this manner, two problems limit the model performance for recognition. First, the cropped video clips are incomplete or even distorted in appearance, resulting in a large gap between the feature representation and semantics of human activity. Second, the useful features of longer video frame sequences are weakened by the repeated stacking of 3D convolution over deep networks due to the limitations of GPU memory and computing ability. This article proposes a method based on a 3D backbone network for multi scale spatial feature representation, which uses a pyramid pooling layer to allow the input of video frames at different scales, and then aggregates short-term spatial–temporal features into a long-term video-level representation. Objection detection is used as a component of model testing to explore the improvement of activity recognition considering the large amount of space–time redundancy in real life videos. An experiment is performed on the principal video dataset, UCF101, and the proposed method presents a competitive performance.

Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations

Video prediction which maps a sequence of past video frames into realistic future video frames is a challenging task because it is difficult to generate realistic frames and model the coherent relationship between consecutive video frames. In this paper, we propose a hierarchical sequence-to-sequence prediction approach to address this challenge. We present an end-to-end trainable architecture in which the frame generator automatically encodes input frames into different levels of latent Convolutional Neural Network (CNN) features, and then recursively generates future frames conditioned on the estimated hierarchical CNN features and previous prediction. Our design is intended to automatically learn hierarchical representations of video and their temporal dynamics. Convolutional Long Short-Term Memory (ConvLSTM) is used in combination with skip connections so as to separately capture the sequential structures of multiple levels of hierarchy of features. We adopt Scheduled Sampling for training our recurrent network in order to facilitate convergence and to produce high-quality sequence predictions. We evaluate our method on the Bouncing Balls, Moving MNIST, and KTH human action dataset, and report favorable results as compared to existing methods.

Retraction Note: Edge-Oriented Spatial Interpolation for Error Concealment of Consecutive Blocks

This paper proposes a low-complexity spatial-domain error concealment (EC) algorithm for recovering consecutive blocks error in still images or intra-coded (I) frames of video sequences. The proposed algorithm works with the following steps. Firstly the Sobel operator is performed on the top and bottom adjacent pixels to detect the most probable edge direction of current block area. After that one-dimensional (1-D) matching is used on the available block boundaries. Displacement between edge direction candidate and most probable edge direction is taken into consideration as an important factor to improve stability of 1-D boundary matching. Then the corrupted pixels are recovered by linear weighting interpolation along the estimated edge direction. Finally the interpolated values are merged to get last recovered picture. Simulation results demonstrate that the proposed algorithms obtain good subjective quality and higher PSNR than the methods in literatures for most images.

Enhancing the alignment between target words and corresponding frames for video captioning

Video captioning aims at translating from a sequence of video frames into a sequence of words with the encoder-decoder framework. Hence, it is critical to align these two different sequences. Most existing methods exploit soft-attention (temporal attention) mechanism to align target words with corresponding frames, where the relevance of them merely depends on the previously generated words (i.e., language context). As we know, however, there is an inherent gap between vision and language, and most of the words in a caption belong to non-visual words (e.g. “a”, “is”, and “in”). Hence, merely with the guidance of the language context, existing temporal attention-based methods cannot exactly align target words with corresponding frames. In order to address this problem, we first introduce pre-detected visual tags from the video to bridge the gap between vision and language. The reason is that visual tags not only belong to textual modality, but also can convey visual information. Then, we present a Textual-Temporal Attention Model (TTA) to exactly align the target words with corresponding frames. The experimental results show that our proposed method outperforms the state-of-the-art methods on two well known datasets, i.e., MSVD and MSR-VTT. 11Our code is available at https://github.com/tuyunbin/Enhancing-the-Alignment-between-Target-Words-and-Corresponding-Frames-for-Video-Captioning

Multi-Turn Video Question Generation via Reinforced Multi-Choice Attention Network

Video question generation is a challenging task in visual information retrieval, which generates questions given a sequence of video frames. The existing methods mainly tackle the problem of single-turn video question generation, but single-turn conversation usually can't meet the needs of video information acquisition. In this paper, we propose a new framework for single-turn VQG, which introduces attention mechanism to process inference of dialog history. And we introduce selection mechanism to choose from the candidate questions generated by each round of dialog history. In the framework, we leverage a recent video question answering model to predict the answer to the generated question and adopt the answer quality as rewards to fine-tune our model based on a reinforced learning mechanism. We also introduce a new task of multi-turn video question generation (M-VQG), which is generating multiple questions based on dialog history and video information to build conversation step by step. Our method achieves the state-of-the-art performance of the single-turn VQG task on two large-scale datasets, YouTube-Clips and TACoS-MultiLevel, and provides a baseline approach for M-VQG task.

FAST AND ACCURATE MULTI-FRAME SUPER-RESOLUTION OF SATELLITE IMAGES

Abstract. Recent constellations of small satellites, such as Planet’s SkySats, offer new acquisition modes where very short videos or bursts of images are acquired instead of a single still image. Compared to sequences of multi-date images, these sequences of consecutive video frames yield a large redundancy of information within the range of seconds. This redundancy enables to increase the spatial resolution using multi-frame super-resolution algorithms. In this paper, we propose a novel super-resolution method based on a high-order spline interpolation model that combines multiple low-resolution frames to produce a high-resolution image. Moreover this method can be implemented efficiently on GPU to process entire images from real satellite acquisitions. Synthetic and real experiments show that the proposed method is able to recover fine details, and measurements of the resulting resolution indicate a gain of 10 cm / pixel with respect to Planet’s SkySat standard imagery products.

Blood Vessel Visualization Method in Human Skin Based on Video Recording of Blood Flow Using a Laparoscope

The possibility of increasing the reliability of early diagnoses of diseases associated with a lesion of the capillary network morphology in human skin is considered. A method for visualizing the capillary network of blood vessels in the studied area of the skin is proposed. This method includes a probe optical-electronic system of an original design. A prototype of a probe optical-electronic video capillaroscopy (VCS) system, measurement technique, and data processing methods have been developed to study the parameters of the capillary network. A method that calculates the capillary map in the studied area, based on the analysis of the recorded sequence of video frames of the capillary blood flow is described. The effectiveness of the method is confirmed experimentally by examining the capillary network in the forearm, hands, and nail bed.

Dynamic mode decomposition via dictionary learning for foreground modeling in videos

Accurate extraction of foregrounds in videos is one of the challenging problems in computer vision. In this study, we propose dynamic mode decomposition via dictionary learning (dl-DMD), which is applied to extract moving objects by separating the sequence of video frames into foreground and background information with a dictionary learned using block patches on the video frames. Dynamic mode decomposition (DMD) decomposes spatiotemporal data into spatial modes, each of whose temporal behavior is characterized by a single frequency and growth/decay rate and is applicable to split a video into foregrounds and the background when applying it to a video. And, in dl-DMD, DMD is applied on coefficient matrices estimated over a learned dictionary, which enables accurate estimation of dynamical information in videos. Due to this scheme, dl-DMD can analyze the dynamics of respective regions in a video based on estimated amplitudes and temporal evolution over patches. The results on synthetic data exhibit that dl-DMD outperforms the standard DMD and compressed DMD (cDMD) based methods. Also, the results of an empirical performance evaluation in the case of foreground extraction from videos using publicly available dataset demonstrates the effectiveness of the proposed dl-DMD algorithm and achieves a performance that is comparable to that of the state-of-the-art techniques in foreground extraction tasks.

Block Based Motion Estimation Algorithms: Analysis

several video coding standards and techniques have been introduced for multimedia applications, particularly h.26x series for video processing. These standards employ motion estimation process in order to reduce the amount of data that is required to store or transmit the video. Motion estimation process is an inextricable part of the video coding as it removes the temporal redundancy between successive frames of video sequences. This paper is about these motion estimation algorithms, their search procedures, complexity, advantages, and limitations. A survey of motion estimation algorithms including full search algorithm, many fast search and fast full search block based algorithms has been presented. An evaluation of up to date motion estimation algorithms, based on a number of empirical results on several test video sequences, is presented as well.

Towards efficient motion-blurred public security video super-resolution based on back-projection networks

In public security, video super-resolution has been an important technology for assisting users in effectively locating potential safety hazards. However, the video collected from video monitoring equipment is usually polluted by motion blur and noise, which results in low video quality after reconstruction. In this paper, we propose a super-resolution reconstruction model for motion-blurred video to reconstruct corresponding high-resolution video from motion-blurred low-resolution video. First, a single-frame image super-resolution model is designed based on the improved deep back-projection network, which primarily includes the design of a recursive method. Second, a multi-frame image super-resolution model is designed based on an optical motion algorithm for motion estimation and motion compensation of video frame sequences. Finally, according to the motion-blurred low-resolution video pollution condition, the de-blurring algorithm and multi-frame image super-resolution are combined to propose a super-resolution reconstruction model for motion-blurred video. Experiments show that the proposed motion-blurred video super-resolution model is superior to other contrasted algorithms in clear video and motion-blurred video reconstruction and has a better reconstruction effect and higher practical value.

Predicting Thermal Runaway in Li-Ion Battery Employing Machine Learning Framework

Failures of lithium-ion batteries (LIBs) arise in both energetic and non-energetic ways. Both modes may occur due to numerous reasons such as cell manufacturing flaws, poor cell design (electrical or mechanical), external abuse of cells (thermal, mechanical, or electrical), defective protection electronics, chargers, etc. In this work, we present a machine learning (ML) study of the most severe energetic failure of the LIBs, which occurs due to the thermal runaway (TRA). In general, the TRA consists of a rapid self-heating of a cell sourced from exothermic (electro-) chemical reactions. During the TRA, the excessive heat has a detrimental influence on the electric characteristics and the state-of-charge (SOC) of the LIBs, which could lead to the accidental explosion of the battery. During the TRA, it is impossible to directly monitor the occurring events in practical operating conditions. However, the change of electric parameters during the TRA like-events could potentially indicate the existence of a failure, thus allowing to foresee the LIBs malfunction.We propose two deep learning techniques to address the problem of predicting the TRA like-events based upon the temporal variation of electric parameters, the SOC and the thermal images in overheating conditions. First, a recurrent neural network (RNN) built upon the long-short-term memory (LSTM) scheme is used to estimate the time dependency of the SOC on the voltage, current and temperature of a cell. In the second method the objective is to predict the presence of the TRA in the LIBs subjected to internal/external heating directly from thermal images collected as sequences of video frames during the battery operation. For this purpose, a Convolutional LSTM (Conv-LSTM) network is constructed and trained to distinguish the recorded thermal images among two classes, specifically the presence or absence of the TRA like-events. Conv-LSTM is a recurrent neural network designed to extract information from sequential images in time series analysis, which combines the image processing capabilities of convolutional neural networks (CNN) and the temporal tracking potential of LSTM. The main advantage of the proposed techniques is that they do not require the physical modeling of the internal battery chemistry, which is cumbersome in the TRA scenario, but they rely only on the experimental/ modeling data and the collected thermal images,

Learning Effective Skeletal Representations on RGB Video for Fine-Grained Human Action Quality Assessment

In this paper, we propose an integrated action classification and regression learning framework for the fine-grained human action quality assessment of RGB videos. On the basis of 2D skeleton data obtained per frame of RGB video sequences, we present an effective representation of joint trajectories to train action classifiers and a class-specific regression model for a fine-grained assessment of the quality of human actions. To manage the challenge of view changes due to camera motion, we develop a self-similarity feature descriptor extracted from joint trajectories and a joint displacement sequence to represent dynamic patterns of the movement and posture of the human body. To weigh the impact of joints for different action categories, a class-specific regression model is developed to obtain effective fine-grained assessment functions. In the testing stage, with the supervision of the action classifier’s output, the regression model of a specific action category is selected to assess the quality of skeleton motion extracted from the action video. We take advantage of the discrimination of the action classifier and the viewpoint invariance of the self-similarity feature to boost the performance of the learning-based quality assessment method in a realistic scene. We evaluate our proposed method using diving and figure skating videos of the publicly available MIT Olympic Scoring dataset, and gymnastic vaulting videos of the recent benchmark University of Nevada Las Vegas (UNLV) Olympic Scoring dataset. The experimental results show that the proposed method achieved an improved performance, which is measured by the mean rank correlation coefficient between the predicted regression scores and the ground truths.

Graph Embedding Multi-Kernel Metric Learning for Image Set Classification With Grassmannian Manifold-Valued Features

In the domain of video-based image set classification, a considerable advance has been made by modeling a sequence of video frames (image set) as a linear subspace, which typically resides on a Grassmannian manifold. As a consequence of the large intra-class variations of the video data, there are two open challenges for the modeling task: how to establish appropriate image set models to encode these variations, and how to effectively measure the similarity between any two image sets. As a possible way to tackle these issues, this paper presents a graph embedding multi-kernel metric learning (GEMKML) algorithm for image set classification. The proposed GEMKML implements set modeling, feature extraction, and classification in two steps. Firstly, the proposed framework constructs a novel cascaded feature learning architecture on Grassmannian manifold with the aim of producing more effective Grassmannian manifold-valued feature representations. To make a better use of these learned features, a graph embedding multi-kernel metric learning scheme is then devised to map them into a lower-dimensional Euclidean space, where the inter-class distances are maximized and the intra-class distances are minimized. We evaluate the proposed GEMKML on five different visual classification tasks using widely adopted datasets. The extensive classification results confirm its superiority over the state-of-the-art methods.

A comparison on visual prediction models for MAMO (multi activity-multi object) recognition using deep learning

Multi activity-multi object recognition (MAMO) is a challenging task in visual systems for monitoring, recognizing and alerting in various public places, such as universities, hospitals and airports. While both academic and commercial researchers are aiming towards automatic tracking of human activities in intelligent video surveillance using deep learning frameworks. This is required for many real time applications to detect unusual/suspicious activities like tracking of suspicious behaviour in crime events etc. The primary purpose of this paper is to render a multi class activity prediction in individuals as well as groups from video sequences by using the state-of-the-art object detector You Look only Once (YOLOv3). By optimum utilization of the geographical information of cameras and YOLO object detection framework, a Deep Landmark model recognize a simple to complex human actions on gray scale to RGB image frames of video sequences. This model is tested and compared with various benchmark datasets and found to be the most precise model for detecting human activities in video streams. Upon analysing the experimental results, it has been observed that the proposed method shows superior performance as well as high accuracy.