Optical Flow Map Research Articles

Pix2Pix-Based Monocular Depth Estimation for Drones with Optical Flow on AirSim.

In this work, we propose a method for estimating depth for an image of a monocular camera in order to avoid a collision for the autonomous flight of a drone. The highest flight speed of a drone is generally approximate 22.2 m/s, and long-distant depth information is crucial for autonomous flights since if the long-distance information is not available, the drone flying at high speeds is prone to collisions. However, long-range, measurable depth cameras are too heavy to be equipped on a drone. This work applies Pix2Pix, which is a kind of Conditional Generative Adversarial Nets (CGAN). Pix2Pix generates depth images from a monocular camera. Additionally, this work applies optical flow to enhance the accuracy of depth estimation. In this work, we propose a highly accurate depth estimation method that effectively embeds an optical flow map into a monocular image. The models are trained with taking advantage of AirSim, which is one of the flight simulators. AirSim can take both monocular and depth images over a hundred meter in the virtual environment, and our model generates a depth image that provides the long-distance information than images captured by a common depth camera. We evaluate accuracy and error of our proposed method using test images in AirSim. In addition, the proposed method is utilized for flight simulation to evaluate the effectiveness to collision avoidance. As a result, our proposed method is higher accuracy and lower error than a state of work. Moreover, our proposed method is lower collision than a state of work.

A Novel Multi-Modal Network-Based Dynamic Scene Understanding

In recent years, dynamic scene understanding has gained attention from researchers because of its widespread applications. The main important factor in successfully understanding the dynamic scenes lies in jointly representing the appearance and motion features to obtain an informative description. Numerous methods have been introduced to solve dynamic scene recognition problem, nevertheless, a few concerns still need to be investigated. In this article, we introduce a novel multi-modal network for dynamic scene understanding from video data, which captures both spatial appearance and temporal dynamics effectively. Furthermore, two-level joint tuning layers are proposed to integrate the global and local spatial features as well as spatial and temporal stream deep features. In order to extract the temporal information, we present a novel dynamic descriptor, namely, Volume Symmetric Gradient Local Graph Structure ( VSGLGS ), which generates temporal feature maps similar to optical flow maps. However, this approach overcomes the issues of optical flow maps. Additionally, Volume Local Directional Transition Pattern ( VLDTP ) based handcrafted spatiotemporal feature descriptor is also introduced, which extracts the directional information through exploiting edge responses. Lastly, a stacked Bidirectional Long Short-Term Memory ( Bi-LSTM ) network along with a temporal mixed pooling scheme is designed to achieve the dynamic information without noise interference. The extensive experimental investigation proves that the proposed multi-modal network outperforms most of the state-of-the-art approaches for dynamic scene understanding.

Three-Stream 3D deep CNN for no-Reference stereoscopic video quality assessment

• An end-to-end learning structure for the Stereoscopic Video Quality Assessment with three pipelines to extract the quality degradation from spatial, temporal, and disparity information. • A non-homogeneous data augmentation system that splits the stereoscopic video into small cube patches and removes the outliers from the created dataset using entropy criteria. • A robust deep depth quality-aware feature extractor used to extract motion and depth maps jointly and feed them synchronously to the deep network to extract quality discriminatory features. • Two popular datasets are used to validate the proposed learning architecture for Stereoscopic Video Quality Assessment and received state-of-the-art results in both datasets. Convolutional Neural Networks (CNNs) have achieved great success in learning computer vision tasks, particularly 3D CNNs, for extracting spatio-temporal features from the given videos. However, 3D CNNs have not been well-examined for the Stereoscopic Video Quality Assessment (SVQA). To our best knowledge, most of the state-of-the-art methods used the traditional hand-crafted feature extraction methods for the SVQA. Very few methods used the power of deep learning for SVQA, and they just considered the spatial information, ignoring the disparity and motion information. In this paper, we propose a No-Reference (NR) deep 3D CNN architecture that jointly focuses on spatial, disparity, and temporal information between consecutive frames. A 3-Stream 3D CNN, shortly 3S-3DCNN , by performing 3D CNNs, extracts features from spatial, motion, and depth channels to estimate the stereo video’s quality. It captures the degradations in the quality of the stereoscopic video in multiple dimensions. Firstly, the scene flow, which is the joint prediction of the optical flow and stereo disparity, is calculated. Then, the spatial information, optical flow, and disparity map of a given video are used as input to the 3S-3DCNN model. The extracted features are concatenated and utilized as inputs to the fully connected layers for doing the regression. We split the input videos into cube patches for data augmentation and remove the cubes that confuse our model from the training and testing sets. Two standard stereoscopic video quality assessment benchmarks of LFOVIAS3DPh2 and NAMA3DS1-COSPAD1 were used to evaluate our method. Experimental results show that our 3S-3DCNN method’s objective score significantly correlates with the subjective SVQ scores in multiple video datasets. The RMSE for NAMA3DS1-COSPAD1 dataset is 0.2757, which outperforms other methods by a large margin. The SROCC value for the blur distortion of the LFOVIAS3DPh2 dataset is more than 98%, indicating that the 3S-3DCNN is consistent with human visual perception.

Short-term Solar Power Prediction Learning Directly from Satellite Images With Regions of Interest

Developing solar power generation technology is an efficient approach to relieving the global environmental crisis. However, solar energy is an energy source with strong uncertainty, which restricts large-scale photovoltaic (PV) applications until accurate solar energy predictions can be achieved. PV power forecasting methods have been widely researched based on existing predictions of satellite-derived solar irradiance, whereas modeling cloud motion directly from satellite images is still a tough task. In this study, an end-to-end short-term forecasting model is proposed to take satellite images as inputs, and it can learn the cloud motion characteristics from stacked optical flow maps. In order to reduce the huge size of measurements, static regions of interest (ROIs) are scoped based on historical cloud velocities. With its well-designed deep learning architecture, the proposed model can output multi-step-ahead prediction results sequentially by shifting receptive attention to dynamic ROIs. According to comparisons with related studies, the proposed model outperforms persistence and derived methods, and enhances its learning capability relative to conventional learning models via the novel architecture. The model can be applied to PV plants or arrays in different areas, suitable for forecast horizons within three hours.

Micro-Expression Recognition Method Based on Spatial Attention Mechanism and Optical Flow Features

<p indent="0mm">In order to solve the locality problem of micro-expression movement, an automatic micro-expression recognition method based on spatial attention mechanism of deep learning and optical flow features of micro-expressions is proposed. First, the frame difference method is used to identify apex frame for some micro-expression samples lack of apex frame labels. Then, TV<italic>-L</italic>₁ optical flow method is used to extract the horizontal and vertical components map of optical flow between the onset frame and the apex frame of micro-expression, and the corresponding optical flow strain pattern map is derived according to the horizontal and vertical components of optical flow. The three optical flow maps are connected in the way of channel superposition to form an optical flow characteristic map of micro-expression. Finally, a kind of spatial attention unit with learnable parameters is designed in the convolutional neural network built by the Inception module, which makes the model pay more attention to the regions with micro-expression motion in the feature extraction process. In the spatial attention unit, two convolution kernels of 3×3 and 7×7 are used for spatial attention inference, so that the model can comprehensively consider the attention inference results of different scale convolution kernels. Experiments on the MEGC2019 comprehensive micro-expression datasets show that the accuracy of the method is 0.788, which is better than the existing automatic micro-expression recognition method.

A Driver's Visual Attention Prediction Using Optical Flow.

Motion in videos refers to the pattern of the apparent movement of objects, surfaces, and edges over image sequences caused by the relative movement between a camera and a scene. Motion, as well as scene appearance, are essential features to estimate a driver’s visual attention allocation in computer vision. However, the fact that motion can be a crucial factor in a driver’s attention estimation has not been thoroughly studied in the literature, although driver’s attention prediction models focusing on scene appearance have been well studied. Therefore, in this work, we investigate the usefulness of motion information in estimating a driver’s visual attention. To analyze the effectiveness of motion information, we develop a deep neural network framework that provides attention locations and attention levels using optical flow maps, which represent the movements of contents in videos. We validate the performance of the proposed motion-based prediction model by comparing it to the performance of the current state-of-art prediction models using RGB frames. The experimental results for a real-world dataset confirm our hypothesis that motion plays a role in prediction accuracy improvement, and there is a margin for accuracy improvement by using motion features.

3D amplified MRI (aMRI).

PurposeAmplified MRI (aMRI) has been introduced as a new method of detecting and visualizing pulsatile brain motion in 2D. Here, we improve aMRI by introducing a novel 3D aMRI approach.Methods3D aMRI was developed and tested for its ability to amplify sub‐voxel motion in all three directions. In addition, 3D aMRI was qualitatively compared to 2D aMRI on multi‐slice and 3D (volumetric) balanced steady‐state free precession cine data and phase contrast (PC‐MRI) acquired on healthy volunteers at 3T. Optical flow maps and 4D animations were produced from volumetric 3D aMRI data.Results3D aMRI exhibits better image quality and fewer motion artifacts compared to 2D aMRI. The tissue motion was seen to match that of PC‐MRI, with the predominant brain tissue displacement occurring in the cranial‐caudal direction. Optical flow maps capture the brain tissue motion and display the physical change in shape of the ventricles by the relative movement of the surrounding tissues. The 4D animations show the complete brain tissue and cerebrospinal fluid (CSF) motion, helping to highlight the “piston‐like” motion of the ventricles.ConclusionsHere, we introduce a novel 3D aMRI approach that enables one to visualize amplified cardiac‐ and CSF‐induced brain motion in striking detail. 3D aMRI captures brain motion with better image quality than 2D aMRI and supports a larger amplification factor. The optical flow maps and 4D animations of 3D aMRI may open up exciting applications for neurological diseases that affect the biomechanics of the brain and brain fluids.

Direct Video Frame Interpolation With Multiple Latent Encoders

We present a simple but effective video interpolation framework that can be applied to various types of videos including conventional videos and 360° videos. Our main idea is to predict the latent feature of an intermediate frame, through the latent feature encoders between encoder and decoder networks, without explicitly computing optical flow or depth maps. The latent feature encoders take latent features of input images and then predict the latent feature of a target image, i . e . an intermediate frame. Afterward, the decoder network reconstructs the target image from the latent feature. The proposed framework consists of fully convolutional networks, and it is therefore end-to-end trainable from scratch without requiring additional information except for consecutive frames. We experimentally verify the superiority of proposed method by comparing it to state-of-the-art methods with various types of datasets. Moreover, an ablation study is carried out to analyze the key components of the proposed method. Our proposed method performs interpolation in latent domain, it is advantageous to apply various video interpolation ( e . g . NIR and depth videos) without limiting the type of input data.

Beyond Learning: Back to Geometric Essence of Visual Odometry via Fusion-Based Paradigm

Camera-based motion estimation, also called visual odometry (VO), is a technique applied in the area of autonomous driving and simultaneous localization and mapping (SLAM), which is gaining huge popularity due to its simplicity and use of limited resources in localization. In this article, we propose to consider the depth and optical flow estimation as a joint learning task and then obtain ego motion based on geometric constraints. To be specific, we construct a network to learn dense optical flow and depth map simultaneously, where the scale of the prediction is aligned by an additional velocity supervision signal. Then, we formulate the uncertainty of the two main geometric methods and leverage covariance intersection (CI) fusion, allowing the ego motion to be efficiently predicted in an automodeling and compensation way, without manual intervention, postprocessing, or pose network training utilized before. Compared with recent approaches to similar tasks, our method requires neither complicated feature descriptor calculations nor expensive supplemental sensors. Moreover, our model flexibly adapts to both unsupervised and weakly supervised learning, which improves the feasibility of the framework in diverse settings. We conduct extensive experiments on several datasets representing driving and walking scenarios, and the results demonstrate a promising performance of the proposed method to state-of-the-art monocular ones.

Сell Population Dynamics Monitoring in Video Based on Integral Optical Flow and Motion Maps

A method for monitoring cell population movement in microscopic video-sequences based on integral optical flow and motion maps is proposed. Through adjustment and calibration of the optical system and averaging consecutive frames, high-quality subsequent images are obtained. Short-term dynamic characteristics are determined by optical flow. Based on optical flow, integral optical flow is calculated and used to create motion maps, and these maps are used to analyze and describe motions in any region of interest. Therefore, different types of cell movements, including directional motion, aggregation and dispersion can be identified. The proposed method does not require training, it can be used for situation monitoring and analysis, or as a component of comprehensive systems. Experiments performed on synthesized and real microscopic video images demonstrate the effectiveness of this method.

Full View Optical Flow Estimation Leveraged From Light Field Superpixel

In this paper, we present a full view optical flow estimation method for plenoptic imaging. Our method employs the structure delivered by the four-dimensional light field over multiple views making use of superpixels. These superpixels are four dimensional in nature and can be used to represent the objects in the scene as a set of slanted-planes in three-dimensional space so as to recover a piecewise rigid depth estimate. Taking advantage of these superpixels and the corresponding slanted planes, we recover the optical flow and depth maps by using a two-step optimization scheme where the flow is propagated from the central view to the other views in the imagery. We illustrate the utility of our method for depth and flow estimation making use of a dataset of synthetically generated image sequences and real-world imagery captured using a Lytro Illum camera. We also compare our results with those yielded by a number of alternatives elsewhere in the literature.

A Convolutional Neural Network for Compound Micro-Expression Recognition.

Human beings are particularly inclined to express real emotions through micro-expressions with subtle amplitude and short duration. Though people regularly recognize many distinct emotions, for the most part, research studies have been limited to six basic categories: happiness, surprise, sadness, anger, fear, and disgust. Like normal expressions (i.e., macro-expressions), most current research into micro-expression recognition focuses on these six basic emotions. This paper describes an important group of micro-expressions, which we call compound emotion categories. Compound micro-expressions are constructed by combining two basic micro-expressions but reflect more complex mental states and more abundant human facial emotions. In this study, we firstly synthesized a Compound Micro-expression Database (CMED) based on existing spontaneous micro-expression datasets. These subtle feature of micro-expression makes it difficult to observe its motion track and characteristics. Consequently, there are many challenges and limitations to synthetic compound micro-expression images. The proposed method firstly implemented Eulerian Video Magnification (EVM) method to enhance facial motion features of basic micro-expressions for generating compound images. The consistent and differential facial muscle articulations (typically referred to as action units) associated with each emotion category have been labeled to become the foundation of generating compound micro-expression. Secondly, we extracted the apex frames of CMED by 3D Fast Fourier Transform (3D-FFT). Moreover, the proposed method calculated the optical flow information between the onset frame and apex frame to produce an optical flow feature map. Finally, we designed a shallow network to extract high-level features of these optical flow maps. In this study, we synthesized four existing databases of spontaneous micro-expressions (CASME I, CASME II, CAS(ME)2, SAMM) to generate the CMED and test the validity of our network. Therefore, the deep network framework designed in this study can well recognize the emotional information of basic micro-expressions and compound micro-expressions.

Detection of motion-compensated frame-rate up-conversion via optical flow-based prediction residue

To increase the temporal continuity of low frame-rate videos, Motion-Compensated Frame-Rate Up-Conversion (MC-FRUC), which is a special frame based video editing manipulation, can be employed to synthesize new intermediate frame between two successive frames. However, MC-FRUC can also be utilized to fake high frame-rate videos causing the wastes of storage space or users misleading. Although existing MC-FRUC detectors can obtain desirable performance for uncompressed videos or compressed videos with high perceptual quality, theirs performance degrades when test videos are encoded with higher quantization parameters. To deal with this problem, this paper presents a new optical flow-based detection method. Our method is divided into four steps. First, the forward or backward optical flow map is estimated by optical flow algorithm between the candidate frame, and its previous frame or following frame, respectively. Second, the prediction residue is calculated on the basis of these estimated optical flow maps to eliminate the side effect of texture information in the non-motion regions and the motion regions. Third, Markov statistic features are extracted. Finally, these features are fed into the ensemble classifier for identification of interpolated frames. We test different types of MC-FRUC methods under higher quantization parameters, and the results have shown the effectiveness of the proposed method compared with the existing relative ones.

Wound Healing Monitoring by Video Sequence Using Integral Optical Flow

A method for monitoring dynamic changes of wound video sequence based on integral optical flow is proposed. In this method, frame-by-frame sequential analysis is performed first, and then integral optical flow and motion maps are constructed for the video sequence. Motion maps allow one to determine the rate of epithelialization of wounds in critical regions that are characterized by impaired tissue growth. The stages of tissue healing are also determined from motion maps. Dynamic characteristics of wound tissue changes are introduced and calculated.

Motion Sickness Prediction in Stereoscopic Videos using 3D Convolutional Neural Networks.

In this paper, we propose a three-dimensional (3D) convolutional neural network (CNN)-based method for predicting the degree of motion sickness induced by a 360° stereoscopic video. We consider the user's eye movement as a new feature, in addition to the motion velocity and depth features of a video used in previous work. For this purpose, we use saliency, optical flow, and disparity maps of an input video, which represent eye movement, velocity, and depth, respectively, as the input of the 3D CNN. To train our machine-learning model, we extend the dataset established in the previous work using two data augmentation techniques: frame shifting and pixel shifting. Consequently, our model can predict the degree of motion sickness more precisely than the previous method, and the results have a more similar correlation to the distribution of ground-truth sickness.

Robust Visual Tracking via Collaborative Motion and Appearance Model

In this paper, robust visual tracking scheme is achieved through a novel sparse tracking via collaborative motion and appearance (TCMA). A coarse-to-fine framework with both motion and holistic appearance information is taken into consideration. In coarse search, we employ an optical flow map for the generation of motion particles. A rough estimation of target image patch is obtained using $l_2$ -regularized least square method in coarse search stage. In fine search, a novel smooth term is proposed in the cost function to improve the robustness of the tracker. With this smooth term, the object appearance in the previous frame will also affect the calculation of sparse coefficient in the current frame. It allows the tracker involving temporal information between consecutive frames instead of only considering single frame appearance information as in the conventional sparse coding-based tracking algorithms. In order to reserve the original and latest appearance information simultaneously in the template, a quadratic-function-like weight allocation scheme combining with particle contributed histogrammic correlation is developed in the updating stage. Both qualitative and quantitative studies are conducted on a set of challenging image sequences. The superior performance over other state-of-the-art algorithms is verified through the experiment.

Multiple feature fusion in convolutional neural networks for action recognition

Action recognition is important for understanding the human behaviors in the video, and the video representation is the basis for action recognition. This paper provides a new video representation based on convolution neural networks (CNN). For capturing human motion information in one CNN, we take both the optical flow maps and gray images as input, and combine multiple convolutional features by max pooling across frames. In another CNN, we input single color frame to capture context information. Finally, we take the top full connected layer vectors as video representation and train the classifiers by linear support vector machine. The experimental results show that the representation which integrates the optical flow maps and gray images obtains more discriminative properties than those which depend on only one element. On the most challenging data sets HMDB51 and UCF101, this video representation obtains competitive performance.

Abnormal event detection and localization in crowded scenes based on PCANet

Detecting and localizing abnormal events in crowded scenes still remains a challenging task among computer vision community. An unsupervised framework is proposed in this paper to address the problem. Low-level features and optical flows (OF) of video sequences are extracted to represent motion information in the temporal domain. Moreover, abnormal events usually occur in local regions and are closely linked to their surrounding areas in the spatial domain. To extract high-level information from local regions and model the relationship in spatial domain, the first step is to calculate optical flow maps and divide them into a set of non-overlapping sub-maps. Next, corresponding PCANet models are trained using the sub-maps at same spatial location in the optical flow maps. Based on the block-wise histograms extracted by PCANet models, a set of one-class classifiers are trained to predict the anomaly scores of test frames. The framework is completely unsupervised because it utilizes only normal videos. Experiments were carried out on UCSD Ped2 and UMN datasets, and the results show competitive performance of this framework when compared with other state-of-the-art methods.

Fuzzy segmentation of video shots using hybrid color spaces and motion information

Video segmentation can be defined as the process of partitioning video into spatio-temporal objects that are homogeneous in some feature space, with the choice of features being very important to the success of the segmentation process. Fuzzy segmentation is a semi-automatic region-growing segmentation algorithm that assigns to each element in an image a grade of membership in an object. In this paper, we propose an extension of the multi-object fuzzy segmentation algorithm to segment pre-acquired color video shots. The color features are selected from channels belonging to different color models using two different heuristics: one that uses the correlations between the color channels to maximize the amount of information used in the segmentation process, and one that chooses the color channels based on the separation of the clusters formed by the seed spels for all possible color spaces. Motion information is also incorporated into the segmentation process by making use of dense optical flow maps. We performed experiments on synthetic videos, with and without noise, as well as on some real videos. The experiments show promising results, with the segmentations of real videos produced using hybrid color spaces being more accurate than the ones produced using three other color models. We also show that our method compares favorably to a state-of-the art video segmentation algorithm.

Phase-correlation guided area matching for realtime vision and video encoding

In computer vision and video encoding applications, one of the first and most important steps is to establish a pixel-to-pixel correspondence between two images of the same scene obtained at slightly different times or points of view. One of the most popular methods to find these correspondences, known as Area Matching, consists in performing a computationally intensive search for each pixel in the first image, around a neighborhood of the same pixel in the second image. In this work we propose a method which significantly reduces the search space to only a few candidates, and permits the implementation of real-time vision and video encoding algorithms which do not require specialized hardware such as GPU's or FPGA's. Theoretical and experimental support for this method is provided. Specifically, we present results from the application of the method to the realtime video compression and transmission, as well as the realtime estimation of dense optical flow and stereo disparity maps, where a basic implementation achieves up to 100 fps in a typical dual-core PC.