Optical Flow Model Research Articles

Neural network‐based optical flow versus traditional optical flow techniques with thermal aerial imaging in real‐world settings

AbstractThe study explores the feasibility of optical flow‐based neural network from real‐world thermal aerial imagery. While traditional optical flow techniques have shown adequate performance, sparse techniques do not work well during cold‐soaked low‐contrast conditions, and dense algorithms are more accurate in low‐contrast conditions but suffer from the aperture problem in some scenes. On the other hand, optical flow from convolutional neural networks has demonstrated good performance with strong generalization from several synthetic public data set benchmarks. Ground truth was generated from real‐world thermal data estimated with traditional dense optical flow techniques. The state‐of‐the‐art Recurrent All‐Pairs Field Transform for the Optical Flow model was trained with both color synthetic data and the captured real‐world thermal data across various thermal contrast conditions. The results showed strong performance of the deep‐learning network against established sparse and dense optical flow techniques in various environments and weather conditions, at the cost of higher computational demand.

TransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers.

Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, the first end-to-end video object detection system based on simple yet effective spatial-temporal Transformer architectures. The first goal of this paper is to streamline the pipeline of current VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow model, relation networks. Besides, benefited from the object query design in DETR, our method does not need post-processing methods such as Seq-NMS. In particular, we present a temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal transformer consists of two components: Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3 %-4 % mAP) on the ImageNet VID dataset. TransVOD yields comparable performances on the benchmark of ImageNet VID. Then, we present two improved versions of TransVOD including TransVOD++ and TransVOD Lite. The former fuses object-level information into object query via dynamic convolution while the latter models the entire video clips as the output to speed up the inference time. We give detailed analysis of all three models in the experiment part. In particular, our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0 % mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7 % mAP while running at around 30 FPS on a single V100 GPU device. Code and models are available at https://github.com/SJTU-LuHe/TransVOD.

CLGAN: a generative adversarial network (GAN)-based video prediction model for precipitation nowcasting

Abstract. The prediction of precipitation patterns up to 2 h ahead, also known as precipitation nowcasting, at high spatiotemporal resolutions is of great relevance in weather-dependent decision-making and early warning systems. In this study, we are aiming to provide an efficient and easy-to-understand deep neural network – CLGAN (convolutional long short-term memory generative adversarial network) – to improve the nowcasting skills of heavy precipitation events. The model constitutes a generative adversarial network (GAN) architecture, whose generator is built upon a u-shaped encoder–decoder network (U-Net) and is equipped with recurrent long short-term memory (LSTM) cells to capture spatiotemporal features. The optical flow model DenseRotation and the competitive video prediction models ConvLSTM (convolutional LSTM) and PredRNN-v2 (predictive recurrent neural network version 2) are used as the competitors. A series of evaluation metrics, including the root mean square error, the critical success index, the fractions skill score, and object-based diagnostic evaluation, are utilized for a comprehensive comparison against competing baseline models. We show that CLGAN outperforms the competitors in terms of scores for dichotomous events and object-based diagnostics. A sensitivity analysis on the weight of the GAN component indicates that the GAN-based architecture helps to capture heavy precipitation events. The results encourage future work based on the proposed CLGAN architecture to improve the precipitation nowcasting and early warning systems.

Nowcast for cloud top height from Himawari‐8 data based on deep learning algorithms

AbstractCloud top height (CTH) indicates the vertical development of clouds. Intensely vertically developing clouds are usually accompanied by extreme weather systems and pose a threat to aviation safety. Therefore, nowcast for CTH is necessary and meaningful to guide aviation flights. In this study, we researched the nowcast for CTH (mainly within 0–2 h) based on deep learning algorithms. With Sichuan Province as the study area, we collected CTH data of Himawari‐8 satellite from 2018 to 2020. Convolutional‐long‐short‐term‐memory (ConvLSTM) and trajectory‐gated‐recurrent‐unit (TrajGRU) were used to build nowcast models in the encoder‐forecaster framework. The optical flow model and persistence were used as benchmarks. The results showed that the deep learning models did not have significant advantages over the benchmarks in the first 20 min. However, with increasing nowcast time, the nowcast skills of the deep learning models were gradually exhibited. For all four seasons, the TrajGRU‐based model showed superior performance over the ConvLSTM‐based model and the benchmarks. In spring, autumn and winter, the results yielded by the ConvLSTM‐based model were second only to those of the TrajGRU‐based model. However, in summer, the ConvLSTM‐based model did not outperform the persistence. The results of the optical flow model worsened significantly with increasing nowcast time. In contrast to the persistence, the optical flow model had almost no nowcast skills after 40 min.

Motion Estimation for Complex Fluid Flows Using Helmholtz Decomposition

In this paper, we proposed a novel motion model with Helmholtz decomposition for complex fluid flows in a filtering-based optical flow framework, where the optimization of the regularization term is treated as a filtering process, and different motion patterns can be captured by finding appropriate filter kernels based on the designed filters. In this framework, we introduce a novel optical flow method with a joint spatial filter, which is based on the Helmholtz decomposition theorem that assumes a local motion field is composed of a curl field and a divergence field. By adjusting the scale of the weights in the filter kernels and combining a curl filter with a divergence filter in a certain ratio, it can simulate different motion patterns. In addition, if the correlation between the horizontal and vertical components of the optical flow field in the filter kernel is eliminated, it will be transformed into a linear motion model. Based on this linear motion model, we also develop a novel optical flow method with an adaptive guided filter. By finding an adaptive filter kernel driven by both the input image and the guided motion field for the designed filter, it can successfully capture different motion patterns, and yield an edge-preserving smoothing optical flow field. Most importantly, the proposed optical flow model provides a new way to design the regularizers for capturing different motion patterns in complex fluid flow. In particular, the designed optical flow method with an adaptive guided filter significantly outperforms the current state-of-the-art optical flow methods in predicting complex fluid flows.

Deep dual recurrence optical flow learning for time-resolved particle image velocimetry

Motion fields estimated from image data have been widely used in physics and engineering. Time-resolved particle image velocimetry (TR-PIV) is considered as an advanced flow visualization technique that measures multi-frame velocity fields from successive images. Contrary to conventional PIV, TR-PIV essentially estimates a velocity field video that provides both temporal and spatial information. However, performing TR-PIV with high computational efficiency and high computational accuracy is still a challenge for current algorithms. To solve these problems, we put forward a novel deep learning network named Deep-TRPIV in this study, to effectively estimate fluid motions from multi-frame particle images in an end-to-end manner. First, based on particle image data, we modify the optical flow model known as recurrent all-pairs field transforms that iteratively updates flow fields through a convolutional gated recurrent unit. Second, we specifically design a temporal recurrent network architecture based on this optical flow model by conveying features and flow information from previous frame. When N successive images are fed, the network can efficiently estimate N – 1 motion fields. Moreover, we generate a dataset containing multi-frame particle images and true fluid motions to train the network supervised. Eventually, we conduct extensive experiments on synthetic and experimental data to evaluate the performance of the proposed model. Experimental evaluation results demonstrate that our proposed approach achieves high accuracy and computational efficiency, compared with classical approaches and related deep learning models.

Video Saliency Object Detection with Motion Quality Compensation

Video saliency object detection is one of the classic research problems in computer vision, yet existing works rarely focus on the impact of input quality on model performance. As optical flow is a key input for video saliency detection models, its quality significantly affects model performance. Traditional optical flow models only calculate the optical flow between two consecutive video frames, ignoring the motion state of objects over a period of time, leading to low-quality optical flow and reduced performance of video saliency object detection models. Therefore, this paper proposes a new optical flow model that improves the quality of optical flow by expanding the flow perception range and uses high-quality optical flow to enhance the performance of video saliency object detection models. Experimental results on the datasets show that the proposed optical flow model can significantly improve optical flow quality, with the S-M values on the DAVSOD dataset increasing by about 39%, 49%, and 44% compared to optical flow models such as PWCNet, SpyNet, and LFNet. In addition, experiments that fine-tuning the benchmark model LIMS demonstrate that improving input quality can further improve model performance.

Computer Vision-based Fire Detection using Enhanced Chromatic Segmentation and Optical Flow Model

Computer Vision-based Fire Detection using Enhanced Chromatic Segmentation and Optical Flow Model

Optical Flow Computation for Video Under the Dynamic Illumination

Optical flow computation for video under the dynamic illumination is a challenging issue in video multimedia applications. In this paper, we solve this issue by introducing an illumination-invariant framework for variational optical flow estimation. It consists of an illumination-invariance model that handles complex illumination changes and a data enhancement model that guarantees highly accurate optical flow estimation. In this framework, we design a log-correlation descriptor for the data term, which handles complex illumination changes by eliminating the common parameters shared by the neighboring pixels in the corresponding illumination model while improving the accuracy of optical flow estimation by enhancing the discriminability of the data term matching. We also introduce a novel optical flow model with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{0}$</tex-math></inline-formula> regularizer, which reconstructs optical flow field by a sparse flow gradient counting scheme. Different from other edge-preserving regularizers, it does not depend on local motion features, but locates important flow edges globally. Therefore, it will not cause edge blurriness due to avoiding local filtering or average operation. It is particularly effective for enhancing major flow edges while eliminating a manageable degree of low-amplitude motion structures to control smoothing and reduce oversegmentation artifacts. Even small-scale motion structures with high contrast can be preserved remarkably well. The experimental results show our method significantly outperforms previous illumination-robust optical flow methods in handling complex illumination changes, and achieves competitive evaluation results on the challenging MPI-Sintel and Kitti datasets.

A FOREST FIRE IDENTIFICATION METHOD FOR UNMANNED AERIAL VEHICLE MONITORING VIDEO IMAGES

An automatic forest fire monitoring system based on UAV (unmanned aerial vehicle) acquired video images was studied in this paper. This novel method was proposed to address current problems in forest fire information monitoring practices such as poor real-time performance and low efficiency. Besides, it aims to realize the dynamic monitoring of forest fires in wild environment. In this paper, a forest fire monitoring method based on active analysis of UAV-acquired video image features is proposed to automatically detect and identify the occurrence of forest fires. The motion detection method based on dense optical flow and background modeling method were used to extract the motion regions for eliminating the influence of image background. By using wavelet energy feature and texture feature, 9 video images acquired by multi-rotor UAV on forest fire monitoring were selected as sample images (8 images for experiment and 1 image for contrast purpose). The mean values and standard deviations of the gray level co-occurrence matrix eigen values (angular second moment, entropy moment and reciprocal differential moment) were calculated as the discriminate basis for identifying forest fires. The experimental results showed that the proposed algorithm can effectively identify the forest fire, which provides a theoretical guarantee for the forest resources protection.

A Cloud-Based Deep Learning Framework for Early Detection of Pushing at Crowded Event Entrances

Crowding at the entrances of large events may lead to critical and life-threatening situations, particularly when people start pushing each other to reach the event faster. Automatic and timely identification of pushing behavior would help organizers and security forces to intervene early and mitigate dangerous situations. In this paper, we propose a cloud-based deep learning framework for automatic early detection of pushing in crowded event entrances. The proposed framework initially modifies and trains the EfficientNetV2B0 Convolutional Neural Network model. Subsequently, it integrates the adapted model with an accurate and fast pre-trained deep optical flow model with the color wheel method to analyze video streams and identify pushing patches in real-time. Moreover, the framework uses live capturing technology and a cloud-based environment to collect video streams of crowds in real-time and provide early-stage results. A novel dataset is generated based on five real-world experiments and their associated ground truth data to train the adapted EfficientNetV2B0 model. The experimental setups simulated a crowded event entrance, while the ground truths for each video experiment was generated manually by social psychologists. Several experiments on the videos and the generated dataset are carried out to evaluate the accuracy and annotation delay time of the proposed framework. The experimental results show that the proposed framework identified pushing behaviors with an accuracy rate of 87% within a reasonable delay time.

Subgrid Variational Optimized Optical Flow Estimation Algorithm for Image Velocimetry

The variational optical flow model is used in this work to investigate a subgrid-scale optimization approach for modeling complex fluid flows in image sequences and estimating their two-dimensional velocity fields. To solve the problem of lack of sub-grid small-scale structure information in variational optical flow estimation, we combine the motion laws of incompressible fluids. Introducing the idea of large eddy simulation, the instantaneous motion can be decomposed into large-scale motion and a small-scale turbulence in the data term. The Smagorinsky model is used to model and solve the small-scale turbulence. The improved subgrid scale Horn-Schunck (SGS-HS) optical flow algorithm provides better results in velocity field estimation of turbulent image sequences than the traditional Farneback dense optical flow algorithm. To make the SGS-HS algorithm equally competent for the open channel flow measurement task, a velocity gradient constraint is chosen for the canonical term of the model, which is used to improve the accuracy of the SGS-HS algorithm in velocimetric experiments in the case of the relatively uniform flow direction of the open channel flow field. The experimental results show that our algorithm has better performance in open channel velocimetry compared with the conventional algorithm.

TSRC: A Deep Learning Model for Precipitation Short-Term Forecasting over China Using Radar Echo Data

Currently, most deep learning (DL)-based models for precipitation forecasting face two conspicuous issues: the smoothing effect in the precipitation field and the degenerate effect of forecasting precipitation intensity. Therefore, this study proposes “time series residual convolution (TSRC)”, a DL-based convolutional neural network for precipitation nowcasting over China with a lead time of 3 h. The core idea of TSRC is it compensates the current local cues with previous local cues during convolution processes, so more contextual information and less uncertain features would remain in deep networks. We use four years’ radar echo reflectivity data from 2017 to 2020 for model training and one year’s data from 2021 for model testing and compare it with two commonly used nowcasting models: optical flow model (OF) and UNet. Results show that TSRC obtains better forecasting performances than OF and UNet with a relatively high probability of detection (POD), low false alarm rate (FAR), small mean absolute error (MAE) and high structural similarity index (SSIM), especially at longer lead times. Meanwhile, the results of two case studies suggest that TSRC still introduces smoothing effects and slightly outperforms UNet at longer lead times. The most considerable result is that our model can forecast high-intensity radar echoes even for typhoon rainfall systems, suggesting that the degenerate effect of forecasting precipitation intensity can be improved by our model. Future works will focus on the combination of multi-source data and the design of the model’s architecture to gain further improvements in precipitation short-term forecasting.

SHEEPFEARNET: Sheep fear test behaviors classification approach from video data based on optical flow and convolutional neural networks

Determining the temperament related traits of sheep, such as the coping style with various stress factors such as people, a new environment and social isolation, is essential in terms of improving animal welfare and increasing productivity. The classification of sheep according to their behavioral responses to the mentioned stress factors is evaluated by objective or subjective methods by expert observers. However, visual examinations that rely on human observation are more likely to make mistakes and are time consuming. Therefore, it is important to make this process faster, easier and more reliable. The phenotypic and genetic heritability of temperament traits in sheep are examined using behavioral tests such as arena and isolation box. The spatial features of the temperament classes in these tests are generally similar. At the same time, since the behavior traits are composed of time series, defining or classifying these features with image-based approaches can present challenges. In this study, we propose a video-based approach to overcome this challenge, using videos of behavioral traits obtained from fear tests. In this approach, we used a combination of optical flow for capturing temporal features and convolutional neural networks for capturing spatial features. The experimental results show that, balanced datasets in terms of the number of sheep, the BOF-VGG19 model trained with the transfer learning method is 90%, the BOF-CovnLSTM model using ConvLSTM networks is 95%, and the BOF-CNN model using CNNs is 100%, were determined as the optical flow models that classify fear test behavior traits the most successfully. The success rate of UNB-CNN and B-CNN models trained on raw images was 70%. As a result, we obtained successful results in classifying behavioral traits in models trained with optical flow pre-processed data sets balanced in terms of sheep numbers. At the same time, using a combination of optical flow and convolutional neural networks in videos where spatial features between temperament classes are similar enhanced the classification accuracy of fear behavior traits by capturing temporal features.

Optical Flow in the Dark

Optical flow estimation in low-light conditions is a challenging task for existing methods and current optical flow datasets lack low-light samples. Even if the dark images are enhanced before estimation, which could achieve great visual perception, it still leads to suboptimal optical flow results because information like motion consistency may be broken during the enhancement. We propose to apply a novel training policy to learn optical flow directly from new synthetic and real low-light images. Specifically, first, we design a method to collect a new optical flow dataset in multiple exposures with shared optical flow pseudo labels. Then we apply a two-step process to create a synthetic low-light optical flow dataset, based on an existing bright one, by simulating low-light raw features from the multi-exposure raw images we collected. To extend the data diversity, we also include published low-light raw videos without optical flow labels. In our training pipeline, with the three datasets, we create two teacher-student pairs to progressively obtain optical flow labels for all data. Finally, we apply a mix-up training policy with our diversified datasets to produce low-light-robust optical flow models for release. The experiments show that our method can relatively maintain the optical flow accuracy as the image exposure descends and the generalization ability of our method is tested with different cameras in multiple practical scenes.

Robust visual odometry using sparse optical flow network

Deep learning-based visual odometry (VO) has recently attracted much attention. In performing computer vision tasks, deep models are often more robust than manually extracted features. Deep models, however, have limited reliability and generalization ability which constrain their application in the VO systems. The existing approaches to address these issues are often based on creating more realistic datasets or employing strategies such as unsupervised learning and online fine-tuning. In contrast to the previous research, here we tackle these issues from the model structure standpoint. We present a simple yet robust VO (namely SF-VO) based on an especially designed sparse optical flow network. We then show that this network becomes suitable for VO applications by decomposing the global optimization problem into a single-point optimization. Combining the network with a consistency verification module and a Perspective-n-Point (PnP) solver, we then form a frame-to-frame VO system using the traditional pose estimation pipeline. Extensive experiments show that the proposed VO system effectively generalizes to real scenes while only synthetic datasets are used in the training process. It is also shown that the proposed model also outperforms other deep learning-based methods with a model size of only 1.69 M. Comparisons with the state-of-art optical flow models and performing expansion experiments further confirm that the designed network demonstrates a higher level of generalization ability and is capable of being trained based on limited datasets.

User‐Controllable Latent Transformer for StyleGAN Image Layout Editing

AbstractLatent space exploration is a technique that discovers interpretable latent directions and manipulates latent codes to edit various attributes in images generated by generative adversarial networks (GANs). However, in previous work, spatial control is limited to simple transformations (e.g., translation and rotation), and it is laborious to identify appropriate latent directions and adjust their parameters. In this paper, we tackle the problem of editing the StyleGAN image layout by annotating the image directly. To do so, we propose an interactive framework for manipulating latent codes in accordance with the user inputs. In our framework, the user annotates a StyleGAN image with locations they want to move or not and specifies a movement direction by mouse dragging. From these user inputs and initial latent codes, our latent transformer based on a transformer encoder‐decoder architecture estimates the output latent codes, which are fed to the StyleGAN generator to obtain a result image. To train our latent transformer, we utilize synthetic data and pseudo‐user inputs generated by off‐the‐shelf StyleGAN and optical flow models, without manual supervision. Quantitative and qualitative evaluations demonstrate the effectiveness of our method over existing methods.

Motion detection in moving camera videos using background modeling and FlowNet

Real-time moving object detection is challenging for moving cameras due to the moving background. Many studies use homography matrix to compensate for global motion by warping the background model to the current frame. Then, the pixel difference between the current frame and the background model is used for background subtraction. Moving pixels are extracted by applying adaptive threshold and some post-processing techniques. On the other hand, deep learning-based dense optical flow can be efficient enough to extract the moving pixels, but it increases computational cost. This study proposes a method to enhance a classical background modeling method with deep learning-based dense optical flow. The main contribution of this paper is to propose a fusing algorithm for dense optical flow and background modeling approach. The background modeling methods are error-prone, especially with continuous camera movement, while the optical flow method alone may not always be efficient. Our hybrid method fuses both techniques to improve the detection accuracy. We propose a software architecture to run background modeling and dense optical flow methods in parallel processes. The proposed implementation approach significantly increases the method’s working speed, while the proposed fusion and combining strategy improve detection results. The experimental results show that the proposed method can run at high speed and has satisfying performance against the methods in the literature.

Semantic-based visual emotion recognition in videos-a transfer learning approach

&lt;p&gt;&lt;span&gt;Automatic emotion recognition is active research in analyzing human’s emotional state over the past decades. It is still a challenging task in computer vision and artificial intelligence due to its high intra-class variation. The main advantage of emotion recognition is that a person’s emotion can be recognized even if he is extreme away from the surveillance monitoring since the camera is far away from the human; it is challenging to identify the emotion with facial expression alone. This scenario works better by adding visual body clues (facial actions, hand posture, body gestures). The body posture can powerfully convey the emotional state of a person in this scenario. This paper analyses the frontal view of human body movements, visual expressions, and body gestures to identify the various emotions. Initially, we extract the motion information of the body gesture using dense optical flow models. Later the high-level motion feature frames are transferred to the pre-trained convolutional neural network (CNN) models to recognize the 17 various emotions in Geneva multimodal emotion portrayals (GEMEP) dataset. In the experimental results, AlexNet exhibits the architecture's effectiveness with an overall accuracy rate of 96.63% for the GEMEP dataset is better than raw frames and 94% for visual geometry group-19 VGG-19, and 93.35% for VGG-16 respectively. This shows that the dense optical flow method performs well using transfer learning for recognizing emotions.&lt;/span&gt;&lt;/p&gt;

See the wind: Wind scale estimation with optical flow and VisualWind dataset

Wind sensing by learning from video clips could empower cameras to sense the wind scale and significantly improve the spatiotemporal resolution of existing professional weather records that are often at the city scale. Humans can interpret the wind scale from the motion of surrounding environment objects, especially the moving dynamics of trees in the wind. The goal of this paper is to train cameras to sense the wind by capturing such motion information using optical flow and machine learning models. To this end, we introduce a novel video dataset of over 6000 labeled video clips, covering eleven wind classes of the Beaufort scale. The videos are collected from social media, pubic cameras, and self-recording with varying numbers of clips in each class. Every video clip has a length of 10 s with varied frame rates, and contains scenes of various trees swaying in different scales of wind from an approximately fixed viewpoint. The variation in scene over the course of a single video is minimal. We propose a dual-branch model to estimate the wind scale including a motion branch, which uses optical flow to extract the tree movement dynamics, and a visual branch, to provide visual complementary clues for wind scale estimation. The two branches are fused adaptively in the end of the network, achieving 86.69 % accuracy which confirms the effectiveness of the proposed method. We have conducted experiments compared with a two-stage baseline model and a model only consisting of the motion branch, achieving the best accuracy with the potential to significantly improve the efficiency in time and storage. The dataset and the code are publicly accessible online.