Optical Flow Computation Research Articles

Multi-Exposure Motion Estimation Based on Deep Convolutional Networks

In motion estimation, illumination change is always a troublesome obstacle, which often causes severely performance reduction of optical flow computation. The essential reason is that most of estimation methods fail to formalize a unified definition in color or gradient domain for diverse environmental changes. In this paper, we propose a new solution based on deep convolutional networks to solve the key issue. Our idea is to train deep convolutional networks to represent the complex motion features under illumination change, and further predict the final optical flow fields. To this end, we construct a training dataset of multi-exposure image pairs by performing a series of non-linear adjustments in the traditional datasets of optical flow estimation. Our multi-exposure flow networks (MEFNet) model consists of three main components: low-level feature network, fusion feature network, and motion estimation network. The former two components belong to the contracting part of our model in order to extract and represent the multi-exposure motion features; the third component is the expanding part of our model in order to learn and predict the high-quality optical flow. Compared with many state-of-the-art methods, our motion estimation method can eliminate the obstacle of illumination change and yield optical flow results with competitive accuracy and time efficiency. Moreover, the good performance of our model is also demonstrated in some multi-exposure video applications, like HDR (high dynamic range) composition and flicker removal.

A new road tracking method based on heading direction detection

Current research on road tracking is mostly based on the visual perception of road boundaries. In this paper, we propose a novel and general road tracker based on optical flow computation, which can be applied to most of road environments including the case of a lack of lane markings or road boundaries. When the heading direction of the vehicle and the road direction are identical, the focus of expansion (FOE) coincides with the road vanishing point (RVP). This is an important foundation for the subsequent heading direction departure decision. By comparing the relative positions of the estimated FOE and RVP, we can learn the traveling state of the vehicle. The experimental results show that the proposed tracker is a simple and efficient road tracking method.

A Method for Calculating the Amount of Movements to Estimate the Self-position of Manta Robots

In recent years, the demand of underwater investigation is increasing in the circumference of a dam, the environmental research of the shallow where approach by ship is difficult, etc. It is known, however, that for man, all over the sea, danger exists mostly, and prolonged diving has a bad influence to a human body. Then, the development of underwater exploration robots that investigate underwater instead of humans is expected. Among underwater exploration robots, it is known that robots imitating aquatic organisms have little influence on underwater environment. Therefore, at this laboratory, a Manta robot using propulsive mechanisms with pectoral fins was developed, imitating the pectoral fin of Manta. Although underwater environmental research needs a function for estimating the self-position, it is not mounted in this Manta robot. This paper explains the amount estimation of movements using optical flows. Especially, a gimbal mechanism is introduced to reduce the influence on the optical flow calculation by pitch motion of the Manta robot. Several experiments are conducted to demonstrate the usefulness of the proposed method.

Research of optical flow aided MEMS navigation based on convex optimization and ROF denoising

Because of its own advantages, micro air vehicles can accomplish reconnaissance / surveillance tasks in low altitude and indoor flight environments, with inertial / satellite integrated navigation system. As a useful supplement to its navigation system, optical flow can assist INS to update position and attitude information in some occasions without GPS assistent. Considering the low calculation precision of optical flow in unconstant illumination condition, optical flow navigation method based on ROF denoising is studied, convex optimization theory and dual principle is discussed, and image decomposition method in the light texture changes is researched on. The obtained texture part is not affected by illumination variations, and used to calculate the movement of field information of the image. Then, the movement information is further calculated for position and attitude of micro air vehichles. The experimental results show that the accuracy of optical flow calculation is improved by decomposing and denoising.

Guided Filtering: Toward Edge-Preserving for Optical Flow

Despite progress made in the accuracy and robustness of optical flow in past years, the problem of over-segmentation and the blurring of image edge and motion boundary caused by the illumination change, complex texture, large displacement, and motion occlusion still remain. Recently, we developed a guided filtering scheme for flow field estimation, which is implemented as an add-on optimal operation during the coarse-to-fine optical flow computation. In this paper, we first review the research progress in optical flow computation and discuss limitations of the currently popular median filtering heuristic for a flow field optimization. We then introduce a general formulation of the guided filtering and provide the detailed illustration. Furthermore, we explore the potential of the guided filtering optimization for the flow field estimation under the coarse-to-fine computing scheme. Finally, we modify some typical and state-of-the-art optical flow methods by applying the proposed guided filtering operation to the baseline models, and test the performances of the basic and developed models through the Middlebury, MPI-Sintel, and KITTI data. The experimental results demonstrate that the guided filtering scheme is able to preserve the image edges and motion boundaries, and to improve the accuracy and robustness of optical flow estimation.

Video Salient Object Detection via Fully Convolutional Networks.

This paper proposes a deep learning model to efficiently detect salient regions in videos. It addresses two important issues: 1) deep video saliency model training with the absence of sufficiently large and pixel-wise annotated video data and 2) fast video saliency training and detection. The proposed deep video saliency network consists of two modules, for capturing the spatial and temporal saliency information, respectively. The dynamic saliency model, explicitly incorporating saliency estimates from the static saliency model, directly produces spatiotemporal saliency inference without time-consuming optical flow computation. We further propose a novel data augmentation technique that simulates video training data from existing annotated image data sets, which enables our network to learn diverse saliency information and prevents overfitting with the limited number of training videos. Leveraging our synthetic video data (150K video sequences) and real videos, our deep video saliency model successfully learns both spatial and temporal saliency cues, thus producing accurate spatiotemporal saliency estimate. We advance the state-of-the-art on the densely annotated video segmentation data set (MAE of .06) and the Freiburg-Berkeley Motion Segmentation data set (MAE of .07), and do so with much improved speed (2 fps with all steps).This paper proposes a deep learning model to efficiently detect salient regions in videos. It addresses two important issues: 1) deep video saliency model training with the absence of sufficiently large and pixel-wise annotated video data and 2) fast video saliency training and detection. The proposed deep video saliency network consists of two modules, for capturing the spatial and temporal saliency information, respectively. The dynamic saliency model, explicitly incorporating saliency estimates from the static saliency model, directly produces spatiotemporal saliency inference without time-consuming optical flow computation. We further propose a novel data augmentation technique that simulates video training data from existing annotated image data sets, which enables our network to learn diverse saliency information and prevents overfitting with the limited number of training videos. Leveraging our synthetic video data (150K video sequences) and real videos, our deep video saliency model successfully learns both spatial and temporal saliency cues, thus producing accurate spatiotemporal saliency estimate. We advance the state-of-the-art on the densely annotated video segmentation data set (MAE of .06) and the Freiburg-Berkeley Motion Segmentation data set (MAE of .07), and do so with much improved speed (2 fps with all steps).

FPGA-Based Hardware Implementation of Real-Time Optical Flow Calculation

Optical flow calculation algorithms are hard to implement on the hardware level in real-time, due to their complexity and high computational load. Therefore, presented works in the literature focusing on the hardware implementation are limited. In this paper, we present a hierarchical block matching-based optical flow algorithm suitable for real-time hardware implementation. The algorithm estimates the initial optical flow with block matching and refines the vectors with local smoothness constraints in each level. We evaluate the proposed algorithm with novel data sets and provide results compared with the ground truth optical flow. Furthermore, we present a reconfigurable hardware architecture of the proposed algorithm for calculating the optical flow in real-time. The presented system can process $640\times 480$ resolution frames at 39 frames/s.

Efficiency of Bilateral Filter Application in Problems of Optical Flow Calculation

It is proposed to use weight coefficients of the bilateral filter to calculate the measure of similarity of image regions in optical flow calculation algorithms. The efficiency of using this measure is studied by an example of a three-dimensional recursive search algorithm. Weight coefficients are used for calculating the optical flow with subpixel accuracy by the Lucas–Kanade algorithm. A possibility of reducing the optical flow calculation error by means of using the proposed approaches is demonstrated by an example of processing of various types of images. A method of choosing the parameters of the weight functions of the bilateral filter is described and analyzed.

OpenOpticalFlow: An Open Source Program for Extraction of Velocity Fields from Flow Visualization Images

This paper describes OpenOpticalFlow, an open source optical flow program in Matlab for extraction of high-resolution velocity fields from various flow visualization images. This program is a useful tool for researchers to use the optical flow method in various flow measurements. The principles of the optical flow method are concisely described, including the physics-based optical flow equation, the variational solution, and errors. The central part of this paper is the descriptions of the main program, relevant subroutines and selection of the relevant parameters in optical flow computation. Examples are given to demonstrate the applications of the optical flow method.

Online Measurement of Lubricating Film Thickness in Slider-on-Disc Contact Based on Dichromatic Optical Interferometry

The present study mainly focuses on a new approach to achieve online measurement of lubricating film thickness in a slider-on-disc contact. The inclination of the slider is obtained by Fourier transform and frequency estimation of the interferogram of the sliding contact. An algorithm that is based on optical flow calculation and dynamic time warping technique is presented for tracking a characteristic point with constant film thickness on the 1-D modulated intensity curve. Online film thickness measurement can thus be achieved. This approach has been validated with known gap thickness of a stationary wedge. In addition, transient film thickness under unsteady speeds was measured with the present approach and some lubrication behaviours have been revealed.

Real‐time AdaBoost cascade face tracker based on likelihood map and optical flow

The authors present a novel face tracking approach where optical flow information is incorporated into a modified version of the Viola Jones detection algorithm. In the original algorithm, detection is static, as information from previous frames is not considered. In addition, candidate windows have to pass all stages of the classification cascade, otherwise they are discarded as containing no face. In contrast, the proposed tracker preserves information about the number of classification stages passed by each window. Such information is used to build a likelihood map, which represents the probability of having a face located at that position. Tracking capabilities are provided by extrapolating the position of the likelihood map to the next frame by optical flow computation. The proposed algorithm works in real time on a standard laptop. The system is verified on the Boston Head Tracking Database, showing that the proposed algorithm outperforms the standard Viola Jones detector in terms of detection rate and stability of the output bounding box, as well as including the capability to deal with occlusions. The authors also evaluate two recently published face detectors based on convolutional networks and deformable part models with their algorithm showing a comparable accuracy at a fraction of the computation time.

Full-field OCT technique for high speed event-based optical flow and particle tracking.

This article introduces a method to extract the speed and density of microparticles in real time at several kHz using an asynchronous event-based camera mounted on a full-field optical coherence tomography (FF-OCT) setup. These cameras detect significant amplitude changes, allowing scene-driven acquisitions. They are composed of an array of autonomously operating pixels. Events are triggered when an illuminance change at the pixel level is significant at 1μs time precision. The event-driven FF-OCT algorithm relies on a time-based optical flow computation to operate directly on incoming events and updates the estimation of velocity, direction and density while reducing both computation and data load. We show that for fast moving microparticles in a range of 0.4 - 6.5mm/s, the method performs faster and more efficiently than existing techniques in real time. The target application of this work is to evaluate erythrocyte dynamics at the microvascular level in vivo with a high temporal resolution.

Time-of-Travel Methods for Measuring Optical Flow on Board a Micro Flying Robot.

For use in autonomous micro air vehicles, visual sensors must not only be small, lightweight and insensitive to light variations; on-board autopilots also require fast and accurate optical flow measurements over a wide range of speeds. Using an auto-adaptive bio-inspired Michaelis–Menten Auto-adaptive Pixel (MAPix) analog silicon retina, in this article, we present comparative tests of two optical flow calculation algorithms operating under lighting conditions from to W·cm (i.e., from 0.2 to 12,000 lux for human vision). Contrast “time of travel” between two adjacent light-sensitive pixels was determined by thresholding and by cross-correlating the two pixels’ signals, with measurement frequency up to 5 kHz for the 10 local motion sensors of the MAPix sensor. While both algorithms adequately measured optical flow between 25 /s and 1000 /s, thresholding gave rise to a lower precision, especially due to a larger number of outliers at higher speeds. Compared to thresholding, cross-correlation also allowed for a higher rate of optical flow output (99 Hz and 1195 Hz, respectively) but required substantially more computational resources.

The differential method for calculation of cloud motion vectors

The problem ofcloud data retrieval for the periods between the fixed terms ofobservations from geostationary satellites is solved using the differential methods for the optical flow. To solve the problem, the possibility of using different methods for computing cloud motion vectors is investigated. The estimates of the accuracy of computed vectors based on artificially generated test sequences are presented. The test results reveal that the accuracy of the optical flow calculation with the modt fied Brox’s method excels the other mentioned methods. The software is implemented for creating the smooth animation of cloud movement using the modified Brox’s method.

Operational Application of Optical Flow Techniques to Radar-Based Rainfall Nowcasting

Hong Kong Observatory has been operating an in-house developed rainfall nowcasting system called “Short-range Warning of Intense Rainstorms in Localized Systems (SWIRLS)” to support rainstorm warning and rainfall nowcasting services. A crucial step in rainfall nowcasting is the tracking of radar echoes to generate motion fields for extrapolation of rainfall areas in the following few hours. SWIRLS adopted a correlation-based method in its first operational version in 1999, which was subsequently replaced by optical flow algorithm in 2010 and further enhanced in 2013. The latest optical flow algorithm employs a transformation function to enhance a selected range of reflectivity for feature tracking. It also adopts variational optical flow computation that takes advantage of the Horn–Schunck approach and the Lucas–Kanade method. This paper details the three radar echo tracking algorithms, examines their performances in several significant rainstorm cases and summaries verification results of multi-year performances. The limitations of the current approach are discussed. Developments underway along with future research areas are also presented.

Synthesizing the motion of the vocal folds using optical flow based techniques

Different playbacks have been proposed to synthesize the dynamical information of the vocal folds. However most of them rely on the delineation of the glottal gap using segmentation techniques which is a complex task and usually requires a manual supervision. In order to solve this issue, three new playbacks based on the optical flow computation are presented. Two of them, called Optical Flow Glottovibrogram and Glottal Optical Flow Waveform, analyze the global dynamics; and the remaining one, called Optical Flow Kymogram, analyzes the local dynamics. The reliability of the proposed playbacks is evaluated by comparison with traditional representations, showing a great correlation in shape with the traditional playbacks, and allowing the identification of the most important instants of time, such as closed-states and maximal opening. In addition, they provide complementary information to the common spatio-temporal representations, although the new playbacks are lightly blurred.

On a spatial-temporal decomposition of optical flow

In this paper we present a decomposition algorithm for computation of the spatial-temporal optical flow of a dynamic image sequence. We consider several applications, such as the extraction of temporal motion features and motion detection in dynamic sequences under varying illumination conditions, such as they appear for instance in psychological flickering experiments. For the numerical implementation we are solving an integro-differential equation by a fixed point iteration. For comparison purposes we use a standard time dependent optical flow algorithm, which in contrast to our method, constitutes in solving a spatial-temporal differential equation.

Quantitative Analysis of Intracellular Motility Based on Optical Flow Model.

Analysis of cell mobility is a key issue for abnormality identification and classification in cell biology research. However, since cell deformation induced by various biological processes is random and cell protrusion is irregular, it is difficult to measure cell morphology and motility in microscopic images. To address this dilemma, we propose an improved variation optical flow model for quantitative analysis of intracellular motility, which not only extracts intracellular motion fields effectively but also deals with optical flow computation problem at the border by taking advantages of the formulation based on L1 and L2 norm, respectively. In the energy functional of our proposed optical flow model, the data term is in the form of L2 norm; the smoothness of the data changes with regional features through an adaptive parameter, using L1 norm near the edge of the cell and L2 norm away from the edge. We further extract histograms of oriented optical flow (HOOF) after optical flow field of intracellular motion is computed. Then distances of different HOOFs are calculated as the intracellular motion features to grade the intracellular motion. Experimental results show that the features extracted from HOOFs provide new insights into the relationship between the cell motility and the special pathological conditions.

Light Field Flow Estimation Based on Occlusion Detection

Light field cameras have a wide area of applications, such as digital refocusing, scene depth information extraction and 3-D image reconstruction. By recording the energy and direction information of light field, they can well solve many technical problems that cannot be done by conventional cameras. An important feature of light field cameras is that a microlens array is inserted between the sensor and main lens, through which a series of sub-aperture images of different perspectives are formed. Based on this feature and the full-focus image acquisition technique, we propose a light-field optical flow calculation algorithm, which involves both the depth estimation and the occlusion detection and guarantees the edge-preserving property. This algorithm consists of three steps: 1) Computing the dense optical flow field among a group of sub-aperture images; 2) Obtaining a robust depth-estimation by initializing the light-filed optical flow using the linear regression approach and detecting occluded areas using the consistency; 3) Computing an improved light-field depth map by using the edge-preserving algorithm to realize interpolation optimization. The reliability and high accuracy of the proposed approach is validated by experimental results.

Object’s scale and rotation estimation using the clustering in tracking methods based on the optic flow calculation

Procedures of scale and rotation estimation of an object using the clustering are proposed. The pro-cedures are oriented to the application in tracking techniques based on sparse optic flow calculation. The new tracking method, which is called FCT (Flow Clustering Tracker) is developed using the mentioned procedures. The obtained results show its ability to achieve the compromise in robustness and accuracy with respect to the existing solutions. References 8, figures 5, tables. 2.