Large Displacement Optical Flow Research Articles

Motion estimation for large displacements and deformations

Large displacement optical flow is an integral part of many computer vision tasks. Variational optical flow techniques based on a coarse-to-fine scheme interpolate sparse matches and locally optimize an energy model conditioned on colour, gradient and smoothness, making them sensitive to noise in the sparse matches, deformations, and arbitrarily large displacements. This paper addresses this problem and presents HybridFlow, a variational motion estimation framework for large displacements and deformations. A multi-scale hybrid matching approach is performed on the image pairs. Coarse-scale clusters formed by classifying pixels according to their feature descriptors are matched using the clusters’ context descriptors. We apply a multi-scale graph matching on the finer-scale superpixels contained within each matched pair of coarse-scale clusters. Small clusters that cannot be further subdivided are matched using localized feature matching. Together, these initial matches form the flow, which is propagated by an edge-preserving interpolation and variational refinement. Our approach does not require training and is robust to substantial displacements and rigid and non-rigid transformations due to motion in the scene, making it ideal for large-scale imagery such as aerial imagery. More notably, HybridFlow works on directed graphs of arbitrary topology representing perceptual groups, which improves motion estimation in the presence of significant deformations. We demonstrate HybridFlow’s superior performance to state-of-the-art variational techniques on two benchmark datasets and report comparable results with state-of-the-art deep-learning-based techniques.

Image registration method using representative feature detection and iterative coherent spatial mapping for infrared medical images with flat regions

In the registration of medical images, nonrigid registration targets, images with large displacement caused by different postures of the human body, and frequent variations in image intensity due to physiological phenomena are substantial problems that make medical images less suitable for intensity-based image registration modes. These problems also greatly increase the difficulty and complexity of feature detection and matching for feature-based image registration modes. This research introduces an automatic image registration algorithm for infrared medical images that offers the following benefits: effective detection of feature points in flat regions (cold patterns) that appear due to changes in the human body’s thermal patterns, improved mismatch removal through coherent spatial mapping for improved feature point matching, and large-displacement optical flow for optimal transformation. This method was compared with various classical gold standard image registration methods to evaluate its performance. The models were compared for the three key steps of the registration process—feature detection, feature point matching, and image transformation—and the results are presented visually and quantitatively. The results demonstrate that the proposed method outperforms existing methods in all tasks, including in terms of the features detected, uniformity of feature points, matching accuracy, and control point sparsity, and achieves optimal image transformation. The performance of the proposed method with four common image types was also evaluated, and the results verify that the proposed method has a high degree of stability and can effectively register medical images under a variety of conditions.

Fast Video Saliency Detection via Maximally Stable Region Motion and Object Repeatability

Motion information is one important cue in unsupervised video salient object detection. In order to estimate motion in videos, most of the methods adopt time-consuming algorithms such as large displacement optical flow estimation (needs more than 8-40s with 640X480 size per frame), which leads to saliency detection with only 0.01-0.1 FPS speed and limits its application. In human visual system, the motion of one object is usually considered as a whole. Therefore, we need not compute the motion of each pixel. Instead, it is desirable to estimate the probability of each pixel belonging to a well identifiable object, which is proposed as maximally stable region (MSR) in recent work, and compute object-level motion. Motivated by this intuition, we firstly propose one fast object-level video motion model based on MSR, which only needs 49 ms for 640X480 size frame. Next, we present spatial-temporal boundary connectivity (BndCon) and spatial-temporal Minimum Barrier Distance (MBD) to estimate background probability and saliency. Then, we propose the repeatability saliency which means the frequency of the object recurs in all video sequences. Besides, we propose one simple yet effective method to combine our unsupervised method and deep learning model to further boost performance. Compared with the state-of-the-art unsupervised methods, our method shows significantly better performance with 12 FPS speed on normal CPU hardware.

IPDDF: an improved precision dense descriptor based flow estimation

Large displacement optical flow algorithms are generally categorised into descriptor-based matching and pixel-based matching. Descriptor-based approaches are robust to geometric variation, however they have inherent localisation precision limitation due to histogram nature. This work presents a novel method called improved precision dense descriptor flow (IPDDF). The authors introduce an additional pixel-based matching cost within an existing dense Daisy descriptor framework to improve the flow estimation precision. Pixel-based features such as pixel colour and gradient are computed on top of the original descriptor in the authors' matching cost formulation. The pixel-based cost only requires a light-weight pre-computation and can be adapted seamlessly into the matching cost formulation. The framework is built based on the Daisy Filter Flow work. In the framework, Daisy descriptor and a filter-based efficient flow inference technique, as well as a randomised fast patch match search algorithm, are adopted. Given the novel matching cost formulation, the framework enables efficiently solving dense correspondence field estimation in a high-dimensional search space, which includes scale and orientation. Experiments on various challenging image pairs demonstrate the proposed algorithm enhances flow estimation accuracy as well as generate a spatially coherent yet edge-aware flow field result efficiently.

Robust Visual Object Tracking With Multiple Features and Reliable Re-Detection Scheme

In recent years, correlation filter based trackers have seen widespread success because of their high efficiency and robustness. However, a single feature based tracker cannot deal with complex scenes such as serious occlusion, motion blur and illumination variation. In this paper, we develop a novel tracking method combining color feature, Hog feature and motion feature. The motion feature is estimated between adjacent frames by large displacement optical flow. Besides, in order to cope with boundary effect existing in traditional correlation filter based trackers, an adaptive cosine window is introduced in our method, which can highlight the target region, suppress the background region and enlarge search region. Meanwhile, a novel judge scheme combining Hog correlation response and color response is adopted to evaluate the reliability of tracking result. Finally, inverse sparse representation is presented to locate coarse positions of target in case of tracking failures. Extensive experiments on five famous tracking benchmarks including OTB100, TColor-128, UAVDT, UAV123 and VOT2016 demonstrate our proposed method outperform other sate-of-the-art methods in terms of robustness and accuracy.

Large Displacement Optical Flow Estimation Based on Robust Interpolation of Sparse Correspondences

Recently, the interpolation of correspondences method has been widely used in optical flow estimation, because it produces an accurate flow field and costs little runtimes. However, most of the existing matching-based optical flow methods are usually susceptible to non-rigid motion and large displacements. We propose in this article a large displacement optical flow estimation method based on robust interpolation of sparse correspondences, named Riscflow. First, we utilize the deep matching model to achieve an initial matching result of two consecutive frames, and then we exploit a grid-based motion statistics optimization scheme to remove the outliers from the initial matching field. Second, we propose a random forest-based motion boundary extraction model and construct a sparse-to-dense interpolation method by using the boundary information to prevent the dense matching field from edge-blurring. Third, we design a global optical flow estimation method by using an energy function to optimize the dense matching field. Finally, we respectively run the proposed method on the MPI-Sintel and UCF101 databases to conduct a comprehensive comparison with some state-of-the-art optical flow approaches including the variational methods, the matching-based methods, and the deep learning-based methods. The comparison results demonstrate that the proposed method has high accuracy and good robustness of optical flow estimation, and especially gains the benefit of edge-preserving under non-rigid motion and large displacements.

Deformed landmark fitting for sequential faces

Fitting facial landmarks on unconstrained videos is a challenging task with broad applications. At present, many methods of one-shot landmark fitting have been proposed with varying degrees of success. However, most of them are heavily sensitive to initializations and usually rely on offline-trained static models, which limit their performance on sequential images with extensive variations. Therefore, they usually can’t align the deformed face very well. To address these limitations, we propose a method of deformed landmark fitting (DLF) for sequential faces, which is designed based on active shape model (ASM) and deformation tracking/correction. This method overcomes the loss of consecutive information between frames, and makes full use of the motion variation information of video sequences in time and space dimensions. Firstly, the optical flow values of several possible deformation points on the face are calculated by the large displacement optical flow (LDOF) model, and the tracking of these points in the current frame are performed through the optical flow motion vector. Secondly, the initial shape of face in each frame is established by the locations of these deformation points and the global shape model in ASM algorithm. Finally, on the basis of initial shape, according to the guidance of local texture model in ASM algorithm, different correction strategies are applied to different landmarks for local search, and then each landmark is reasonably suppressed to obtain the ultimate results. Our DLF observably improves the fitting accuracy for deformed faces, and takes full advantage of the continuity among video sequences. Compared with some state-of-the-art landmarkers, extensive experiments on landmark fitting for sequential faces show that our DLF performs outstandingly in terms of accuracy and robustness.

An Analysis and Speedup of the FALDOI Method for Optical Flow Estimation

We present a detailed analysis of FALDOI, a large displacement optical flow method proposed by P.Palomares et al. This method requires a set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from the matches. It follows a two-step minimization method at the finest scale which is very robust to the outliers of the sparse matcher and can capture large displacements of small objects. The results shown in the original paper consistently outperformed the coarse-to-fine approaches and achieved good qualitative and quantitative performance on the standard optical flow benchmarks. In this paper we revise the proposed method and the changes done to significantly reduce its execution time while reporting nearly the same accuracy. Finally, we also compare it against the current state-of-the-art to assess its performance.

SemFlow: Semantic-Driven Interpolation for Large Displacement Optical Flow

This paper presents a semantic-guided interpolation scheme (SemFlow) to handle motion boundaries and occlusions in large displacement optical flow. The basic idea is to segment images into superpixels and estimate their homographies for interpolation. In order to ensure each superpixel can be approximated as a plane, a semantic-guided refinement method is introduced. Moreover, we put forward a homography estimation model weighted by the distance between each superpixel and its K-nearest neighbors. Our newly-proposed distance metric combines the texture and semantic information to find proper neighbors. Our homography model performs better than the original affine model, since it accords with the real world projection relationship. The experiments on KITTI dataset demonstrate that SemFlow outperforms other state-of-the-art methods, especially in solving the problem of large scale motions and occlusions.

Rapid handling of outliers in dense sampling descriptor correspondence fields

The sparse-to-dense approach is considered to be the standard method of capturing the long-range motion of small objects during large displacement optical flow. Despite progress in the matching and interpolation of this approach, little work has focused on improving the handling of outliers after dense sampling descriptor matching. We propose an improved grid-based statistical matching method that can quickly remove outliers without calculating backward flow. First, a multigrid statistical matching method is developed to remove the most outliers of the dense sampling descriptor correspondence field. Second, to improve the accuracy of outliers handling, the misjudgment match in the edge grid is corrected based on the statistical matching constraint. The results of extensive experiments on public optical flow datasets demonstrate the effectiveness of the proposed method.

Hierarchical coherency sensitive hashing and interpolation with RANSAC for large displacement optical flow

Nearest Neighbor Field (NNF) has shown excellent performance for large displacement optical flow estimation recently. However, it contains much noise and lacks of global constraint. In this paper we present an effective approach, named HCSH (Hierarchical Coherency Sensitive Hashing), which combines the coarse-to-fine scheme and random search strategy, to enable NNF to enjoy the inherent smooth of coarse-to-fine framework. Then besides the forward–backward check for NNF, we also use auto-correlation to remove the unreliable correspondences in flat regions, where the NNF is often considered ambiguous and the motion can be naturally recovered by the latter interpolation. Inspired by EpicFlow, we propose edge-aware interpolation (EAI-Flow) to filling the gaps by removing correspondence. RANSAC is introduced to improve the locality-weighted affine transformation estimation, with neighbor propagation of affine model to reduce required iterations and speed up the computation. Experimental validation shows that our approach outperforms the state-of-the-art with more accurate optical flow.

Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation.

Modern large displacement optical flow algorithms usually use an initialization by either sparse descriptor matching techniques or dense approximate nearest neighbor fields. While the latter have the advantage of being dense, they have the major disadvantage of being very outlier-prone as they are not designed to find the optical flow, but the visually most similar correspondence. In this article we present a dense correspondence field approach that is much less outlier-prone and thus much better suited for optical flow estimation than approximate nearest neighbor fields. Our approach does not require explicit regularization, smoothing (like median filtering) or a new data term. Instead we solely rely on patch matching techniques and a novel multi-scale matching strategy. We also present enhancements for outlier filtering. We show that our approach is better suited for large displacement optical flow estimation than modern descriptor matching techniques. We do so by initializing EpicFlow with our approach instead of their originally used state-of-the-art descriptor matching technique. We significantly outperform the original EpicFlow on MPI-Sintel, KITTI 2012, KITTI 2015 and Middlebury. In this extended article of our former conference publication we further improve our approach in matching accuracy as well as runtime and present more experiments and insights.

Detection of Atypical and Typical Infant Movements using Computer-based Video Analysis.

The diagnosis of cerebral palsy (CP) is difficult before 2 years of age. The general movements assessment (GMA) is a method for predicting CP from the spontaneous movements of infants in the first months of life. This assessment has shown high accuracy in predicting CP, but its use is limited by a lack of trained clinicians and its subjective nature. An objective and cost-effective alternative is the automatic videobased assessment of infant movements. Retrospective videos with clinical GMA outcomes were evaluated against eligibility criteria for the automatic analysis consisting of a skin model for segmentation and large displacement optical flow (LDOF) for motion tracking. Kinematic features were extracted to classify the movements as typical or atypical using different classification algorithms. Preliminary classification results obtained from the analysis of 127 videos of preterm infants showed up to 92% of accuracy in predicting CP. A computerbased assessment would provide clinicians with an objective tool for early diagnosis of CP, to facilitate early intervention and improve functional outcomes.

Detecting Glacier Surface Motion by Optical Flow

In this study, we assessed the feasibility of using optical flow, in particular, large displacement optical flow (<small>LDOF</small>) method as a possible solution to obtain surface movement data to derive ice flow velocities in a glacier. Tests were carried out at the Viedma Glacier, located at the South Patagonia Icefield, Argentina, where terrestrial monoscopic image sequences were acquired by a calibrated camera from April 2014 until April 2016. As for preprocessing, the Correlated Analysis method was implemented to avoid and minimize errors due to the measurable changes in lighting, shadows, clouds, and snow. The results show a flow field with a maximum surface velocity value of 3.5 m/d. The errors were minimized by averaging the image sequence results based on seasons, in which the Total Error Reconstruction yielded fairly good mean accuracy (0.36 m/d). In summary, it was demonstrated that <small>LDOF</small> can provide accurate and robust solution to detect daily changes in the glacier surface.

Context-adaptive matching for optical flow

Modern sparse-to-dense optical flow estimation algorithms usually achieve state-of-art performance. Those algorithms need two steps: matching and interpolation. Matching is often unreliable for very large displacement optical flow due to illumination changes, deformations and occlusion etc. Moreover, conspicuous errors around motion discontinuities still keep serious as most methods consider edge only at interpolation step. The context-adaptive matching (CAM) is proposed for optical flow which is better at large displacement and edge preserving. The CAM is selective in feature extraction, adaptive in flow propagation and search radius adjusting. Selective features are proposed to consider edge preserving in matching step. Except for the usually used SIFT descriptor, the local directional pattern flow (LDPF) is introduced to keep more edge structure, and the oriented fast and rotated brief (ORB) is utilized to select out several most similar candidates. Unlike coarse-to-fine matching, which proposed a propagation step with only neighbors, we propose adaptive propagation to extend the matching candidates in order to improve the possibility of getting right correspondences. Furthermore, guided by prior knowledge and taking advantage of upper layers results, adaptive radius instead of constrained radius are proposed at finer layers. The CAM interpolated by EpicFlow is fast and robust for large displacements especially for fast moving objects and also preserves the edge structure well. Extensive experiments show that our algorithm is on par with the state-of-art optical flow methods on MPI-Sintel, KITTI and Middlebury.

Siamese network ensemble for visual tracking

Abstract Visual object tracking is a challenging task considering illumination variation, occlusion, rotation, deformation and other problems. In this paper, we extend a Siamese INstance search Tracker (SINT) with model updating mechanism to improve its tracking robustness. SINT uses convolutional neural network (CNN) features, and compares the new frame features with the target features in the first frame. The candidate region with the highest similarity score is considered as the tracking result. However, SINT is not robust against large target variation because the matching model is not updated during the whole tracking process. To combat this defect, we propose an Ensemble Siamese Tracker (EST), where the final similarity score is also affected by the similarity with tracking results in recent frames instead of solely considering the first frame. Tracking results in recent frames are used to adjust the model for continuous target change. Meanwhile, we combine large displacement optical flow method with EST to further improve the performance (called EST+). We test the proposed EST and EST+ on a standard tracking benchmark OTB. It turns out the average overlap ratio of EST and EST+ increase 2.72% and 3.55% respectively compared with SINT on OTB 2013, which contains 51 video sequences. For the OTB 100, the average overlap ratio gain is 4.2%.

FALDOI: A New Minimization Strategy for Large Displacement Variational Optical Flow

We propose a large displacement optical flow method that introduces a new strategy to compute a good local minimum of any optical flow energy functional. The method requires a given set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from those matches. In particular, the matches are used to guide a structured coordinate descent of the energy functional around these keypoints. It results in a two-step minimization method at the finest scale which is very robust to the inevitable outliers of the sparse matcher and able to capture large displacements of small objects. Its benefits over other variational methods that also rely on a set of sparse matches are its robustness against very few matches, high levels of noise, and outliers. We validate our proposal using several optical flow variational models. The results consistently outperform the coarse-to-fine approaches and achieve good qualitative and quantitative performance on the standard optical flow benchmarks.

OPTICAL FLOW APPLIED TO TIME-LAPSE IMAGE SERIES TO ESTIMATE GLACIER MOTION IN THE SOUTHERN PATAGONIA ICE FIELD

In this work, we assessed the feasibility of using optical flow to obtain the motion estimation of a glacier. In general, former investigations used to detect glacier changes involve solutions that require repeated observations which are many times based on extensive field work. Taking into account glaciers are usually located in geographically complex and hard to access areas, deploying time-lapse imaging sensors, optical flow may provide an efficient solution at good spatial and temporal resolution to describe mass motion. Several studies in computer vision and image processing community have used this method to detect large displacements. Therefore, we carried out a test of the proposed Large Displacement Optical Flow method at the Viedma Glacier, located at South Patagonia Icefield, Argentina. We collected monoscopic terrestrial time-lapse imagery, acquired by a calibrated camera at every 24 hour from April 2014 until April 2015. A filter based on temporal correlation and RGB color discretization between the images was applied to minimize errors related to changes in lighting, shadows, clouds and snow. This selection allowed discarding images that do not follow a sequence of similarity. Our results show a flow field in the direction of the glacier movement with acceleration in the terminus. We analyzed the errors between image pairs, and the matching generally appears to be adequate, although some areas show random gross errors related to the presence of changes in lighting. The proposed technique allowed the determination of glacier motion during one year, providing accurate and reliable motion data for subsequent analysis.

OPTICAL FLOW APPLIED TO TIME-LAPSE IMAGE SERIES TO ESTIMATE GLACIER MOTION IN THE SOUTHERN PATAGONIA ICE FIELD

Abstract. In this work, we assessed the feasibility of using optical flow to obtain the motion estimation of a glacier. In general, former investigations used to detect glacier changes involve solutions that require repeated observations which are many times based on extensive field work. Taking into account glaciers are usually located in geographically complex and hard to access areas, deploying time-lapse imaging sensors, optical flow may provide an efficient solution at good spatial and temporal resolution to describe mass motion. Several studies in computer vision and image processing community have used this method to detect large displacements. Therefore, we carried out a test of the proposed Large Displacement Optical Flow method at the Viedma Glacier, located at South Patagonia Icefield, Argentina. We collected monoscopic terrestrial time-lapse imagery, acquired by a calibrated camera at every 24 hour from April 2014 until April 2015. A filter based on temporal correlation and RGB color discretization between the images was applied to minimize errors related to changes in lighting, shadows, clouds and snow. This selection allowed discarding images that do not follow a sequence of similarity. Our results show a flow field in the direction of the glacier movement with acceleration in the terminus. We analyzed the errors between image pairs, and the matching generally appears to be adequate, although some areas show random gross errors related to the presence of changes in lighting. The proposed technique allowed the determination of glacier motion during one year, providing accurate and reliable motion data for subsequent analysis.

DeepMatching: Hierarchical Deformable Dense Matching

We introduce a novel matching algorithm, called DeepMatching, to compute dense correspondences between images. DeepMatching relies on a hierarchical, multi-layer, correlational architecture designed for matching images and was inspired by deep convolutional approaches. The proposed matching algorithm can handle non-rigid deformations and repetitive textures and efficiently determines dense correspondences in the presence of significant changes between images. We evaluate the performance of DeepMatching, in comparison with state-of-the-art matching algorithms, on the Mikolajczyk (Mikolajczyk et al. A comparison of affine region detectors, 2005), the MPI-Sintel (Butler et al. A naturalistic open source movie for optical flow evaluation, 2012) and the Kitti (Geiger et al. Vision meets robotics: The KITTI dataset, 2013) datasets. DeepMatching outperforms the state-of-the-art algorithms and shows excellent results in particular for repetitive textures. We also apply DeepMatching to the computation of optical flow, called DeepFlow, by integrating it in the large displacement optical flow (LDOF) approach of Brox and Malik (Large displacement optical flow: descriptor matching in variational motion estimation, 2011). Additional robustness to large displacements and complex motion is obtained thanks to our matching approach. DeepFlow obtains competitive performance on public benchmarks for optical flow estimation.