Disparity Estimation Research Articles

A Convolutional Attention Residual Network for Stereo Matching

Deep learning based on convolutional neural network (CNN) has been successfully applied to stereo matching, which has achieved greater improvement in speed and accuracy compared with traditional methods. However, existing CNN-based stereo matching frameworks frequently encounter two problems. First, the existing stereo matching network has a large number of parameters, which results in too long matching running time since excessive network width and excessive number of convolution kernels. Second, in some areas where reflection, refraction and fine structure may lead to ill-posed problems, the disparity estimation errors can be occurred. In this paper, we proposed a lightweight network, convolution attention residual network (CAR-Net), which can balance the real-time matching and matching accuracy for stereo matching. Besides, a multi-scale residual network called CBAM-ResNeXt, which combines attention, was proposed for features extraction. With an aim is to simplify the parameters of the network model by reducing the size of filters and to extract the semantic features such as categories and locations from the image through convolutional block attention module (CBAM). Here, the CBAM consists of channel attention module and spatial attention module, where the semantic information of the feature map can be fully maintained after the parameters were simplified. Moreover, we proposed a dimension-extended 3D-CBAM, which is connected to 3DCNN for cost aggregation. By combining these two sub-modules of attention, the network is guided to selectively focus on the foreground or background regions, so as to improve the disparity accuracy in the ill-posed regions. The experimental results showed that our proposed method generated high accuracy and optimized the velocity compared to the state-of-the-art benchmark on KITTI 2012, KITTI 2015 and Scene Flow.

A Lightweight Neural Network for Monocular View Generation With Occlusion Handling.

In this article, we present a very lightweight neural network architecture, trained on stereo data pairs, which performs view synthesis from one single image. With the growing success of multi-view formats, this problem is indeed increasingly relevant. The network returns a prediction built from disparity estimation, which fills in wrongly predicted regions using a occlusion handling technique. To do so, during training, the network learns to estimate the left-right consistency structural constraint on the pair of stereo input images, to be able to replicate it at test time from one single image. The method is built upon the idea of blending two predictions: a prediction based on disparity estimation and a prediction based on direct minimization in occluded regions. The network is also able to identify these occluded areas at training and at test time by checking the pixelwise left-right consistency of the produced disparity maps. At test time, the approach can thus generate a left-side and a right-side view from one input image, as well as a depth map and a pixelwise confidence measure in the prediction. The work outperforms visually and metric-wise state-of-the-art approaches on the challenging KITTI dataset, all while reducing by a very significant order of magnitude (5 or 10 times) the required number of parameters (6.5 M).

Disparity Estimation Using Stereo Images With Different Focal Lengths

This study proposes a disparity estimation method for images captured by cameras with different focal lengths. Whereas a short focal length camera captures a close scene, a long focal length camera captures a far scene. The proposed method computes full disparity maps for both the close and far scenes. To achieve this, we introduce a stereo rectification method that works directly on images with different focal lengths and parameterizes the zoom levels between the two cameras. To evaluate the proposed method, we set up a stereo system with two cameras with different focal lengths and captured sequences of stereo images. In addition, we used KITTI, EISATS, and SceneFlow stereo datasets to simulate images with different focal lengths. Experimental results show that the proposed disparity estimation method successfully computed disparity maps for both near and far scenes, with significantly better performance than state-of-the-art monocular disparity estimation methods. Source code and experimental results are available online at https://github.com/comvisdinh/disparityestimation .

A Hybrid Approach on Single Image Dehazing using Adaptive Gamma Correction

Since the last eruption of Mt. Kelud, a surveillance has been conducted using camera to observe the crater lake and its surroundings. The key point of this observation is based on the captured haze-free images. However, these images suffer from hazy condition because of degassing. Degassing obscures the camera vision so that the observer barely maintain to keep track of lake phenomenon. Furthermore, even many dehazing techniques have been proposed to tackle this such problems, it still leaves some other problems such as oversaturation, color distortion, and halo effects. In this paper, a hybrid method is proposed. This work incorporates two the state-of-the-art visibility restoration and contrast enhancement methods which are color attenuation prior (CAP) and adaptive gamma correction (AGC). This method is separated into three different modules. They are disparity estimation (DispE) module, transmission map enhancement (TME) module, and hazy image restoration module (ImRec). The DispE module adapts the color attenuation model for its powerful depth estimation for handling outdoor image. Subsequently, the TME module is enhanced using AGC. And the last module is calculated from the modified hazy model. The experimental results are quite impressive. It is able to maintain edge and minimize image distortion qualitatively. And the fog density measured with Fog-Aware Density Estimation (FADE) estimated at 1.448.

Texture discrimination-enforced matching cost computation and smoothness-weighted cost regularization for stereo matching

In stereo matching, energy-minimization optimizers such as dynamic programming (DP) could achieve efficient disparity estimation. However, DP-based optimizers still suffer from relatively high energy due to too many artificial penalty parameters and a complex energy function, inevitably leading to matching failure in depth discontinuities, occlusions, and low-textured regions. To reach dense high-accuracy disparity estimation, we propose an effective stereo-matching method using texture discrimination-enforced matching cost computation and smoothness-weighted cost regularization. First, pixel adjustment is introduced to the gradient difference calculation so that initial matching accuracy, especially in low-textured regions, can be improved. For determining the degree of pixel adjustment, a gray-gradient co-occurrence matrix and the fuzzy c-means method are adopted to differentiate between two varieties of reference images based on textural property. Second, the support weight in local methods and the smoothness constraints in global optimizations are leveraged. Color information and the confidence map of the input image are combined to form the support weight. By ameliorating the smoothness constraint function during weighted horizontal DP passes, the adverse effects of excessive penalty can be reduced. Third, a postprocessing method is proposed for refining results in both occluded and low-textured regions. The proposed method considers test cases from the Middlebury v.2 and v.3 datasets and can outperform the state-of-the-art stereo-matching algorithms.

Fast and Accurate 3D Measurement Based on Light-Field Camera and Deep Learning.

The precise combination of image sensor and micro-lens array enables light-field cameras to record both angular and spatial information of incoming light, therefore, one can calculate disparity and depth from one single light-field image captured by one single light-field camera. In turn, 3D models of the recorded objects can be recovered, which means a 3D measurement system can be built using a light-field camera. However, reflective and texture-less areas in light-field images have complicated conditions, making it hard to correctly calculate disparity with existing algorithms. To tackle this problem, we introduce a novel end-to-end network VommaNet to retrieve multi-scale features from reflective and texture-less regions for accurate disparity estimation. Meanwhile, our network has achieved similar or better performance in other regions for both synthetic light-field images and real-world data compared to the state-of-the-art algorithms.

Learning on the Edge: Investigating Boundary Filters in CNNs

Convolutional neural networks (CNNs) handle the case where filters extend beyond the image boundary using several heuristics, such as zero, repeat or mean padding. These schemes are applied in an ad-hoc fashion and, being weakly related to the image content and oblivious of the target task, result in low output quality at the boundary. In this paper, we propose a simple and effective improvement that learns the boundary handling itself. At training-time, the network is provided with a separate set of explicit boundary filters. At testing-time, we use these filters which have learned to extrapolate features at the boundary in an optimal way for the specific task. Our extensive evaluation, over a wide range of architectural changes (variations of layers, feature channels, or both), shows how the explicit filters result in improved boundary handling. Furthermore, we investigate the efficacy of variations of such boundary filters with respect to convergence speed and accuracy. Finally, we demonstrate an improvement of 5–20% across the board of typical CNN applications (colorization, de-Bayering, optical flow, disparity estimation, and super-resolution). Supplementary material and code can be downloaded from the project page: http://geometry.cs.ucl.ac.uk/projects/2019/investigating-edge/.

Unsupervised Dense Light Field Reconstruction with Occlusion Awareness

AbstractLight field (LF) reconstruction is a fundamental technique in light field imaging and has applications in both software and hardware aspects. This paper presents an unsupervised learning method for LF‐oriented view synthesis, which provides a simple solution for generating quality light fields from a sparse set of views. The method is built on disparity estimation and image warping. Specifically, we first use per‐view disparity as a geometry proxy to warp input views to novel views. Then we compensate the occlusion with a network by a forward‐backward warping process. Cycle‐consistency between different views are explored to enable unsupervised learning and accurate synthesis. The method overcomes the drawbacks of fully supervised learning methods that require large labeled training dataset and epipolar plane image based interpolation methods that do not make full use of geometry consistency in LFs. Experimental results demonstrate that the proposed method can generate high quality views for LF, which outperforms unsupervised approaches and is comparable to fully‐supervised approaches.

High-Throughput Hardware Design for 3D-HEVC Disparity Estimation

Editor’s note : Disparity refers to the fact, that, for example, human beings see an object from a different angle with the left eye and the right eye. The disparity es-timation in images produces a computationally intensive effort. The HEVC disparity estimation can be analyzed according to its efficient implementa-tion and high throughput. Disparity estimation is one crucial part of HEVC, and the optimization in terms of throughput affects, therefore, the complete realization on hardware. -- Michael Hubner, Brandenburg University of Technology

Joint view synthesis and disparity refinement for stereo matching

Typical stereo algorithms treat disparity estimation and view synthesis as two sequential procedures. In this paper, we consider stereo matching and view synthesis as two complementary components, and present a novel iterative refinement model for joint view synthesis and disparity refinement. To achieve the mutual promotion between view synthesis and disparity refinement, we apply two key strategies, disparity maps fusion and disparity-assisted plane sweep-based rendering (DAPSR). On the one hand, the disparity maps fusion strategy is applied to generate disparity map from synthesized view and input views. This strategy is able to detect and counteract disparity errors caused by potential artifacts from synthesized view. On the other hand, the DAPSR is used for view synthesis and updating, and is able to weaken the interpolation errors caused by outliers in the disparity maps. Experiments on Middlebury benchmarks demonstrate that by introducing the synthesized view, disparity errors due to large occluded region and large baseline are eliminated effectively and the synthesis quality is greatly improved.

PRECISE DISPARITY ESTIMATION FOR NARROW BASELINE STEREO BASED ON MULTISCALE SUPERPIXELS AND PHASE CORRELATION

Abstract. With the rapid development of subpixel matching algorithms, the estimation of image shifts with an accuracy of higher than 0.05 pixels is achieved, which makes the narrow baseline stereovision possible. Based on the subpixel matching algorithm using the robust phase correlation (PC), in this work, we present a novel hierarchical and adaptive disparity estimation scheme for narrow baseline stereo, which consists of three main steps: image coregistration, pixel-level disparity estimation, and subpixel refinement. The Fourier-Mellin transform with subpixel PC is used to co-register two input images. Then, the pixel-level disparities are estimated in an iterative manner, which is achieved through multiscale superpixels. The pixel-level PC is performed with the window sizes and locations adaptively determined according to superpixels, with the disparity values calcualted. Fast weighted median filtering based on edge-aware filter is adopted to refine the disparity results. At last, the accurate disparities are calculated via a robust subpixel PC method. The combination of multiscale superpixel hierarchy, adaptive determination of the window size and location of correlation, fast weighted median filtering and subpixel PC make the proposed scheme be able to overcome the issues of either low-texture areas or fattening effect. Experimental results on a pair of UAV images and the comparison with the fixed-window PC methods, the iterative scheme with fixed variation strategy, and a sophisticated implementation using global optimization demonstrate the superiority and reliability of the proposed scheme.

Energy-Aware Motion and Disparity Estimation System for 3D-HEVC With Run-Time Adaptive Memory Hierarchy

The popularization of multimedia services has pushed forward the development of 2D/3D video-capable embedded mobile devices. Such devices require efficient energy/memory-management strategies to deal with severe memory/processing requirements and limited energy supply. Therefore, we propose a motion and disparity estimation (ME and DE) system—the most memory/processing demanding encoding steps—for the 3D High Efficiency Video Coding (3D-HEVC) standard. It was designed for low energy consumption, featuring a run-time adaptive memory hierarchy. The processing unit employs flexible coding order and optimizations to reduce the computational effort by exploring the inter-channel and inter-view redundancies. The memory hierarchy features window-based prefetching, data reuse, subsampling, and dynamic voltage scaling controlled by our depth-based dynamic search window resizing algorithm. Memory results demonstrate an average on-chip energy reduction of 79% in comparison to the widely used Level-C solution for a 45-nm technology. The proposed energy-aware ME and DE system dissipates 7.55 W while processing three HD 1080p views (video + depth) at 30 frames per second and presents a mean energy consumption of 0.107 J per access unit. To the best of our knowledge, this is the first work that proposes a real-time ME/DE system for the 3D-HEVC standard with an adaptive memory hierarchy.

A Computational Model for the Neural Representation and Estimation of the Binocular Vector Disparity from Convergent Stereo Image Pairs.

The depth cue is a fundamental piece of information for artificial and living beings who interact with the surrounding environment in order to handle objects and to avoid obstacles: in such situations, the disparity patterns, which arise when agents fixate objects, are vector fields. We propose a biologically-inspired computational model to estimate dense horizontal and vertical disparity maps by exploiting the cortical paradigms of the primate visual system: in particular, we aim to model the disparity sensitivity of the V1-MT visual pathway. The proposed model is based on a first processing stage composed of a bank of spatial band-pass filters and a static nonlinearity, mimicking complex binocular cells. Then, subsequent pooling stages and decoding strategies allow the model to estimate the vector disparity, after having represented it as a population of MT-like units. We assess the proposed model by using standard benchmarking stereo images, the Middlebury dataset, and specific stereo images that have horizontal and vertical disparities, which characterize the stimuli produced by active vision systems. Moreover, we systemically analyze how the different processing stages affect the model performance, and we discuss their implications for the neural modeling.

Four-Dimensional Anisotropic Diffusion Framework With PDEs for Light Field Regularization and Inverse Problems

In this paper, we consider the problem of vector-valued regularization of light fields based on PDEs. We propose a regularization method operating in the four-dimensional (4-D) ray space that does not require prior estimation of disparity maps. The method performs a PDE-based anisotropic diffusion along directions defined by local structures in the 4-D ray space. We analyze light field regularization in the 4-D ray space using the proposed 4-D anisotropic diffusion framework by first considering a light field toy example, i.e., a tesseract. This simple light field example allows an in-depth analysis of how each eigenvector influences the diffusion process. We then illustrate the diffusion effect for several light field processing applications: denoising, angular, and spatial interpolation, regularization for enhancing disparity estimation as well as inpainting.

NIPM-sWMF: Toward Efficient FPGA Design for High-Definition Large-Disparity Stereo Matching

Large disparity stereo matching is critical to the application of a stereo vision system especially for outdoor scenes. Nevertheless, how to efficiently design high accuracy large-disparity stereo matching on a field-programmable gate array (FPGA) is still a grand challenge. The computational complexity of previously proposed stereo matching is inevitably proportional to disparity range; hence their hardware designs become very inefficient when the disparity range is large. Motivated by the original PatchMatch and weighted median filtering (WMF) algorithms, this paper proposes a non-iterative PatchMatch and separable WMF (NIPM-sWMF) algorithm to significantly reduce the computational complexity of stereo matching and make it independent of disparity range. Moreover, we also propose a fully pipelined architecture design on FPGA that employs several hardware techniques to efficiently implement the proposed NIPM-sWMF. The disparity quality of the proposed NIPM-sWMF algorithm is evaluated on both KITTI2015 and Middlebury V3 stereo data sets, and the proposed architecture design is implemented and synthesized on Xilinx FPGA. Evaluation results demonstrate that the proposed NIPM-sWMF design on FPGA reaches the real-time performance of 1920 × 1080@60 Hz at the disparity range of 128, and can achieve almost the same disparity estimation accuracy, 4.5× processing throughput, while reducing the hardware cost of LUT, Register, DSP, and BRAM by 40%, 47%, 100%, and 68%, respectively, compared with the reference stereo matching design. Therefore, the proposed NIPM-sWMF design is an efficient way to address the challenge of large-disparity stereo matching.

Automatic waterline detection and 3D reconstruction in model ship tests using stereo vision

The changes of the waterlines alongside ship hulls in model ship tests provide important information to estimate the resistance performance of the full-scale ships and to improve its hull form design. This Letter presents a systematic procedure for detecting a waterline and reconstructing its three-dimensional (3D) geometry using a stereo vision system. The proposed method utilises the information from non-distinctive geometric features and shapes characterised in the waterline around a model ship without using any specified colours or artificial landmarks. For reconstructing the 3D geometry of the detected waterline, a vertical stereo system and a direct disparity estimation method are employed along with the consideration of uncertainties in the stereo correspondence matching. Experimental results are shown to demonstrate the feasibility of the proposed method using an image dataset obtained in a model ship test in a towing tank environment.

DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation From Stereo Imagery

Recent work has shown that convolutional neural networks (CNNs) can be applied successfully in disparity estimation, but these methods still suffer from errors in regions of low-texture, occlusions and reflections. Concurrently, deep learning for semantic segmentation has shown great progress in recent years. In this paper, we design a CNN architecture that combines these two tasks to improve the quality and accuracy of disparity estimation with the help of semantic segmentation. Specifically, we propose a network structure in which these two tasks are highly coupled. One key novelty of this approach is the two-stage refinement process. Initial disparity estimates are refined with an embedding learned from the semantic segmentation branch of the network. The proposed model is trained using an unsupervised approach, in which images from one half of the stereo pair are warped and compared against images from the other camera. Another key advantage of the proposed approach is that a single network is capable of outputting disparity estimates and semantic labels. These outputs are of great use in autonomous vehicle operation; with real-time constraints being key, such performance improvements increase the viability of driving applications. Experiments on KITTI and Cityscapes datasets show that our model can achieve state-of-the-art results and that leveraging embedding learned from semantic segmentation improves the performance of disparity estimation.

A Real-Time High-Quality Complete System for Depth Image-Based Rendering on FPGA

Depth image-based rendering (DIBR) techniques have recently drawn more attention in various 3D applications. In this paper, a real-time high-quality DIBR system that consists of disparity estimation and view synthesis is proposed. For disparity estimation, a local approach that focuses on depth discontinuities and disparity smoothness is presented to improve the disparity accuracy. For view synthesis, a method that contains view interpolation and extrapolation is proposed to render high-quality virtual views. Moreover, the system is designed with an optimized parallelism scheme to achieve a high throughput, and can be scaled up easily. It is implemented on an Altera Stratix IV FPGA at a processing speed of 45 frames per second for 1080p resolution. Evaluated on selected image sets of the Middlebury benchmark, the average error rate of the disparity maps is 6.02%; the average peak signal to noise ratio and structural similarity values of the virtual views are 30.07 dB and 0.9303, respectively. The experimental results indicate that the proposed DIBR system has the top-performing processing speed and its accuracy performance is among the best of state-of-the-art hardware implementations.

Outlier detection and disparity refinement in stereo matching

Disparity estimation in ill-posed regions, such as occlusions, repetitive patterns and textureless regions, is a challenging problem in stereo matching. The initial disparities obtained in these regions tend to be regarded as outliers that must be detected and addressed. In this paper, two outlier detection methods are proposed, i.e., the efficient approach and the accurate approach. The efficient approach detects outliers by exploring the disparity map for the left image only and reduces runtime and memory costs. First, the match fixed point jumps (MFPJ) algorithm is proposed as an initial solution to detect outliers. Then, a high-probability outlier detection algorithm is proposed to accomplish denser outlier detection with less noise. The accurate approach first classifies outliers as occlusions or mismatches. Then, 3D label assignment is performed for occlusion outliers and normal-based plane fitting is conducted for mismatch outliers to refine the disparities of the outliers and to achieve an accurate stereo matching result. Evaluations of the Middlebury datasets demonstrate that the proposed methods effectively improve the stereo matching performance.

As‐global‐as‐possible stereo matching with adaptive smoothness prior

More global matching (MGM) overcomes the limitation of one-dimensional scanline optimisation in semi-global matching (SGM). Nevertheless, the possible weaknesses of the MGM algorithm are as follows: (i) only two directions are considered for each image traversal direction, which may lead to massive mismatches; (ii) disparity estimation around the object boundaries usually performs terrible since the smoothness term is designed independent of the image prior. In this research, the authors consider all of the four directions for each image traversal direction through a novel model. Besides utilising the prior of neighboured pixels' correlation, adaptive smoothness terms are modelled and augmented into the energy function. These contributions encourage ‘as-global-as-possible (AGAP)’. More importantly, different from the recent works in which the aggregated algorithms have been conducted as the data term of an energy function, conversely, the authors make the energy function as a part of cost aggregation framework. Performance evaluations on Middlebury v.2 and v.3 stereo data sets demonstrate that the proposed AGAP outperforms other four most challenging stereo matching algorithms, and also performs better on Microsoft i2i stereo videos. In addition, under various strategies of parallelisation, the presented AGAP shows a near real-time execution time.