Disparity Estimation Algorithm Research Articles

Take Your Model Further: A General Post-refinement Network for Light Field Disparity Estimation via BadPix Correction

Most existing light field (LF) disparity estimation algorithms focus on handling occlusion, texture-less or other areas that harm LF structure to improve accuracy, while ignoring other potential modeling ideas. In this paper, we propose a novel idea called Bad Pixel (BadPix) correction for method modeling, then implement a general post-refinement network for LF disparity estimation: Bad-pixel Correction Network (BpCNet). Given an initial disparity map generated by a specific algorithm, we assume that all BadPixs on it are in a small range. Then BpCNet is modeled as a fine-grained search strategy, and a more accurate result can be obtained by evaluating the consistency of LF images in this limited range. Due to the assumption and the consistency between input and output, BpCNet can perform as a general post-refinement network, and can work on almost all existing algorithms iteratively. We demonstrate the feasibility of our theory through extensive experiments, and achieve remarkable performance on the HCI 4D Light Field Benchmark.

A light‐weight stereo matching network based on multi‐scale features fusion and robust disparity refinement

In recent years, convolutional-neural-network based stereo matching methods have achieved significant gains compared to conventional methods in terms of both speed and accuracy. Current state-of-the-art disparity estimation algorithms require many parameters and large amounts of computational resources and are not suited for applications on edge devices. In this paper, an end-to-end light-weight network (LWNet) for fast stereo matching is proposed, which consists of an efficient backbone with multi-scale feature fusion for feature extraction, a 3D U-Net aggregation architecture for disparity computation, and color guidance in a 2D convolutional neural network (CNN) for disparity refinement. MobileNetV2 is adopted as an efficient backbone in feature extraction. The channel attention module is applied to improve the representational capacity of features and multi-resolution information is adaptively incorporated into the cost volume via cross-scale connections. Further, a left-right consistency check and color guidance refinement are introduced and a robust disparity refinement network is designed with skip connections and dilated convolutions to capture global context information and improve disparity estimation accuracy with little computational cost and memory space. Extensive experiments on Scene Flow, KITTI 2015, and KITTI 2012 demonstrate that the proposed LWNet achieves competitive accuracy and speed when compared with state-of-the-art stereo matching methods.

A comparative evaluation of 3D geometries of scenes estimated using factor graph based disparity estimation algorithms

Passive stereo vision systems are useful for estimating 3D geometries from digital images similar to the human biological system. In general, two cameras are situated at a known distance from the object and simultaneously capture images of the same scene from different views. This paper presents a comparative evaluation of 3D geometries of scenes estimated by three disparity estimation algorithms, namely the hybrid stereo matching algorithm (HCS), factor graph-based stereo matching algorithm (FGS), and a multi-resolution FGS algorithm (MR-FGS). Comparative studies were conducted using our stereo imaging system as well as hand-held, consumer-market digital cameras and camera phones of a variety of makes/models. Based on our experimental results, the factor graph algorithm (FGS) and multi-resolution factor graph algorithm (MR-FGS) result in a higher level of 3D reconstruction accuracy than the HCS algorithm. When compared with the FGS algorithm, MR-FGS provides a significant improvement in the disparity contrast along the depth boundaries and minimal depth discontinuities.

A novel factor graph-based optimization technique for stereo correspondence estimation

Dense disparities among multiple views are essential for estimating the 3D architecture of a scene based on the geometrical relationship between the scene and the views or cameras. Scenes with larger extents of homogeneous textures, differing scene illumination among the multiple views and with occluding objects affect the accuracy of the estimated disparities. Markov random fields based methods for disparity estimation address these limitations using spatial dependencies among the observations and among the disparity estimates. These methods, however, are limited by spatially fixed and smaller neighborhood systems or cliques. Recent learning-based methods generate rich set of stereo features for generating cost volume and estimating disparity. In this work, we present a new factor graph-based probabilistic graphical model for disparity estimation that allows a larger and a spatially variable neighborhood structure determined based on the local scene characteristics. Our algorithm improves the accuracy of disparity estimates in stereo image pairs with varying texture and illumination characteristics by enforcing spatial dependencies among scene characteristics as well as among disparity estimates. We evaluated our method using the Middlebury benchmark stereo datasets and the Middlebury evaluation dataset version 3.0 and compared its performance with recent state-of-the-art disparity estimation algorithms. Our factor graph-based algorithm provided disparity estimates with higher accuracy when compared to the recent non-learning- and learning-based disparity estimation algorithms. The factor graph formulation can be used for obtaining maximum a posteriori estimates from models or optimization problems with complex dependency structure among hidden variables. The strategies of using a priori distributions with shorter support and spatial dependencies were useful for reducing the computational cost and improving message convergence in the model. The factor-graph algorithm is also useful for other dense estimation problems such as optical flow estimation.

HMSM-Net: Hierarchical multi-scale matching network for disparity estimation of high-resolution satellite stereo images

Disparity estimation of satellite stereo images is an essential and challenging task in photogrammetry and remote sensing. Recent researches have greatly promoted the development of disparity estimation algorithms by using CNN (Convolutional Neural Networks) based deep learning techniques. However, it is still difficult to handle intractable regions that are mainly caused by occlusions, disparity discontinuities, texture-less areas, and repetitive patterns. Besides, the lack of training datasets for satellite stereo images remains another major issue that blocks the usage of CNN techniques due to the difficulty of obtaining ground-truth disparities. In this paper, we propose an end-to-end disparity learning model, termed hierarchical multi-scale matching network (HMSM-Net), for the disparity estimation of high-resolution satellite stereo images. First, multi-scale cost volumes are constructed by using pyramidal features that capture spatial information of multiple levels, which learn correspondences at multiple scales and enable HMSM-Net to be more robust in intractable regions. Second, stereo matching is executed in a hierarchical coarse-to-fine manner by applying supervision to each scale, which allows a lower scale to act as prior knowledge and guides a higher scale to attain finer matching results. Third, a refinement module that incorporates the intensity and gradient information of the input left image is designed to regress a detailed full-resolution disparity map for local structure preservation. For network training and testing, a dense stereo matching dataset is created and published by using GaoFen-7 satellite stereo images. Finally, the proposed network is evaluated on the Urban Semantic 3D and GaoFen-7 datasets. Experimental results demonstrate that HMSM-Net achieves superior accuracy compared with state-of-the-art methods, and the improvement on intractable regions is noteworthy. Additionally, results and comparisons of different methods on the GaoFen-7 dataset show that it can severs as a challenging benchmark for performance assessment of methods applied to disparity estimation of satellite stereo images. The source codes and evaluation dataset are made publicly available at https://github.com/Sheng029/HMSM-Net.

A calibration-free workflow for image-based mixed reality navigation of total shoulder arthroplasty

ABSTRACT Accurate pin placement to guide renaming of the glenoid surface in total shoulder arthroplasty (TSA) is a critical step to restore range of motion in the glenohumeral joint. Achieving proper pin position with free-hand is complicated due to inadequate intra-operative availability of pre-operative planning data. Mixed reality provides a new modality of surgical navigation for arthroplasty without distracting surgeon from the patient anatomy. However, achieving accurate alignment between the pre-operative plan and the intra-operative anatomy is challenging, especially if workflow-compliant solutions are desired. In this paper, we present a calibration-free mixed reality navigation workflow for glenoid pin placement in TSA that contributes to solving some open challenges. We use a visible light stereo camera on a surgical support arm combined with a transformer-based disparity estimation algorithm to accurately reconstruct the 3D anatomy despite surgical lighting. Besides, we introduce a slide-on marker calibrated by design to overcome perceptual difficulties that arise when trying to align real to virtual objects. In a user study on a semi-anthropomorphic phantom, we assess the end-to-end pin placement accuracy using both our system and a clinical reference standard solution that relies on patient-specific drill guides. Averaging over 15 drilled trajectories, our mixed reality system achieved a pin placement accuracy of ( mm, ).

Three-Dimensional Reconstruction of Light Field Based on Phase Similarity

Light field imaging plays an increasingly important role in the field of three-dimensional (3D) reconstruction because of its ability to quickly obtain four-dimensional information (angle and space) of the scene. In this paper, a 3D reconstruction method of light field based on phase similarity is proposed to increase the accuracy of depth estimation and the scope of applicability of epipolar plane image (EPI). The calibration method of the light field camera was used to obtain the relationship between disparity and depth, and the projector calibration was removed to make the experimental procedure more flexible. Then, the disparity estimation algorithm based on phase similarity was designed to effectively improve the reliability and accuracy of disparity calculation, in which the phase information was used instead of the structure tensor, and the morphological processing method was used to denoise and optimize the disparity map. Finally, 3D reconstruction of the light field was realized by combining disparity information with the calibrated relationship. The experimental results showed that the reconstruction standard deviation of the two objects was 0.3179 mm and 0.3865 mm compared with the ground truth of the measured objects, respectively. Compared with the traditional EPI method, our method can not only make EPI perform well in a single scene or blurred texture situations but also maintain good reconstruction accuracy.

An improved algorithm for disparity estimation of SGM stereo matching based on edge detection

In the fields of depth measurement and computer vision, generating sufficiently accurate disparity maps through various stereo matching algorithms is an important foundation. The SGM semi-global matching algorithm is one of the best stereo matching algorithms, But its filling accuracy is poor in the invalid parallax area. Therefore, on the original basis, an improved algorithm of SGM stereo matching disparity filling based on edge detection is proposed. The experiment uses a standard data set to verify the algorithm. The results show that compared with the original SGM stereo matching algorithm, the paper algorithm can achieve better disparity estimation results in the invalid disparity area.

Dense stereo matching in depth discontinuity areas based on edge detection and nearest valid point search

ABSTRACT Dense stereo matching based on two viewpoint images is an important step in 3D reconstruction. In this study, a new disparity estimation algorithm for depth discontinuity areas is proposed. Dense stereo matching is implemented on the basis of the semi-global matching algorithm. If the ground truth is known, then mismatched pixels are selected as those with a difference of more than one pixel from the ground truth. Otherwise, the left-right consistency checking process is used. Then, the Canny operator is used to detect image edges. For each mismatched pixel, a maximum total of eight nearest edge points exists in four pairs of symmetric directions. Basically, the process of searching for the nearest valid point proceeds along directions in the following sequence: horizontal, vertical, 45° oblique and 135° oblique. Specifically, for each pair of symmetric directions, the nearest valid pixel is searched along the direction of the distant edge point. If the disparity is successfully reassigned by the nearest valid pixel found, the searching process is terminated. Comparative results obtained using well-known benchmark datasets and unmanned aerial vehicle stereo images demonstrate the excellent performance of the proposed method.

A New Hybrid Stereo Disparity Estimation Algorithm with Guided Image Filtering-Based Cost Aggregation

Stereo matching algorithms are useful for estimating a dense depth characteristic of a scene by finding corresponding points from stereo images of the scene. Several factors such as occlusion, noise, and illumination inconsistencies in the scene affect the disparity estimates and make this process challenging. Algorithms developed to overcome these challenges can be broadly categorized as learning-based and non-learning based disparity estimation algorithms. The learning-based approaches are more accurate but computationally expensive. In contrary, non-learning based algorithms are widely used and are computationally efficient algorithms. In this paper, we propose a new stereo matching algorithm using guided image filtering (GIF)-based cost aggregation. The main contribution of our approach is a cost calculation framework which is a hybrid of cross-correlation between stereo-image pairs and scene segmentation (HCS). The performance of our HCS technique was compared with state-ofthe- art techniques using version 3 of the benchmark Middlebury dataset. Our results confirm the effective performance of the HCS technique.

Local low rank approximation with a parametric disparity model for light field compression.

We address the problem of light field dimensionality reduction for compression. We describe a local low rank approximation method using a parametric disparity model. The local support of the approximation is defined by super-rays. A super-ray can be seen as a set of super-pixels that are coherent across all light field views. A dedicated super-ray construction method is first described that constrains the super-pixels forming a given super-ray to be all of the same shape and size, dealing with occlusions. This constraint is needed so that the super-rays can be used as supports of angular dimensionality reduction based on low rank matrix approximation. The light field low rank assumption depends on how much the views are correlated, i.e. on how well they can be aligned by disparity compensation. We first introduce a parametric model describing the local variations of disparity within each super-ray. We then consider two methods for estimating the model parameters. The first method simply fits the model on an input disparity map. We then introduce a disparity estimation method using a low rank prior. This method alternatively searches for the best parameters of the disparity model and of the low rank approximation. We assess the proposed disparity parametric model, first assuming that the disparity is constant within a super-ray, and second by considering an affine disparity model. We show that using the proposed disparity parametric model and estimation algorithm gives an alignment of super-pixels across views that favours the low rank approximation compared with using disparity estimated with classical computer vision methods. The low rank matrix approximation is computed on the disparity compensated super-rays using a singular value decomposition (SVD). A coding algorithm is then described for the different components of the proposed disparity-compensated low rank approximation. Experimental results show performance gains, with a rate saving going up to 92.61%, compared with the JPEG Pleno anchor, for real light fields captured by a Lytro Illum camera. The rate saving goes up to 37.72% with synthetic light fields. The approach is also shown to outperform an HEVC-based light field compression scheme.

Segmentation-Based Color Channel Registration for Disparity Estimation of Dual Color-Filtered Aperture Camera.

Single-lens-based optical range finding systems were developed as an efficient, compact alternative for conventional stereo camera systems. Among various single-lens-based approaches, a multiple color-filtered aperture (MCA) system can generate disparity information among color channels, as well as normal color information. In this paper, we consider a dual color-filtered aperture (DCA) system as the most minimal version of the MCA system and present a novel inter-color image registration algorithm for disparity estimation. This proposed registration algorithm consists of three steps: (i) color channel independent feature extraction; (ii) feature-based adaptive weight disparity estimation; and (iii) color mapping matrix (CMM)-based cross-channel image registration. Experimental results show that the proposed method can not only generate an accurate disparity map, but also realize high quality cross-channel registration with a disparity prior for DCA-based range finding and color image enhancement.

Fast Nonparametric Road Disparity Estimation and Gradient Constrained Obstacle Detection for UGV Navigation

Road estimation and obstacle detection are two key techniques for autonomous driving vehicles, requiring high accuracy, efficiency and robustness. In this paper, we propose an efficient nonparametric road disparity estimation algorithm that makes full use of the characteristics of the road disparity. It can adapt to both planar and nonplanar road. In addition, to satisfy the real-time requirements of the application, we exploit a new texture gradient based constraint to compute the free space and detect the obstacle, which will also help to obtain more accurate results. Experimental results with real stereo sequences show that the proposed algorithm outperforms the standard methods both in terms of detection accuracy and runtime performance, which can run over 200 fps with normal PC.

Road Surface 3D Reconstruction Based on Dense Subpixel Disparity Map Estimation.

Various 3D reconstruction methods have enabled civil engineers to detect damage on a road surface. To achieve the millimetre accuracy required for road condition assessment, a disparity map with subpixel resolution needs to be used. However, none of the existing stereo matching algorithms are specially suitable for the reconstruction of the road surface. Hence in this paper, we propose a novel dense subpixel disparity estimation algorithm with high computational efficiency and robustness. This is achieved by first transforming the perspective view of the target frame into the reference view, which not only increases the accuracy of the block matching for the road surface but also improves the processing speed. The disparities are then estimated iteratively using our previously published algorithm where the search range is propagated from three estimated neighbouring disparities. Since the search range is obtained from the previous iteration, errors may occur when the propagated search range is not sufficient. Therefore, a correlation maxima verification is performed to rectify this issue, and the subpixel resolution is achieved by conducting a parabola interpolation enhancement. Furthermore, a novel disparity global refinement approach developed from the Markov Random Fields and Fast Bilateral Stereo is introduced to further improve the accuracy of the estimated disparity map, where disparities are updated iteratively by minimising the energy function that is related to their interpolated correlation polynomials. The algorithm is implemented in C language with a near real-time performance. The experimental results illustrate that the absolute error of the reconstruction varies from 0.1 mm to 3 mm.

High-Speed Low-Complexity Guided Image Filtering-Based Disparity Estimation

Stereo vision is a methodology to obtain depth in a scene based on the stereo image pair. In this paper, we introduce a discrete wavelet transform (DWT)-based methodology for a state-of-the-art disparity estimation algorithm that resulted in significant performance improvement in terms of speed and computational complexity. In the initial stage of the proposed algorithm, we apply DWT to the input images, reducing the number of samples to be processed in subsequent stages by 50%, thereby decreasing computational complexity and improving processing speed. Subsequently, the architecture has been designed based on this proposed methodology and prototyped on a Xilinx Virtex-7 FPGA. The performance of the proposed methodology has been evaluated against four standard Middlebury Benchmark image pairs viz. Tsukuba , Venus , Teddy , and Cones . The proposed methodology results in the improvement of about 44.4% cycles per frame, 52% frames/s, and 61.5% and 59.6% LUT and register utilization, respectively, compared with state-of-the-art designs.

Runtime Performance and Power Optimization of Parallel Disparity Estimation on Many-Core Platforms

This article investigates the use of many-core systems to execute the disparity estimation algorithm, used in stereo vision applications, as these systems can provide flexibility between performance scaling and power consumption. We present a learning-based runtime management approach that achieves a required performance threshold while minimizing power consumption through dynamic control of frequency and core allocation. Experimental results are obtained from a 61-core Intel Xeon Phi platform for the aforementioned investigation. The same performance can be achieved with an average reduction in power consumption of 27.8% and increased energy efficiency by 30.04% when compared to Dynamic Voltage and Frequency Scaling control alone without runtime management.

Optimal reference view selection algorithm for low complexity disparity estimation

Multi-view video coding (MVC) is composed of multiple video sequences captured simultaneously by multiple closely spaced cameras. Such a system demands a higher computational complexity due to the increased number of cameras. Disparity estimation (DE) is one of the compression techniques used to remove inter-view redundancy in MVC, which causes high complexity. In order to overcome the problem of high computational complexity in real-time MVC, an optimal reference view selection algorithm is proposed. The proposed scheme decreases the coding complexity by reducing one-directional DE operation and minimizing the decrease in coding performance in MVC.

A resolution enhancement algorithm for an asymmetric resolution stereo video

Abstract This paper presents a resolution enhancement algorithm in an asymmetric resolution stereo video for improving the video resolution. In this stereo video architecture, a scene is captured by two cameras to form two views, one view is a lower-resolution video and the other is a full-resolution video. The goal is to enhance the lower-resolution video to a full-resolution video. In the lower-resolution video, frames synchronized with full-resolution video are enhanced via disparity estimation algorithm, while the rest frames are improved by mono-view video super-resolution based on key frames method. The experimental results demonstrate that the proposed method is effective for both visual and objective qualities.

Efficient Motion and Disparity Estimation Optimization for Low Complexity Multiview Video Coding

The use of variable block-size motion estimation (ME), disparity estimation (DE), and multiple reference frames selection aims to improve the coding efficiency of multiview video coding (MVC), however, this is at the cost of high computational complexity of these advanced coding techniques, which are not suitable for real-time video broadcasting applications. In this paper, we propose an efficient ME and DE algorithm for reducing the computational complexity of MVC. Firstly, according to the characteristics of the coded block pattern and rate distortion (RD) cost, an early DIRECT mode decision algorithm is proposed. Then, based on the characteristics of the initial search point in the ME/DE process and the observation that the best point is center-biased, an early ME/DE termination strategy is proposed. If the ME/DE early termination is not satisfied, the ME/DE search window will be reduced by applying the optimal theory. At last, two block matching search strategies are proposed to predict the best point for the ME/DE. Experimental results show that the proposed algorithm can achieve 50.05% to 77.61%, 64.83% on average encoding time saving. Meanwhile, the RD performance degradation is negligible. Especially, the proposed algorithm can be applied to not only the odd views but also the even views.

The VLSI implementation of a high-resolution depth-sensing SoC based on active structured light

This paper presents the full VLSI implementation of a new high-resolution depth-sensing system on a chip (SoC) based on active infrared structured light, which estimates the 3D scene depth by matching randomized speckle patterns, akin to the Microsoft Kinect. We present a module to enhance the consistency of speckle patterns for robust matching, and hence improve the range of depth estimation. We present a simple and efficient hardware structure for a block-matching-based disparity estimation algorithm, which facilitates rapid generation of disparity maps in real time. For depth estimation from disparity maps, we propose a hardware friendly solution by producing a lookup table from a curve fitted to calculate and calibrate the depth value in a single step, which does not need to explicitly calibrate the parameters of the imaging sensors, such as the length of the baseline. We have implemented these ideas in an end-to-end SoC using FPGA. Compared with the Kinect, our depth-sensing SoC has wider effective ranging limit (0.6---4.5 m), and has an uppermost $$1280\times 1024$$1280×1024 processing capacity with 60 Hz frame frequency. Its depth resolution is 1 mm@0.82 m. Our system is superior to Kinect in terms of operating range, processing frame rate, and resolution.