Disparity Space Image Research Articles

Convex Vision-Based Negative Obstacle Detection Framework for Autonomous Vehicles

This study investigates the convex optimization frameworks for accurate and fast negative obstacle detection as well as depth estimations that are vital for real-time, superior autonomous vehicle operations. A convex framework is proposed in conjunction with a unique stereo perception approach to manage the complex nature of such invisible obstacles. The proposed stereo configuration incorporates an alignment that displaces the cameras along a vertical baseline to extract useful information pertaining to negative obstacle features for various illustrative terrain settings. The convex framework exploits these properties to evaluate depth jumps in the disparity space image and perform geometrical analysis of potential occlusion regions. The corresponding convex formulations are based on linear matrix inequalities to detect sudden disparity, angle profile, and intensity variations for potential negative obstacles and to estimate the internal depth of the detected obstacles. Experimental results demonstrate that consecutive processes can be structured in a convex framework to efficiently identify negative obstacle attributes at a range of distances for texture varying environments, providing a vital extension to real-time vehicular system implementations with superior detection rates.

Multi‐Event‐Camera Depth Estimation and Outlier Rejection by Refocused Events Fusion

Event cameras are bio‐inspired sensors that offer advantages over traditional cameras. They operate asynchronously, sampling the scene at microsecond resolution and producing a stream of brightness changes. This unconventional output has sparked novel computer vision methods to unlock the camera's potential. Here, the problem of event‐based stereo 3D reconstruction for SLAM is considered. Most event‐based stereo methods attempt to exploit the high temporal resolution of the camera and the simultaneity of events across cameras to establish matches and estimate depth. By contrast, this work investigates how to estimate depth without explicit data association by fusing disparity space images (DSIs) originated in efficient monocular methods. Fusion theory is developed and applied to design multi‐camera 3D reconstruction algorithms that produce state‐of‐the‐art results, as confirmed by comparisons with four baseline methods and tests on a variety of available datasets.

IMPROVING PAIRWISE DSM WITH 3SGM: A SEMANTIC SEGMENTATION FOR SGM USING AN AUTOMATICALLY REFINED NEURAL NETWORK

Abstract. The amount of very high resolution optical satellite images at our disposal is continuously increasing. Besides, associated satellite programs often come with high revisit rates and geometric properties that allow for either opportunistic or by-design 3D stereo reconstruction. Digital Surface Models (DSM) computed from these satellite images offer new possibilities. In the past, the high revisit rate has largely benefited glacier monitoring studies. Now, DSM with increased resolution provided on urban areas can be used for smart city applications as well. However, most of these require 3D modeling of buildings with level of details ranging from 0 to 2. This is where the need for better reconstructed buildings inside DSM arises. Indeed, building edges and corners tend to be smoothed and softened by the stereo matching step of a DSM computation pipeline. This undesired behavior can mostly be linked to the difficult task of optimizing the Disparity Space Image, thus finding good balance between smoothing untextured areas while conserving sharp discontinuities where needed. In this paper, we show how the optimization can benefit from an input building semantic segmentation. We also provide a method to create it from a very high satellite image in epipolar geometry using a convolutional neural network. To help our network generalize well on unseen areas we propose an interactive learning method based on clicked annotations. Eventually, we show that annotations can be automatically created, hence removing the need for an operator and making our solution suitable for operational conditions.

LOCK-FREE MULTITHREADED SEMI-GLOBAL MATCHING WITH AN ARBITRARY NUMBER OF PATH DIRECTIONS

Abstract. This paper describes an efficient implementation of the semi-global matching (SGM) algorithm on multi-core processors that allows a nearly arbitrary number of path directions for the cost aggregation stage. The scanlines for each orientation are discretized iteratively once, and the regular substructures of the obtained template are reused and shifted to concurrently sum up the path cost in at most two sweeps per direction over the disparity space image. Since path overlaps do not occur at any time, no expensive thread synchronization will be needed. To further reduce the runtime on high counts of path directions, pixel-wise disparity gating is applied, and both the cost function and disparity loop of SGM are optimized using current single instruction multiple data (SIMD) intrinsics for two major CPU architectures. Performance evaluation of the proposed implementation on synthetic ground truth reveals a reduced height error if the number of aggregation directions is significantly increased or when the paths start with an angular offset. Overall runtime shows a speedup that is nearly linear to the number of available processors.

Optimised Implementation of Dense Stereo Correspondence for Resource Limited Hardware

Computer stereo vision is a passive sensing technique that helps to recover 3D information of an environment from 2D images. The stereo correspondence is a challenging task that finds out matching pixels between the stereo image pair based on Lambertian criteria and its result is a disparity space image. The depth of the objects from the camera can be calculated from this disparity value by using the principle of triangulation. For the vision guided robot navigation, the requirement of stereo matching algorithms on low power dedicated hardware that can achieve a high frame rate is unambiguous. A new, highly optimized implementation of correlation based, Sum of Absolute Differences correspondences algorithm on a low cost resource limited FPGA is presented here. This System-on-Programmable-Chip architecture based system achieved a higher frame rate of 50 fps with 64 disparity levels without using a microprocessor. On performance evaluation, the disparity map shows a maximum error of 0.308% only. This reconfigurable parallel processing, high speed architecture of the algorithm implementation takes only 43% of available resources of low density Altera Cyclone II. This hardware implementation of stereo vision system outperforms in terms of accuracy, speed and resource utilization of all the other existing stereo systems of its similar kind. Also, it offers a better trade-off between run-time speed and accuracy and is found suitable for most of the range finding real-time applications.

Depth sensing with disparity space images and three-dimensional recursive search

ABSTRACTWe present a coarse-to-fine stereo matching optimization applicable to methods utilizing the Disparity Space Image (DSI) structure. With the Three-dimensional Recursive Search algorithm (3DRS), a coarse disparity seed is obtained first, with minimal computational effort. The coarse disparity seed is then used as a guidance to locally compute the DSI disparity space with a reduced number of disparity hypotheses, yielding significantly shorter execution times for the disparity computation. The method performance was measured on the well-known Dynamic Programming (DP) DSI-based method and the images from the Middlebury set. The DP method with the DSI optimization applied maintains or improves the overall level of disparity map accuracy while delivering a near sevenfold speed-up of execution in comparison to DP alone. We furthermore show that the optimized method's performance does not depend on the expected input disparity range, which is commonly restricted, or expected to be defined upfront, for DSI-based stereo matching methods.

SEMI-GLOBAL MATCHING WITH SELF-ADJUSTING PENALTIES

Abstract. The demand for 3D models of various scales and precisions is strong for a wide range of applications, among which cultural heritage recording is particularly important and challenging. In this context, dense image matching is a fundamental task for processes which involve image-based reconstruction of 3D models. Despite the existence of commercial software, the need for complete and accurate results under different conditions, as well as for computational efficiency under a variety of hardware, has kept image-matching algorithms as one of the most active research topics. Semi-global matching (SGM) is among the most popular optimization algorithms due to its accuracy, computational efficiency, and simplicity. A challenging aspect in SGM implementation is the determination of smoothness constraints, i.e. penalties P1, P2 for disparity changes and discontinuities. In fact, penalty adjustment is needed for every particular stereo-pair and cost computation. In this work, a novel formulation of self-adjusting penalties is proposed: SGM penalties can be estimated solely from the statistical properties of the initial disparity space image. The proposed method of self-adjusting penalties (SGM-SAP) is evaluated using typical cost functions on stereo-pairs from the recent Middlebury dataset of interior scenes, as well as from the EPFL Herz-Jesu architectural scenes. Results are competitive against the original SGM estimates. The significant aspects of self-adjusting penalties are: (i) the time-consuming tuning process is avoided; (ii) SGM can be used in image collections with limited number of stereo-pairs; and (iii) no heuristic user intervention is needed.

Shape-indifferent stereo disparity based on disparity gradient estimation

A new algorithm capable of estimating disparity gradients to produce accurate dense disparities is proposed. Such a disparity gradient plays a critical role in acquiring accurate disparities for scenes with many different object shapes. The target is a road traffic scene because it contains various objects, including the road surface, vehicles, pedestrians, sidewalks, and walls. In this paper, we adopt several methods, such as initial matching cost computation, scanline optimization, left/right consistency check, and cost aggregation. However, disparity accuracy is slightly improved by the simple organization of such methods. Disparity quality decisively relies on the application of disparity gradients. Accordingly, in the proposed algorithm, cost aggregation is performed along the direction of the estimated disparity gradient in a disparity space image. This approach improves disparity quality significantly. However, this cost aggregation is time consuming. To reduce the time required, we designed a new 2D integral cost technique. The robustness of the proposed algorithm is demonstrated through the disparity maps obtained from standard images on the Web, indoor images, and outdoor images of various road traffic scenes.

Modified Dual Winner Takes All Approach for Tri-Stereo Image Matching Using Disparity Space Images

In this research paper, a new framework is proposed to increase the total number of correct matches for stereo correspondence using tri-stereo images. The research work investigates some of the less explored properties of Disparity Space Image (DSI), and considers the local maxima in addition to the global maximum of the cost function and propose a new tri-stereo matching method as Modified Dual Winner Takes All (MDWTA) using edges of disparity space image. Conventionally using a single DSI from a stereo pair optimization techniques are applied to get the disparity, whereas the proposed approach uses Winner Takes All (WTA) approach using two DSIs from a triplet. The dual property of DSI that is the manner in which it stores cost for rows and columns for forward and reverse matching is introduced. This property is applied to check consistency for forward and reverse matching in a single pass, which gives initial correct matches. Next a left-centre-right consistency check is applied to discard inconsistent disparities obtained from these three views. Subsequently to obtain disparity for gaps thus generated a rule is formulated using local maxima, edges and adjacent global maximum to find correct disparity. Evaluation of the results obtained is carried by comparing them with block matching WTA method and other recent methods such as, Dynamic Programming (DP) and Semi-Global Matching (SGM) applied on images from Terrain Mapping Camera of Chandrayaan-1 and PRISM sensor of the ALOS mission. Methodology proposed is also verified using standard Middlebury stereo dataset. Experimental results show that the proposed method gives an increased number of correct matches by 10–15 % as compared to basic WTA and DP. However the results obtained are better to SGM for certain regions and features.

Efficient methods using slanted support windows for slanted surfaces

The frontal-parallel assumption is made by many matching algorithms, but this assumption fails for slanted surfaces. This study proposes a matching algorithm intended to improve the matching results for slanted surfaces. First, a mathematical model is constructed to prove that slanted surfaces in the environment have corresponding slanted disparity surfaces in the disparity space image, and the model is to help find the proper plane parameters of slanted support windows, then improved cost aggregation and post-processing methods are proposed. The algorithm is tested using the Middlebury and Karlsruhe Institute of Technology and Toyota Technical Institute at Chicago (KITTI) benchmarks. The results demonstrate that the algorithm exhibits good performance and is efficient for slanted surfaces.

Stereo matching based on segmented B‐spline surface fitting and accelerated region belief propagation

The authors propose a new stereo matching algorithm based on an iterative optimisation framework including bi‐cubic B‐spline surface fitting and accelerated region belief propagation (BP). They first compute the initial cost and disparity map by the adaptive support‐weight approach and then launch the iterative process in which the disparity space image is refined via the bi‐cubic B‐spline fitting and optimised via the accelerated region BP. Two innovations are contained in the algorithm: (i) disparity space image refinement based on segmented bi‐cubic B‐spline surface fitting; and (ii) an accelerated region message passing approach for BP. The algorithm is verified on the Middlebury benchmark and experimental results show the algorithm is effective and achieves the state‐of‐the‐art accuracy.

A New Approach for Stereo Matching Algorithm with Dynamic Programming

Stereo matching algorithms are one of heavily researched topic in binocular stereo vision. Massive 3D information can be obtained by finding correct correspondence of different points between images captured from different views. Development of stereo matching algorithm is done for obtaining disparity maps i.e. depth information. When disparities computed for scan lines then dense reconstruction becomes time consuming for vision navigation systems. So for pair of stereo images proposed method extracts features points those are at contours in images and then a dynamic program is used to find the corresponding points from each image and calculates disparities. Also to reduce the noise which may lead to incorrect results in stereo correspondence, a new stereo matching algorithm based on the dynamic programming is proposed. Generally dynamic programming finds the global minimum for independent scan lines in polynomial time. While efficient, its performance is far from desired one because vertical consistency between scan lines is not enforced. This method review the use of dynamic programming for stereo correspondence by applying it to a contour instead to individual scan lines. Proposed methodology will obtain the global minimum for contour array in linear time using Longest Common Subsequent (LCS) dynamic programming method with no disparity space image (DSI). KeywordsStereo Correspondence, Contour, Dynamic Programming, Disparity space Image, Longest common subsequent. ________________________________________________________*****_______________________________________________________

A Hybridized Disparity Computation Fusing Disparity Space Image and Multi Resolution Image Segmentation

D information is getting more importance nowadays.There are several ways to get 3D information.One way to get 3D information involves the laser based depth estimators, which is a very costly process. Other way is to extract 3D information from 2D stereo image pairs by using stereo matching techniques. Stereo matching is a vast area of research. It can be classified into area (window) based approach, feature based approach, and optimization based approach. Area based approach generally generates dense disparity map with low accuracy and low computation time. Feature based method produces highly accurate and sparse disparity map with low computation time.Optimization based method produces dense disparity map with high accuracy and high execution time.By keeping these things in mind, we proposed a new hybrid way for disparity computation. The proposed method consists of three steps. Theyare disparityestimation, image segmentation, and disparity refinement. Since the ultimate goal of stereo matching is to obtain dense disparity map with high accuracy and low execution time, we select optimization based approach for disparity estimation step and for image segmentation step we select multi resolution image segmentation. At the end, disparity refinement is done by combining the result of both the previous steps.As there are several optimization techniques, we choose disparity space image (DSI) baseddynamic programming (DP).We tested the proposed method on several stereo pairs and found that method produced reasonably good quality results. Index Terms—Stereo matching, DSI, DP.

Scan line optimization for Tri stereo planetary images

Abstract. In this paper, we propose a new scan line optimization method for matching the triplet of images. In the present paper, the triplets are initially matched using an area based local method. The cost is stored in a structure called as the Disparity Space Image (DSI). Using the global minimum of this cost the initial disparity is generated. Next the local minima are considered as potential matches where global minimum gives erroneous results. These local minima are used for optimization of disparity. As the method is a scanned line optimization, it use popularly resampled images. The experiment is performed using Terrain Mapping Camera images from the Chandrayaan-1 mission. In order to validate the result for accuracy, Lunar Orbiter Laser Altimeter dataset from Lunar Reconnaissance Orbiter mission is used. The method is again verified using standard Middlebury stereo dataset with ground truth. From experiments, it has been observed that using optimization technique for triplets, the total number of correct matches has increased by 5–10 % in comparison to direct methods. The method particularly gives good results at smooth regions, where dynamic programming and blockmatching gives limited accuracy.

High Performance Stereo System for Dense 3-D Reconstruction

3-D stereo reconstruction, a technique that estimates per-pixel depth in a scene, is still a challenging problem mainly due to some prohibitive factors that limit its performance and computational ability. The aim of this paper is to present a new hardware-efficient disparity map computation, which is based on disparity space image processing using discrete dynamic systems. The hardware architecture of the proposed system was implemented on a high-end field programmable gate array (FPGA) device, offering real-time 3-D reconstruction speeds using a hardware aware architecture based on parallelism and process pipelining. The proposed architecture fulfills the requirements of real-world applications regarding resource usage, frame rates, and disparity resolution, while its implementation on an Altera Stratix IV family FPGA device can extract disparity maps of up to \(\mathbf {1280 \times 1024}\) pixels with up to 128 disparity levels under real-time or near real-time conditions at a clock rate of 168 MHz. Qualitative and quantitative results also demonstrate its performance and improvement over previous hardware-related studies, making our approach a suitable candidate for applications in which timing and processing constraints are critical.

Objective Quality Assessment of Stereoscopic Images: From Perspective of Disparity Space Image

Objective Quality Assessment of Stereoscopic Images: From Perspective of Disparity Space Image

Dense Stereo Matching Method Based on Local Affine Model

A new method for constructing an accurate disparity space image and performing an efficient cost aggregation in stereo matching based on local affine model is proposed in this paper. The key algorithm includes a new self-adapting dissimilarity measurement used for calculating the matching cost and a local affine model used in cost aggregation stage. Different from the traditional region-based methods, which try to change the matching window size or to calculate an adaptive weight to do the aggregation, the proposed method focuses on obtaining the efficient and accurate local affine model to aggregate the cost volume while preserving the disparity discontinuity. Moreover, the local affine model can be extended to the color space. Experimental results demonstrate that the proposed method is able to provide subpixel precision disparity maps compared with some state-of-the-art stereo matching methods.

A Improved Stereo Matching Fast Algorithm Based on Dynamic Programming

Compared with the local algorithm in stereo matching, the high quality disparity space image is calculated by the global algorithm, which is difficult to use in practical application for its long computation time. The dynamic programming is one of the global algorithms with a fast matching speed, but it has strip blemish in matching result. In this paper, a new dynamic programming based method is proposed to accelerate the matching speed and improve the matching quality. Firstly, the color feature of two images are calculated using the Laplacians of Gaussian pyramid algorithm, and the color feature of the image pair obtained are matched. Secondly, the matching points are taken as the ground control points of the scan line, which is cut into several short line segments. Finally, all line segments are matched to obtain the disparity of the scan line. The experimental results show that the matching speed is accelerated greatly with improved disparity image quality

Accurate real-time neural disparity MAP estimation with FPGA

We propose in this paper a new method for real-time dense disparity map computing using a stereo pair of rectified images. Based on the neural network and Disparity Space Image (DSI) data structure, the disparity map computing consists of two main steps: initial disparity map estimation by combining the neuronal network and the DSI structure, and its refinement. Four improvements are introduced so that an accurate and fast result will be reached. The first one concerns the proposition of a new strategy in order to optimize the computation time of the initial disparity map. In the second one, a specific treatment is proposed in order to obtain more accurate disparity for the neighboring pixels to boundaries. The third one, it concerns the pixel similarity measure for matching score computation and it consists of using in addition to the traditional pixel intensities, the magnitude and orientation of the gradients providing more accuracy. Finally, the processing time of the method has been decreased consequently to our implementation of some critical steps on FPGAs. Experimental results on real datasets are conducted and a comparative evaluation of the obtained results relative to the state-of-art methods is presented.

Accurate Real-Time Disparity Map Computation Based on Variable Support Window

We propose in this paper a real-time dense stereo matching algorithm using variable support window for disparity map computing. Basic real-time local algorithms relying on a fixed and rectangular correlation window suffer from the difficulty for window-based methods lies in determining the best window shape and size for each pixel. This work proposes a novel local approach using a combination of the DSI (Disparity Space Image) structure and gradient information. Two improvements are introduced so that an accurate and fast result will be reached. The first one concerns the proposition of a new strategy in order to optimize the computation time of the initial disparity map. The second one, concerns the pixel similarity measure for matching score computation and it consists to use in addition to the traditional pixel intensities, the magnitude and orientation of the gradients providing more accuracy. Experimental results on real data sets are conducted and a comparative evaluation of the obtained results relatively to the state-of-art methods is presented.