Disparity Map Computation Research Articles

New obstacle detection method for a mobile robot based on improved disparity map processing

Stereovision is a crucial technique for vision-based obstacle detection as it provides accurate 3D information for analyzing scenes and detecting obstacles. Despite its effectiveness, existing methods often struggle with real-time performance and computational efficiency. This paper introduces a new stereovision-based obstacle detection system that integrates two novel approaches: Optimized Disparity Map Computation (ODMC), which enhances Hirschmüller’s method for faster disparity map generation, and Disparity Seeded Region Growing (D-SRG), an improved version of the SRG algorithm optimized for identifying obstacles in real-time. Our system is designed to address key challenges such as high memory consumption and slow processing speed, enabling efficient real-time processing from image acquisition to obstacle detection using a 3D camera. Experimental results show that:

Three 1-bit speckle-embedded pulse-width modulation patterns for robust absolute 3D measurement

In three-dimensional (3D) shape measurement techniques using structured light, 1-bit pulse-width modulation (PWM) patterns and 1-bit speckle patterns can be projected at high speed. However, when combining PWM and speckle patterns to integrate their advantages, the decoupling problem is insurmountable. In this work, a novel 1-bit speckle-embedded PWM (SPPWM) method was proposed to achieve absolute 3D shape measurement using only three binary patterns. Our method consists of three main steps: First, a sinusoidal pattern reconstruction network was proposed to eliminate the high-order harmonics and speckle patterns in the SPPWM patterns and obtain high-quality sinusoidal patterns. Second, a multi-temporal spatial correlation matching algorithm was proposed to obtain a coarse disparity map from the three SPPWM patterns. Third, the high-accuracy wrapped phase map is used as an additional constraint for refining the coarse disparity map to obtain the final high-accuracy disparity map for absolute 3D measurement without phase unwrapping. Our method combines the advantages of fringe projection profilometry techniques for high-precision wrapped phase retrieval and speckle correlation matching algorithms for robust and unambiguous disparity map calculation. The experimental results demonstrated that our method could realize high-precision absolute 3D shape measurement with an accuracy of 0.057 mm using only three 1-bit SPPWM patterns. Furthermore, different simulation noises were used to demonstrate the robustness of the proposed method.

FOSS4G DATE FOR DSM GENERATION: SENSITIVITY ANALYSIS OF THE SEMI-GLOBAL BLOCK MATCHING PARAMETERS

Abstract. DATE (Digital Automatic Terrain Extractor) is a Free and Open Source Software for Geospatial (FOSS4G), which combines photogrammetric and computer vision algorithms in order to automatically generate DSMs from multi-view SAR and optical high resolution satellite imagery, following an iterative and pyramidal workflow in order to refine a coarse DSM used as reference. Consequently, DATE is able to face both the issues of DSM generation and epipolar resampling of satellite imagery. The aim of this work is to evaluate DATE performance, by carrying out a sensitivity analysis based on the dense matching parameters. In particular, DATE implements the Semi-Global Block Matching (SGBM) algorithm, a modified version of Semi-Global Matching method: thus, the sensitivity analysis aims at assessing how SGBM parameters – namely, the difference between maximum and minimum disparity (ndisparities), the minimum disparity value (minimumDisp) and the matched block size (SADWindowSize) – affect the efficiency of the disparity map computation and the final DSM accuracy. The analysis focuses on the case study of Trento and of the Adige Valley, which was chosen due to its geomorphological heterogeneity and complexity, allowing to perform an accuracy evaluation on four tiles, characterized by specific roughness frequencies and morphologies (thus having different effects on disparity variations). Several practical indications on the optimal and critical parameter combinations were retrieved; in addition to this, this work highlighted the most influential parameters both in terms of accuracy (minimumDisp) and computation time (ndisparities), paving the way to further principal component analyses. Finally, the obtained results showed no clear relationship between the area morphology and the solution structure.

Disparity map computation of tree using stereo vision system and effects of canopy shapes and foliage density

Monitoring the growth of trees, plants, and crops is an important work in precision agriculture. Tree canopy geometric characteristics are related to tree growth and productivity. Computer vision techniques can be used to map tree canopy volume, which is useful for planning management. This study investigates the potential of using stereo vision system for obtaining tree disparity map for the analysis of geometric attributes. Experiments were conducted to examine the effects of the canopy shapes and foliage density on the performance of stereo vision system in disparity map computation. Two canopy shapes (conic and ellipse) and three foliage density levels were evaluated using two algorithms (algorithm based on local methods (ABLM) and algorithm based on global methods (ABGM)) to match pair stereo images. The well-known Middlebury dataset was considered and the performance of algorithms was evaluated on that. The results showed that the ABGM studied algorithm succeeded in computation disparity map on both Middlebury and trees images because it aggregated matching cost from several directions. The tree canopy shapes and foliage density did not affect the results of algorithms. Also, noises were numerous and more dispersed in ABLM matching algorithm. It was observed that maximum disparity limits search space. Best results were obtained in real value. For smaller value of maximum disparity than that of real value, disparity map had missed disparities. Window size was affected on maps and noise level and best results were obtained when this parameter was set to 15. Smaller value obtained more detailed map but with noise and un-smooth. The algorithm was robust for real trees in natural conditions.

REAL-TIME ON-BOARD OBSTACLE AVOIDANCE FOR UAVS BASED ON EMBEDDED STEREO VISION

Abstract. In order to improve usability and safety, modern unmanned aerial vehicles (UAVs) are equipped with sensors to monitor the environment, such as laser-scanners and cameras. One important aspect in this monitoring process is to detect obstacles in the flight path in order to avoid collisions. Since a large number of consumer UAVs suffer from tight weight and power constraints, our work focuses on obstacle avoidance based on a lightweight stereo camera setup. We use disparity maps, which are computed from the camera images, to locate obstacles and to automatically steer the UAV around them. For disparity map computation we optimize the well-known semi-global matching (SGM) approach for the deployment on an embedded FPGA. The disparity maps are then converted into simpler representations, the so called U-/V-Maps, which are used for obstacle detection. Obstacle avoidance is based on a reactive approach which finds the shortest path around the obstacles as soon as they have a critical distance to the UAV. One of the fundamental goals of our work was the reduction of development costs by closing the gap between application development and hardware optimization. Hence, we aimed at using high-level synthesis (HLS) for porting our algorithms, which are written in C/C++, to the embedded FPGA. We evaluated our implementation of the disparity estimation on the KITTI Stereo 2015 benchmark. The integrity of the overall realtime reactive obstacle avoidance algorithm has been evaluated by using Hardware-in-the-Loop testing in conjunction with two flight simulators.

Edge points-based stereo matching approach for omnidirectional images

We address the problem of matching stereo-omnidirectional images acquired by a catadioptric stereo system. The proposed approach deals with omnidirectional stereo image sequences, which allows taking advantage of the temporal information in both rectification and stereo matching steps. First, spherical edge points detection is performed on the spherical images in order to compute the disparity map only on significant features, i.e., edge points. Second, a spherical rectification is performed on the three omnidirectional pairs that are involved in the disparity map computation at the current frame, such as the current pair, the current left image, and its preceding one, and the current right image and its preceding one. Third, dynamic programming-based stereo matching is applied in each of the three stereo pairs. The disparity maps computed at the preceding frame are exploited to deduce the disparity range that will be used by the dynamic programming. Fourth, the three disparity maps provided by the last step are combined together into one 360-deg disparity map. The proposed method has been tested on a high-resolution omnidirectional images captured with a stereo pair of catadioptric cameras, and the obtained results are satisfactory.

Non parametric methods of disparity computation

Disparity is inversely proportional to depth. Informationabout depth is a key factor in many real time applicationslikecomputer vision applications, medical diagnosis, model precision etc. Disparity is measured first in order to calculate the depth that suitsthe real world applications. There are two approaches viz., active and passive methods. Due to its cost effectiveness, passive approach is the most popular approach. In spite of this, the measures arelimited by its occlusion, more number of objects and texture areas. So, effective and efficient stereo depth estimation algorithms have taken the toll on the researchers. Theimportant goal of stereo vision algorithms is the disparity map calculation between twoimages clicked the same time. These pictures are taken using two cameras. We have implemented the non-parametric algorithmsfor stereo vision viz., Rank and Census transform in both single processor and multicore processors are implemented andthe results showsits time efficient by 1500 times.

Accurate stereo matching using pixel normalized cross correlation in time domain

This paper proposes a novel method for stereo matching which is based on combination of active and passive stereo 3D reconstruction approaches. A laser line is used to scan the reconstructed scene and a stereo camera pair is used for the image acquisition. Each image pixel is scanned at a specific scan time so that the intensity time patterns of the correspondent pixels are highly correlated. This yield in highly confident and accurate disparity map calculation and also allows the reconstruction of poorly textured as well as the extremely textured surface which are very hard to deal with using the conventional passive stereo approaches. The occluded regions are also detected successfully. This method is not computationally intensive and can be used for turning the smartphone into the practical 3D scanner as presented in this work.

Bilaterally Weighted Patches for Disparity Map Computation

Visual correspondence is the key for 3D reconstruction in binocular stereovision. Local methods perform block-matching to compute the disparity, or apparent motion, of pixels between images. The simplest approach computes the distance of patches, usually square windows, and assumes that all pixels in the patch have the same disparity. A prominent artifact of the method is the foreground fattening effect near depth discontinuities. In order to find a more appropriate support, Yoon and Kweon introduced the use of weights based on color similarity and spatial distance, analogous to those used in the bilateral filter. This paper presents the theory of this method and the implementation we have developed. Moreover, some variants are discussed and improvements are used in the final implementation. Several examples and tests are presented and the parameters and performance of the method are analyzed.

Improved Disparity Map Computation on Stereoscopic Streaming Video with Multi-core Parallel Implementation

Stereo vision has become an important technical issue in the field of 3D imaging, machine vision, robotics, image analysis, and so on. The depth map extraction from stereo video is a key technology of stereoscopic 3D video requiring stereo correspondence algorithms. This is the matching process of the similarity measure for each disparity value, followed by an aggregation and optimization step. Since it requires a lot of computational power, there are significant speed-performance advantages when exploiting parallel processing available on processors. In this situation, multi-core CPU may allow many parallel programming technologies to be realized in users computing devices. This paper proposes parallel implementations for calculating disparity map using a shared memory programming and exploiting the streaming SIMD extension technology. By doing so, we can take advantage both of the hardware and software features of multi-core processor. For the performance evaluation, we implemented a parallel SAD algorithm with OpenMP and SSE2. Their processing speeds are compared with non parallel version on stereoscopic streaming video. The experimental results show that both technologies have a significant effect on the performance and achieve great improvements on processing speed.

High Performance Stereo System for Dense 3-D Reconstruction

3-D stereo reconstruction, a technique that estimates per-pixel depth in a scene, is still a challenging problem mainly due to some prohibitive factors that limit its performance and computational ability. The aim of this paper is to present a new hardware-efficient disparity map computation, which is based on disparity space image processing using discrete dynamic systems. The hardware architecture of the proposed system was implemented on a high-end field programmable gate array (FPGA) device, offering real-time 3-D reconstruction speeds using a hardware aware architecture based on parallelism and process pipelining. The proposed architecture fulfills the requirements of real-world applications regarding resource usage, frame rates, and disparity resolution, while its implementation on an Altera Stratix IV family FPGA device can extract disparity maps of up to \(\mathbf {1280 \times 1024}\) pixels with up to 128 disparity levels under real-time or near real-time conditions at a clock rate of 168 MHz. Qualitative and quantitative results also demonstrate its performance and improvement over previous hardware-related studies, making our approach a suitable candidate for applications in which timing and processing constraints are critical.

Hardware-software vision system for moving people counting based on 3D information

Abstract The article presents a hardware-software system for people counting based on analysis of 3D information. It consists of a 3D video source (3D camera), an FPGA evaluation board and a PC computer. The computationaly demanding disparity map calculation and median filtering are realized in hardware. The software part is responsible for people segmentation, tracking and counting. The data between system nodes is transferred via Ethernet. In the paper the system architecture, proposed algorithms and obtained results are presented.

Pixel-based approach for building heights determination by SAR radargrammetry

Abstract. Numerous advances have been made recently in photogrammetry, laser scanning, and remote sensing for the creation of 3D city models. More and more cities are interested in getting 3D city models, be it for urban planning purposes or for supporting public utility companies. In areas often affected by natural disaster, rapid updating of the 3D information may also be useful for helping rescue forces. The high resolutions that can be achieved by the new spaceborne SAR sensor generation enables the analysis of city areas at building level and make those sensors attractive for the extraction of 3D information. Moreover, they present the advantage of weather and sunlight independency, which make them more practicable than optical data, in particular for tasks where rapid response is required. Furthermore, their short revisit time and the possibility of multi-sensor constellation enable providing several acquisitions within a few hours. This opens up the floor for new applications, especially radargrammetric applications, which consider acquisitions taken under different incidence angles. In this paper, we present a new approach for determining building heights, relying only on the radargrammetric analysis of building layover. By taking into account same-side acquisitions, we present the workflow of building height determination. Focus is set on some geometric considerations, pixel-based approach for disparity map calculation, and analysis of the building layover signature for different configurations in order to determine building height.

Scalable hardware architecture for disparity map computation and object location in real-time

We present the disparity map computation core of a hardware system for isolating foreground objects in stereoscopic video streams. The operation is based on the computation of dense disparity maps using block-matching algorithms and two well-known metrics: sum of absolute differences and Census transform. Two sets of disparity maps are computed by taking each of the images as reference so that a consistency check can be performed to identify occluded pixels and eliminate spurious foreground pixels. Taking advantage of parallelism, the proposed architecture is highly scalable and provides numerous degrees of adjustment to different application needs, performance levels and resource usage. A version of the system for 640 × 480 images and a maximum disparity of 135 pixels was implemented in a system based on a Xilinx Virtex II-Pro FPGA and two cameras with a frame rate of 25 fps (less than the maximum supported frame rate of 40 fps on this platform). Implementation of the same system on a Virtex-5 FPGA is estimated to achieve 80 fps, while a version with increased parallelism is estimated to run at 140 fps (which corresponds to the calculation of more than 5.9 × 109 disparity-pixels per second).

Edge-Directed Hardware Architecture for Real-Time Disparity Map Computation

Stereo Vision, a technique aimed at inferring depth information from stereo images, has been used in a wide range of computer vision applications, with real-time requirements in emerging embedded vision systems. Computation of the disparity map, a vital step in extracting depth information from stereo images, requires a significant amount of computational resources. As such, existing software implementations require high-end hardware platforms to achieve real-time frame rates, suggesting that dedicated hardware mechanisms might be more suitable for embedded applications. In this paper, we present a disparity map computation architecture targeting embedded stereo vision applications with hard real-time requirements. The architecture integrates a hardware edge detection mechanism that reduces the search space, improving the overall performance, and is configurable in terms of various application parameters, making it suitable for a number of application environments. The paper also presents a study on the impact of the various parameters in terms of the performance and hardware/power overheads. An experimental prototype of the architecture was implemented on the Xilinx ML505 FPGA Evaluation Platform, achieving 50 Frames Per Second (fps) for 1,280 × 1,024 image sizes. Moreover, the quality of the disparity maps generated by the proposed system is comparable to other existing hardware implementations featuring local stereo correspondence methods.

SAD-Based Stereo Vision Machine on a System-on-Programmable-Chip (SoPC)

This paper, proposes a novel solution for a stereo vision machine based on the System-on-Programmable-Chip (SoPC) architecture. The SOPC technology provides great convenience for accessing many hardware devices such as DDRII, SSRAM, Flash, etc., by IP reuse. The system hardware is implemented in a single FPGA chip involving a 32-bit Nios II microprocessor, which is a configurable soft IP core in charge of managing the image buffer and users' configuration data. The Sum of Absolute Differences (SAD) algorithm is used for dense disparity map computation. The circuits of the algorithmic module are modeled by the Matlab-based DSP Builder. With a set of configuration interfaces, the machine can process many different sizes of stereo pair images. The maximum image size is up to 512 K pixels. This machine is designed to focus on real time stereo vision applications. The stereo vision machine offers good performance and high efficiency in real time. Considering a hardware FPGA clock of 90 MHz, 23 frames of 640 × 480 disparity maps can be obtained in one second with 5 × 5 matching window and maximum 64 disparity pixels.

High-definition three-dimensional television disparity map computation

By reconsidering some two-dimensional video inherited approaches and by adapting them to the stereoscopic video content and to the human visual system peculiarities, a new disparity map is designed. First, the inner relation between the left and the right views is modeled by some weights discriminating between the horizontal and vertical disparities. Second, the block matching operation is achieved by considering a visual related measure (normalized cross correlation) instead of the traditional pixel differences (mean squared error or sum of absolute differences). The advanced three-dimensional (3-D) video-new three step search (3DV-NTSS) disparity map (3-D Video-New Three Step Search) is benchmarked against two state-of-the-art algorithms, namely NTSS and full-search MPEG (FS-MPEG), by successively considering two corpora. The first corpus was organized during the 3DLive French national project and regroups 20 min of stereoscopic video sequences. The second one, with similar size, is provided by the MPEG community. The experimental results demonstrate the effectiveness of 3DV-NTSS in both reconstructed image quality (average gains between 3% and 7% in both PSNR and structural similarity, with a singular exception) and computational cost (search operation number reduced by average factors between 1.3 and 13). The 3DV-NTSS was finally validated by designing a watermarking method for high definition 3-D TV content protection.

Depth Estimation from Stereo Images based on Adaptive Weight and Segmentation

Stereo vision is one of the most researched areas to develop human like vision capability into machines for the purpose of automatic navigation and reconstruction of the real world from images. Point correspondence matching for disparity map calculation is a vital research issue of stereo vision system. Window-based cost aggregation methods used for the correspondence problem have attracted researches as it can be implemented in real time. In this paper, a new hybrid cost aggregation strategy for similarity evaluation based on adaptive weight and color segmentation of stereo image is proposed. In this strategy, pixels which lie on same segment and are spatially closer are given higher weight. Experimental results show that the proposed method is effective in improving overall disparity map and at depth discontinuity.

Stereo correspondence using efficient hierarchical belief propagation

In this paper, a new algorithm is presented to compute the disparity map from a stereo pair of images by using Belief Propagation (BP). While many algorithms have been proposed in recent years, the real-time computation of an accurate disparity map is still a challenging task. The computation time and run-time memory requirements are two very important factors for all real-time applications. The proposed algorithm divides the matching process into two steps; they are initial matching and disparity map refinement. Initial matching is performed by memory efficient hierarchical belief propagation algorithm that uses less than half memory at run-time and minimizes the energy function at much faster rate as compare to other hierarchical BP algorithms that makes it more suitable for real-time applications. Disparity map refinement uses a simple but very effective single-pass approach that improves the accuracy without affecting the computation cost. Experiments by using Middlebury dataset demonstrate that the performance of our algorithm is the best among other real-time stereo matching algorithms.

Efficient 2D to 3D video conversion implemented on DSP

An efficient algorithm to generate three-dimensional (3D) video sequences is presented in this work. The algorithm is based on a disparity map computation and an anaglyph synthesis. The disparity map was first estimated by employing the wavelet atomic functions technique at several decomposition levels in processing a 2D video sequence. Then, we used an anaglyph synthesis to apply the disparity map in a 3D video sequence reconstruction. Compared with the other disparity map computation techniques such as optical flow, stereo matching, wavelets, etc., the proposed approach produces a better performance according to the commonly used metrics (structural similarity and quantity of bad pixels). The hardware implementation for the proposed algorithm and the other techniques are also presented to justify the possibility of real-time visualization for 3D color video sequences.