Depth Discontinuities Research Articles

Context pyramidal network for stereo matching regularized by disparity gradients

Also after many years of research, stereo matching remains to be a challenging task in photogrammetry and computer vision. Recent work has achieved great progress by formulating dense stereo matching as a pixel-wise learning task to be resolved with a deep convolutional neural network (CNN). However, most estimation methods, including traditional and deep learning approaches, still have difficulty to handle real-world challenging scenarios, especially those including large depth discontinuity and low texture areas.To tackle these problems, we investigate a recently proposed end-to-end disparity learning network, DispNet (Mayer et al., 2015), and improve it to yield better results in these problematic areas. The improvements consist of three major contributions. First, we use dilated convolutions to develop a context pyramidal feature extraction module. A dilated convolution expands the receptive field of view when extracting features, and aggregates more contextual information, which allows our network to be more robust in weakly textured areas. Second, we construct the matching cost volume with patch-based correlation to handle larger disparities. We also modify the basic encoder-decoder module to regress detailed disparity images with full resolution. Third, instead of using post-processing steps to impose smoothness in the presence of depth discontinuities, we incorporate disparity gradient information as a gradient regularizer into the loss function to preserve local structure details in large depth discontinuity areas.We evaluate our model in terms of end-point-error on several challenging stereo datasets including Scene Flow, Sintel and KITTI. Experimental results demonstrate that our model decreases the estimation error compared with DispNet on most datasets (e.g. we obtain an improvement of 46% on Sintel) and estimates better structure-preserving disparity maps. Moreover, our proposal also achieves competitive performance compared to other methods.

Visibility restoration of single image captured in dust and haze weather conditions

Inclement weather existence of fog, haze, and dust generally degrades the visibility of outdoor images. Bad visibility may cause failure in computer vision applications. Existing dehazing methods cannot work well on dust haze images. Such images appear yellowish due to the absorption of blue light by dust particles. In order to solve this problem, we propose an optical compensation method (OCM), which uses histogram matching to change the RGB color channel. In this method, take the red channel as a reference curve and then fit the color curves of the blue and green channels close to the red curve, thereby dust haze images are transformed into haze images. Furthermore, we develop a novel single image dehazing method based on Gaussian adaptive transmission (GAT). GAT uses a Gaussian function with a linear coefficient constraint to optimize the transmission, which can prevent halo artifacts near edges of depth discontinuity and obtaining a more accurate estimate of the transmission, especially in bright areas (e.g., sky). The experimental results show that OCM can eliminate the color cast of dust haze images, and GAT can enhance the visibility of dust haze images effectively.

Bin-Picking for Planar Objects Based on a Deep Learning Network: A Case Study of USB Packs.

Random bin-picking is a prominent, useful, and challenging industrial robotics application. However, many industrial and real-world objects are planar and have oriented surface points that are not sufficiently compact and discriminative for those methods using geometry information, especially depth discontinuities. This study solves the above-mentioned problems by proposing a novel and robust solution for random bin-picking for planar objects in a cluttered environment. Different from other research that has mainly focused on 3D information, this study first applies an instance segmentation-based deep learning approach using 2D image data for classifying and localizing the target object while generating a mask for each instance. The presented approach, moreover, serves as a pioneering method to extract 3D point cloud data based on 2D pixel values for building the appropriate coordinate system on the planar object plane. The experimental results showed that the proposed method reached an accuracy rate of 100% for classifying two-sided objects in the unseen dataset, and 3D appropriate pose prediction was highly effective, with average translation and rotation errors less than 0.23 cm and 2.26°, respectively. Finally, the system success rate for picking up objects was over 99% at an average processing time of 0.9 s per step, fast enough for continuous robotic operation without interruption. This showed a promising higher successful pickup rate compared to previous approaches to random bin-picking problems. Successful implementation of the proposed approach for USB packs provides a solid basis for other planar objects in a cluttered environment. With remarkable precision and efficiency, this study shows significant commercialization potential.

Multi-frame stereo matching with edges, planes, and superpixels

We present a multi-frame narrow-baseline stereo matching algorithm based on extracting and matching edges across multiple frames. Edge matching allows us to focus on the important features at the very beginning, and deal with occlusion boundaries as well as untextured regions. Given the initial sparse matches, we fit overlapping local planes to form a coarse, over-complete representation of the scene. After breaking up the reference image in our sequence into superpixels, we perform a Markov random field optimization to assign each superpixel to one of the plane hypotheses. Finally, we refine our continuous depth map estimate using a piecewise-continuous variational optimization. Our approach successfully deals with depth discontinuities, occlusions, and large textureless regions, while also producing detailed and accurate depth maps. We show that our method out-performs competing methods on high-resolution multi-frame stereo benchmarks and is well-suited for view interpolation applications.

Object occlusion guided stereo image retargeting

The resizing of stereo images while preserving salient content and geometric consistency between the image pair is an essential problem to be addressed for 3D visualization. Existing stereo image retargeting techniques are observed to be incurring salient object deformation. In this paper, we formulate a seam carving method to perform stereo image retargeting using graph-cuts having node size as the number of pixels in one of the stereo images. We define the object map with depth ordering of each object from the camera. The seams are allowed to pass along the object-object occlusion boundary at depth discontinuity in order to prevent salient object deformation. We propose adaptive occlusion boundary weights as a function of an object area to be occluded to preserve small objects. The seam passing through object-object occlusion boundary in one image may not follow the exact boundary in the other image. We propose a refinement of the seam to follow object-object occlusion boundary in both the stereo images. By comparing the proposed method with other state-of-the-art stereo image retargeting methods, we show that the proposed method is able to preserve the salient object(s) along with geometric consistency between the stereo image pair.

Dense Stereo Matching Based on Multiobjective Fitness Function—A Genetic Algorithm Optimization Approach for Stereo Correspondence

Dense stereo image matching in remotely sensed images is a challenging problem, though it has been studied for more than two decades, due to occlusions, discontinuities, geometric, and radiometric distortions. A novel multiobjective fitness function-based dense stereo matching approach using genetic algorithms (GAs) is proposed in this paper. The proposed method is useful for estimating dense disparity map with an improved number of inliers for a stereo image pair, despite the constraint of finding correct disparity at depth discontinuities. In this paper, the steps of GA, such as initialization of the population, fitness function, and crossover and mutation operation, are designed and implemented to effectively deal with the problem of dense stereo image matching. To initialize the population, a Scale Invariant Feature Transform (SIFT) descriptor is computed for each pixel and multiple-size window-based matching is performed, using the similarity measures: 1) Euclidean distance and 2) spectral angle mapper. The generated disparity maps are pruned to choose a suitable subset using the designed fitness functions, considering the constraints related to stereo image pair, such as epipolar constraint, which encodes the epipolar geometry and the similarity measure that is useful to decide accuracy of the correspondences. The two objective functions are the number of inliers computed using the fundamental matrix and an energy minimization function, considering discontinuities and occlusions. The usefulness of this approach for remotely sensed stereo image pairs is demonstrated by improving the number of inliers and favorably comparing with state-of-the-art dense stereo image matching methods.

Depth Map Upsampling via Multi-Modal Generative Adversarial Network.

Autonomous robots for smart homes and smart cities mostly require depth perception in order to interact with their environments. However, depth maps are usually captured in a lower resolution as compared to RGB color images due to the inherent limitations of the sensors. Naively increasing its resolution often leads to loss of sharpness and incorrect estimates, especially in the regions with depth discontinuities or depth boundaries. In this paper, we propose a novel Generative Adversarial Network (GAN)-based framework for depth map super-resolution that is able to preserve the smooth areas, as well as the sharp edges at the boundaries of the depth map. Our proposed model is trained on two different modalities, namely color images and depth maps. However, at test time, our model only requires the depth map in order to produce a higher resolution version. We evaluated our model both quantitatively and qualitatively, and our experiments show that our method performs better than existing state-of-the-art models.

A Real-Time High-Quality Complete System for Depth Image-Based Rendering on FPGA

Depth image-based rendering (DIBR) techniques have recently drawn more attention in various 3D applications. In this paper, a real-time high-quality DIBR system that consists of disparity estimation and view synthesis is proposed. For disparity estimation, a local approach that focuses on depth discontinuities and disparity smoothness is presented to improve the disparity accuracy. For view synthesis, a method that contains view interpolation and extrapolation is proposed to render high-quality virtual views. Moreover, the system is designed with an optimized parallelism scheme to achieve a high throughput, and can be scaled up easily. It is implemented on an Altera Stratix IV FPGA at a processing speed of 45 frames per second for 1080p resolution. Evaluated on selected image sets of the Middlebury benchmark, the average error rate of the disparity maps is 6.02%; the average peak signal to noise ratio and structural similarity values of the virtual views are 30.07 dB and 0.9303, respectively. The experimental results indicate that the proposed DIBR system has the top-performing processing speed and its accuracy performance is among the best of state-of-the-art hardware implementations.

Motion parallax for 360° RGBD video

We present a method for adding parallax and real-time playback of 360° videos in Virtual Reality headsets. In current video players, the playback does not respond to translational head movement, which reduces the feeling of immersion, and causes motion sickness for some viewers. Given a 360° video and its corresponding depth (provided by current stereo 360° stitching algorithms), a naive image-based rendering approach would use the depth to generate a 3D mesh around the viewer, then translate it appropriately as the viewer moves their head. However, this approach breaks at depth discontinuities, showing visible distortions, whereas cutting the mesh at such discontinuities leads to ragged silhouettes and holes at disocclusions. We address these issues by improving the given initial depth map to yield cleaner, more natural silhouettes. We rely on a three-layer scene representation, made up of a foreground layer and two static background layers, to handle disocclusions by propagating information from multiple frames for the first background layer, and then inpainting for the second one. Our system works with input from many of today's most popular 360° stereo capture devices (e.g., Yi Halo or GoPro Odyssey), and works well even if the original video does not provide depth information. Our user studies confirm that our method provides a more compelling viewing experience than without parallax, increasing immersion while reducing discomfort and nausea.

A Perception-driven Hybrid Decomposition for Multi-layer Accommodative Displays.

Multi-focal plane and multi-layered light-field displays are promising solutions for addressing all visual cues observed in the real world. Unfortunately, these devices usually require expensive optimizations to compute a suitable decomposition of the input light field or focal stack to drive individual display layers. Although these methods provide near-correct image reconstruction, a significant computational cost prevents real-time applications. A simple alternative is a linear blending strategy which decomposes a single 2D image using depth information. This method provides real-time performance, but it generates inaccurate results at occlusion boundaries and on glossy surfaces. This paper proposes a perception-based hybrid decomposition technique which combines the advantages of the above strategies and achieves both real-time performance and high-fidelity results. The fundamental idea is to apply expensive optimizations only in regions where it is perceptually superior, e.g., depth discontinuities at the fovea, and fall back to less costly linear blending otherwise. We present a complete, perception-informed analysis and model that locally determine which of the two strategies should be applied. The prediction is later utilized by our new synthesis method which performs the image decomposition. The results are analyzed and validated in user experiments on a custom multi-plane display.

VR Exploration Assistance through Automatic Occlusion Removal

Virtual Reality (VR) applications allow a user to explore a scene intuitively through a tracked head-mounted display (HMD). However, in complex scenes, occlusions make scene exploration inefficient, as the user has to navigate around occluders to gain line of sight to potential regions of interest. When a scene region proves to be of no interest, the user has to retrace their path, and such a sequential scene exploration implies significant amounts of wasted navigation. Furthermore, as the virtual world is typically much larger than the tracked physical space hosting the VR application, the intuitive one-to-one mapping between the virtual and real space has to be temporarily suspended for the user to teleport or redirect in order to conform to the physical space constraints. In this paper we introduce a method for improving VR exploration efficiency by automatically constructing a multiperspective visualization that removes occlusions. For each frame, the scene is first rendered conventionally, the z-buffer is analyzed to detect horizontal and vertical depth discontinuities, the discontinuities are used to define disocclusion portals which are 3D scene rectangles for routing rays around occluders, and the disocclusion portals are used to render a multiperpsective image that alleviates occlusions. The user controls the multiperspective disocclusion effect, deploying and retracting it with small head translations. We have quantified the VR exploration efficiency brought by our occlusion removal method in a study where participants searched for a stationary target, and chased a dynamic target. Our method showed an advantage over conventional VR exploration in terms of reducing the navigation distance, the view direction rotation, the number of redirections, and the task completion time. These advantages did not come at the cost of a reduction in depth perception or situational awareness, or of an increase in simulator sickness.

Receiver function imaging of mantle transition zone discontinuities and the origin of volcanism beneath Libya

The origin of magma at Libya’s volcanic provinces is still poorly understood. In order to constrain these sources we report results of the first P-to-S receiver function investigation of the 410 km and 660 km depth discontinuities bounding the mantle transition zone (MTZ) beneath Libya. The dataset used by this study is a combination of eighteen seismic stations belonging to the Libyan Center for Remote Sensing and Space Science and the Incorporated Research Institutions for Seismology Data Management Center. The average thickness of the MTZ is found to be 249 ± 14 km, which is similar to the global average. A 10 km thinning of the MTZ was observed beneath the Miocene – Holocene volcanic provinces in central Libya, suggesting higher-than-normal temperatures in those parts of the MTZ. Our preferred model suggests that the origin of volcanism in Libya is due to higher temperatures in the MTZ beneath this region. However, in Eastern Libya, the thickness of the MTZ increases from 249 km to 270 km which indicates a colder-than-normal MTZ which may relate to ancient subducted slabs due to the Africa-Europe convergence since the late Mesozoic.

An Efficient Dense Stereo Matching Method for Planetary Rover

Stereo matching is one of the most important and challenging subjects in the field of planetary rover with a stereo vision system. The generated disparity map can make rover to avoid the obstacle and explore the planetary surface automatically. In this paper, we propose an efficient dense stereo matching method to generate disparity maps for a planetary rover, which relies on 3-D plane fitting, adaptive penalties, and coarse-to-fine disparity constraint. In order to achieve efficient stereo matching at the coarsest level of the pyramid, we present a 3-D plane fitting to reduce the disparity search range and propose adaptive penalties in the more-global matching method for obtaining an accurate disparity map. At the finer level, our method then estimates the disparity search range based on coarse-to-fine disparity constraint and utilizes adaptive penalties to obtain an accurate disparity map. The extensive experiments with stereo images of Chang'e-3 rover demonstrate that our approach can generate disparity maps efficiently and accurately compared with the most state-of-the-art semi-global matching methods, especially at low texture regions, depth discontinuities, and occlusion regions.

Learning Efficient Stereo Matching Network With Depth Discontinuity Aware Super-Resolution

Deep convolutional neural networks (CNNs) have shown great potential to provide accurate depth estimation based on stereo images. Previous work has focused on developing robust stereo matching architectures, while little attention has been paid on improving the network efficiency. In this paper, we propose an efficient Siamese CNN architecture that combines the low resolution disparity estimation and the depth discontinuity aware super-resolution. Specifically, we propose to construct, filter and perform regression on a low resolution cost volume through the designed stereo matching backbone network. A fast depth discontinuity aware super-resolution subnetwork is proposed for upsampling the low resolution disparity map to the desired resolution. Under the guidance of the intensity edge features extracted from the left color image, depth edge residuals are hierarchically learned to refine the upsampled depth map. A delayed upsampling structure is designed to ensure that the computational complexity is proportional to the spatial size of the input disparity map. We also propose to supervise the first derivative loss of the predicted disparity map that makes the network adaptively aware of the depth discontinuity edges. Experiments show that the proposed stereo matching network achieves a comparable prediction accuracy and much faster running speed compared with state-of-the-art methods.

Disparity Refinement in Depth Discontinuity Using Robustly Matched Straight Lines for Digital Surface Model Generation

Dense image matching using remote sensing images is of particular importance for providing ground geometric data for object modeling and analysis. These methods normally operate in a rectified (epipolar) space producing disparity images that are correlated with the final three-dimensional geometry. However, the resulting digital surface models can be problematic on discontinuities of the terrain object, e.g., building edges, which largely limit their practical use for highly accurate object modeling. Existing works put forth efforts for dealing with this issue by defining better edge constraints and more global energy optimization, while intended to improve the disparity maps, it might be challenging to leverage the result of all the pixels through a single energy minimization, leading to either oversmoothed object boundaries or noisy surfaces. In this paper, we propose an intuitive method that integrates straight line primitives to enhance the disparity map. For each matched line, we perform a local discontinuity analysis and propose an intensity-based weighting method for a local plane fitting using iteratively solved weighted least squares adjustment, such that straightness of the object's edges (e.g., buildings) can be preserved, as the straight lines are detected and matched. Experiments on both aerial and satellite dataset show that the proposed method yields visually clear object edges and has achieved higher accuracy (2-3 pixels improvement) around edges than DSM generated from typical stereo matching result (i.e., from semiglobal matching).

Light-field-based absolute phase unwrapping.

Ambiguity caused by a wrapped phase is an intrinsic problem in fringe projection-based 3D shape measurement. Among traditional methods for avoiding phase ambiguity, spatial phase unwrapping is sensitive to sensor noise and depth discontinuity, and temporal phase unwrapping requires additional encoding information that leads to an increase of image sequence acquisition time or a reduction of fringe contrast. Here, to the best of our knowledge, we report a novel method of absolute phase unwrapping based on light field imaging. In a recorded light field under structured illumination, i.e., a structured light field, a wrapped phase-encoded field can be retrieved and resampled in diverse image planes associated with several possible fringe orders in a measurement volume. Then, by leveraging phase consistency constraint in the resampled wrapped phase-encoded field, correct fringe orders can be determined to unwrap the wrapped phase without any additional encoding information. Experimental results demonstrated that the proposed method was suitable for accurate and robust absolute phase unwrapping.

3D Stereo Reconstruction of SEM Images

In this work is proposed a new fully automated methodology using computer vision and dynamic programming to obtain a 3D reconstruction model of surfaces using scanning electron microscope (SEM) images based on stereovision. The horizontal stereo matching step is done with a robust and efficient algorithm based on semi-global matching. The cost function used in this study is very simple since the brightness and contrast change of corresponding pixels is negligible for the small tilt involved in stereo SEM. It is used a sum of absolute differences (SAD) over a variable pixel size window. Since it relies on dynamic programming, the matching algorithm uses an occlusion parameter which penalizes large depth discontinuities and, in practice, smooths the disparity map and the corresponding reconstructed surface. This step yields a disparity map, i.e. the differences between the horizontal coordinates of the matching points in the stereo images. The horizontal disparity map is finally converted into heights according to the SEM acquisition parameters: tilt angle, image magnification and pixel size. A validation test was first performed using as reference a microscopic grid with manufacturer specifications. Finally, with the 3D model are proposed some applications in materials science as roughness parameters estimation and wear measurements.

Weighted Large Margin Nearest Center Distance-Based Human Depth Recovery With Limited Bandwidth Consumption.

This paper proposes a weighted large margin nearest center (WLMNC) distance-based human depth recovery method for tele-immersive video interaction systems with limited bandwidth consumption. In the remote stage, the proposed method highly compresses the depth data of the remote human into skeletal block structures by learning the WLMNC distance, which is equivalent to downsampling the human depth map at $64{\times}$ the sampling rate. In the local stage, the method first recovers a rough human depth map based on a WLMNC distance augmented clustering approach and then obtains a fine depth map based on a rough depth-guided autoregressive model to preserve the depth discontinuities and suppress texture copy artifacts. The proposed WLMNC distance is learned by the large margin clustering problem with a weighted hinge loss to balance the clustering accuracy and depth recovery accuracy and is verified to be able to preserve depth discontinuities between skeletal block structures with occlusion. A theoretical analysis is conducted to verify the effectiveness of using the weighted hinge loss. Furthermore, a novel data set containing various types of human postures with self-occlusion is built to benchmark the human depth recovery methods. The quantitative comparison with the state-of-the-art depth recovery methods on the introduced benchmark data set demonstrates the effectiveness of the proposed method for human depth recovery with such a high upsampling rate.

Elemental image array generation method by using optimized depth image‐based rendering algorithm for integral imaging display

AbstractWe propose an elemental image array (EIA) generation method by using an optimized depth image‐based rendering (DIBR) algorithm. In this method, the EIA is synthesized by the reference and virtual viewpoint elemental images, and the virtual viewpoint elemental images at the given locations are generated by DIBR algorithm. We optimize the existing DIBR algorithm by adaptively repairing the warped depth images in the processing part and extend the generation dimension of the virtual viewpoint elemental images from one dimension to two dimensions. The optimized DIBR algorithm can effectively solve the problem: the low quality of virtual viewpoint elemental images caused by discontinuous depth values and disocclusion regions. We also implement the generations of virtual viewpoint elemental images and EIA in graph processing unit to reduce the time cost. Experimental results show that the proposed method can not only improve the quality of the virtual viewpoint images but also accelerate the generations of the virtual viewpoint elemental images and EIA.

DepthCut: improved depth edge estimation using multiple unreliable channels

In the context of scene understanding, a variety of methods exists to estimate different information channels from mono or stereo images, including disparity, depth, and normals. Although several advances have been reported in the recent years for these tasks, the estimated information is often imprecise particularly near depth discontinuities or creases. Studies have however shown that precisely such depth edges carry critical cues for the perception of shape, and play important roles in tasks like depth-based segmentation or foreground selection. Unfortunately, the currently extracted channels often carry conflicting signals, making it difficult for subsequent applications to effectively use them. In this paper, we focus on the problem of obtaining high-precision depth edges (i.e., depth contours and creases) by jointly analyzing such unreliable information channels. We propose DepthCut, a data-driven fusion of the channels using a convolutional neural network trained on a large dataset with known depth. The resulting depth edges can be used for segmentation, decomposing a scene into depth layers with relatively flat depth, or improving the accuracy of the depth estimate near depth edges by constraining its gradients to agree with these edges. Quantitatively, we compare against 18 variants of baselines and demonstrate that our depth edges result in an improved segmentation performance and an improved depth estimate near depth edges compared to data-agnostic channel fusion. Qualitatively, we demonstrate that the depth edges result in superior segmentation and depth orderings. (Code and datasets will be made available.)