Accurate Depth Map Research Articles

Depth map enhancement using adaptive moving least squares method with a total variation minimization

Accurate and fast depth map acquisition and enhancement is an important issue in the area of computer vision and image processing. In this study, we present a novel method for enhancing noisy depth maps using adaptive total variation minimization, which facilitates noise smoothing and boundary sharpening for a given depth map image but without previous information. We filter the noise in the depth map with a refined total variation minimization technique. Our experimental results demonstrate that the proposed method outperforms other competitive methods in both objective and subjective comparisons of depth map enhancement and denoising.

Hallucinating Stereoscopy from a Single Image

AbstractWe introduce a novel method for enabling stereoscopic viewing of a scene from a single pre‐segmented image. Rather than attempting full 3D reconstruction or accurate depth map recovery, we hallucinate a rough approximation of the scene's 3D model using a number of simple depth and occlusion cues and shape priors. We begin by depth‐sorting the segments, each of which is assumed to represent a separate object in the scene, resulting in a collection of depth layers. The shapes and textures of the partially occluded segments are then completed using symmetry and convexity priors. Next, each completed segment is converted to a union of generalized cylinders yielding a rough 3D model for each object. Finally, the object depths are refined using an iterative ground fitting process. The hallucinated 3D model of the scene may then be used to generate a stereoscopic image pair, or to produce images from novel viewpoints within a small neighborhood of the original view. Despite the simplicity of our approach, we show that it compares favorably with state‐of‐the‐art depth ordering methods. A user study was conducted showing that our method produces more convincing stereoscopic images than existing semi‐interactive and automatic single image depth recovery methods.

Snow depth mapping in high-alpine catchments using digital photogrammetry

Abstract. Information on snow depth and its spatial distribution is crucial for numerous applications in snow and avalanche research as well as in hydrology and ecology. Today, snow depth distributions are usually estimated using point measurements performed by automated weather stations and observers in the field combined with interpolation algorithms. However, these methodologies are not able to capture the high spatial variability of the snow depth distribution present in alpine terrain. Continuous and accurate snow depth mapping has been successfully performed using laser scanning but this method can only cover limited areas and is expensive. We use the airborne ADS80 optoelectronic scanner, acquiring stereo imagery with 0.25 m spatial resolution to derive digital surface models (DSMs) of winter and summer terrains in the neighborhood of Davos, Switzerland. The DSMs are generated using photogrammetric image correlation techniques based on the multispectral nadir and backward-looking sensor data. In order to assess the accuracy of the photogrammetric products, we compare these products with the following independent data sets acquired simultaneously: (a) manually measured snow depth plots; (b) differential Global Navigation Satellite System (dGNSS) points; (c) terrestrial laser scanning (TLS); and (d) ground-penetrating radar (GPR) data sets. We demonstrate that the method presented can be used to map snow depth at 2 m resolution with a vertical depth accuracy of ±30 cm (root mean square error) in the complex topography of the Alps. The snow depth maps presented have an average accuracy that is better than 15 % compared to the average snow depth of 2.2 m over the entire test site.

Beyond the interference problem: hierarchical patterns for multiple-projector structured light system.

Three-dimensional reconstruction of a dynamic object based on the structured light (SL) technique has attracted intensive research. Since a single projector only covers a limited area of the scene, multiple projectors should be employed simultaneously to extend the imaging area for a 3D object. However, patterns projected by different projectors superpose each other in a light-intersected area, which makes it difficult to recognize the original patterns and obtain the correct depth maps. To solve such a problem, we propose a method to design hierarchical patterns that can be separated from each other. In the proposed patterns, each pixel in binary patterns based on the de Bruijn sequence is replaced by a different bin with limited size. Then the proposed patterns can be separated by identifying distributions of colors in each bin in superposed patterns, and depth maps are obtained by decoding the separated patterns. To verify the performance of the proposed method, we design two hierarchical patterns and conduct several experiments in different scenes. Experimental results demonstrate that the proposed patterns can be separated in a multiple-projector SL system to obtain accurate depth maps, and they are robust for different conditions.

2D to 3D conversion with motion-type adaptive depth estimation

2D to 3D conversion is an important task for 3DTV broadcasting services due to the lack of stereoscopic 3D contents. In this paper, we propose 2D to 3D conversion with motion-type adaptive depth estimation. Because the most important depth cue is motion parallax in our method, we first perform motion estimation between sequential video frames. Then, we adopt a motion-type adaptive approach to depth map estimation because videos have different depth structures according to the type of motion. To be specific, depth from motion is exploited to estimate depth maps in the case of global motion while the depth maps are generated based on the depth from template with the local motion-guided refinement in the case of local motion. Finally, we employ depth image-based rendering (DIBR) to generate stereoscopic virtual views from the depth maps. Experimental results demonstrate that the proposed 2D to 3D conversion is very effective in generating accurate depth maps and providing realistic 3D effects.

Moving object-based depth map estimation using relabeling and hybrid matching

Schemes to enhance human visual perception in three-dimensional (3-D) video applications with depth map data are proposed. Depth estimation is an important part of free viewpoint television and 3-DTV because the accuracy of depth information directly affects the synthesized video quality at an intermediate viewpoint. However, generating an accurate depth map is a complex computational process that makes real-time implementation challenging. In order to obtain accurate depth information with low complexity, a depth map relabeling algorithm and a hybrid matching algorithm are proposed in the depth estimation step. These techniques in acquisition for a depth map are based on human perception, which is more sensitive to moving objects than to a static background. Also, they consider the importance of appropriate processing of object boundaries. Experimental results demonstrate that the proposed schemes provide a synthesized view with both higher subjective visual quality and better objective quality in terms of peak signal-to-noise ratio than legacy depth estimation reference software.

Inpainting algorithm for Kinect depth map based on foreground segmentation

The depth information of the scene indicates the distance between the object and the camera, and depth extraction is a key technology in 3D video system. The emergence of Kinect makes the high resolution depth map capturing possible. However, the depth map captured by Kinect can not be directly used due to the existing holes and noises, which needs to be repaired. We propose a texture combined inpainting algorithm in this paper. Firstly, the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map. Secondly, region growing is used to determine the match region of the hole in the depth map, and to accurately position the match region according to the texture information. Then the match region is weighted to fill the hole. Finally, a Gaussian filter is used to remove the noise in the depth map. Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map, which can be used to render a virtual image with good quality.

Depth error compensation for camera fusion system

When the three-dimensional (3-D) video system includes a multiview video generation technique using depth data to provide more realistic 3-D viewing experiences, accurate depth map acquisition is an important task. In order to generate the precise depth map in real time, we can build a camera fusion system with multiple color cameras and one time-of-flight (TOF) camera; however, this method is associated with depth errors, such as depth flickering, empty holes in the warped depth map, and mixed pixels around object boundaries. In this paper, we propose three different methods for depth error reduction to minimize such depth errors. In order to reduce depth flickering in the temporal domain, we propose a temporal enhancement method using a modified joint bilateral filtering at the TOF camera side. Then, we fill the empty holes in the warped depth map by selecting a virtual depth and applying a weighted depth filtering method. After hole filling, we remove mixed pixels and replace them with new depth values using an adaptive joint multilateral filter. Experimental results show that the proposed method reduces depth errors significantly in near real time.

Joint Depth Map and Color Consistency Estimation for Stereo Images with Different Illuminations and Cameras

Abstract—In this paper, we propose a method that infers both accurate depth maps and color-consistent stereo images for radiometrically varying stereo images. In general, stereo matching and performing color consistency between stereo images are a chicken-and-egg problem since it is not a trivial task to simultaneously achieve both goals. Hence, we have developed an iterative framework in which these two processes can boost each other. First, we transform the input color images to log-chromaticity color space, from which a linear relationship can be established during constructing a joint pdf of transformed left and right color images. From this joint pdf, we can estimate a linear function that relates the corresponding pixels in stereo images. Based on this linear property, we present a new stereo matching cost by combining Mutual Information (MI), SIFT descriptor, and segment-based plane-fitting to robustly find correspondence for stereo image pairs which undergo radiometric variations. Meanwhile, we devise a Stereo Color Histogram Equalization (SCHE) method to produce color-consistent stereo image pairs, which conversely boost the disparity map estimation. Experimental results show that our method produces both accurate depth maps and color-consistent stereo images, even for stereo images with severe radiometric differences.

Depth image enlargement using an evolutionary approach

Accurate depth map at high resolution is required in many 3D video display concepts. Given a low-resolution depth map, this paper studies how to enhance its resolution with a registered high-resolution color image. The idea of the proposed approach is that pixels with similar color values and small distances should have similar depth values. Therefore, the known depth values in input depth map can be propagated to estimate the unknown depth values of their neighboring pixels with similar color values and small distances in high-resolution depth map. Different from conventional approaches, the proposed approach utilizes the ant colony optimization (ACO) technique to dispatch artificial ants moving on a coupled graph, which consists of a depth map and a color image. Then these artificial ants propagate the known depth information from the observed low-resolution depth map to its up-sampled counterpart. Experimental results show that the proposed approach achieves high-resolution depth map with more desirable quality than that of several conventional approaches.

Depth estimation for speckle projection system using progressive reliable points growing matching

In this paper, we propose a progressive reliable points growing matching scheme to estimate the depth from the speckle projection image. First a self-adapting binarization is introduced to reduce the influence of inconsistent intensity. Then we apply local window-based correlation matching to get the initial disparity map. After the initialization, we formulate a progressive updating scheme to update the disparity estimation. There are two main steps in each round of updation. At first new reliable points are progressively selected based on three aspects of criterion including matching degree, confidence, and left-right consistency; then prediction-based growing matching is adopted to recalculate the disparity map from the reliable points. Finally, the more accurate depth map can be obtained by subpixel interpolation and transformation. The experimental results well demonstrate the effectiveness and low computational cost of our scheme.

AUTOMATED 3D ROAD SIGN MAPPING WITH STEREOVISION-BASED MOBILE MAPPING EXPLOITING DISPARITY INFORMATION FROM DENSE STEREO MATCHING

Abstract. This paper presents algorithms and investigations on the automated detection, classification and mapping of road signs which systematically exploit depth information from stereo images. This approach was chosen due to recent progress in the development of stereo matching algorithms enabling the generation of accurate and dense depth maps. In comparison to mono imagery-based approaches, depth maps also allow 3D mapping of the objects. This is essential for efficient inventory and for future change detection purposes. Test measurements with the mobile mapping system by the Institute of Geomatics Engineering of the FHNW University of Applied Sciences and Arts Northwestern Switzerland demonstrated that the developed algorithms for the automated 3D road sign mapping perform well, even under difficult to poor lighting conditions. Approximately 90% of the relevant road signs with predominantly red, blue and yellow colors in Switzerland can be detected, and 85% can be classified correctly. Furthermore, fully automated mapping with a 3D accuracy of better than 10 cm is possible.

다양한 스테레오 카메라 배열을 위한 효율적인 깊이 지도 생성 방법

본 논문에서는 평행형 카메라 배열 뿐 아니라 수렴형 카메라 배열에서도 깊이 지도를 직접적으로 구할 수 있는 방법을 제안한다. 기존의 스테레오 정합 방법들은 복잡도를 줄이고 정확도를 향상시키기 위해서 영상 정렬화를 수행한다. 하지만 수렴형 배열에서의 영상 정렬화는 원치 않는 결과를 발생시킨다. 따라서 제안하는 방법은 문제가 되는 영상 정렬화 과정을 생략하고, 영상 자체의 에피폴라 제약 사항을 이용하여 깊이 값을 직접적으로 추출한다. 깊이 지도의 정확도를 보다 향상시키기 위해서 폐색 영역을 탐지하고 처리하는 과정을 추가적으로 수행한다. 탐지한 폐색 영역은 보이는 주변 영역의 화소들과의 거리와 색상차를 고려하여 적절한 깊이 값으로 채워진다. 실험을 통해 제안한 방법이 기존의 방법에 비해 카메라 배열에 대한 제약이 적고, 안정적으로 깊이 지도를 생성할 수 있음을 확인했다. In this paper, we propose a direct depth map acquisition method for the convergence camera array as well as the parallel camera array. The conventional methods perform image rectification to reduce complexity and improve accuarcy. However, image rectification may lead to unwanted consequences for the convergence camera array. Thus, the proposed method excludes image rectification and directly extracts depth values using the epipolar constraint. In order to acquire a more accurate depth map, occlusion detection and handling processes are added. Reasonable depth values are assigned to the obtained occlusion region by the distance and color differences from neighboring pixels. Experimental results show that the proposed method has fewer limitations than the conventional methods and generates more accurate depth maps stably.

Content Adaptive Enhancement of Multi-View Depth Maps for Free Viewpoint Video

Depth map estimation is an important part of the multi-view video coding and virtual view synthesis within the free viewpoint video applications. However, computing an accurate depth map is a computationally complex process, which makes real-time implementation challenging. Alternatively, a simple estimation, though quick and promising for real-time processing, might result in inconsistent multi-view depth map sequences. To exploit this simplicity and to improve the quality of depth map estimation, we propose a novel content adaptive enhancement technique applied to the previously estimated multi-view depth map sequences. The enhancement method is locally adapted to edges, motion and depth-range of the scene to avoid blurring the synthesized views and to reduce the computational complexity. At the same time, and very importantly, the method enforces consistency across the spatial, temporal and inter-view dimensions of the depth maps so that both the coding efficiency and the quality of the synthesized views are improved. We demonstrate these improvements in the experiments, where the enhancement method is applied to several multi-view test sequences and the obtained synthesized views are compared to the views synthesized using other methods in terms of both numerical and perceived visual quality.

3D dense reconstruction from 2D video sequence via 3D geometric segmentation

3D reconstruction is a major problem in computer vision. This paper considers the problem of reconstructing 3D structures, given a 2D video sequence. This problem is challenging since it is difficult to identify the trajectory of each object point/pixel over time. Traditional stereo 3D reconstruction methods and volumetric 3D reconstruction methods suffer from the blank wall problem, and the estimated dense depth map is not smooth, resulting in loss of actual geometric structures such as planes. To retain geometric structures embedded in the 3D scene, this paper proposes a novel surface fitting approach for 3D dense reconstruction. Specifically, we develop an expanded deterministic annealing algorithm to decompose 3D point cloud to multiple geometric structures, and estimate the parameters of each geometric structure. In this paper, we only consider plane structure, but our methodology can be extended to other parametric geometric structures such as spheres, cylinders, and cones. The experimental results show that the new approach is able to segment 3D point cloud into appropriate geometric structures and generate accurate 3D dense depth map.

Using Graphical Models for Depth Map Estimation

Using Graphical Models for Depth Map Estimation Two slightly different projections of the same scene allow the stereovision algorithms to reconstruct its 3D structure and to estimate the distance to particular object. However commonly used real-time correlation-based solutions usually suffer from inaccuracy. Therefore, finding an efficient and accurate algorithm for depth reconstruction is still a challenging task to do. The approach to stereo matching, presented in this paper is described as a problem of correlating different local observations that evaluate the dissimilarities between left and right images in order to obtain smooth and more accurate depth map. The results obtained with the proposed method are evaluated and compared with other state of the art methods.

Generation of multi-view video using a fusion camera system for 3D displays

In this paper, we present a fusion camera system combining one time-of-flight depth camera and two video cameras to generate multi-view video sequences. In order to obtain the multi-view video using the fusion camera system for 3D displays, we capture a stereo video using a pair of video cameras and a single view depth video with the depth camera. After performing a 3D warping operation for the depth video to obtain an initial depth map at each viewpoint, we refine it using segment-based stereo matching. To reduce mismatched depth values along object boundaries, we detect moving objects using color difference between frames. Finally, we recompute the depth value of each pixel in every segment using stereo matching with a new cost function. Experimental results show that the proposed fusion system produces multi-view video sequences with accurate depth maps, especially along the boundaries of objects. Therefore, it is suitable for generating more natural 3D views for 3D displays than previous works.

Generation of high-quality depth maps using hybrid camera system for 3-D video

In this paper, we present a hybrid camera system combining one time-of-flight depth camera and multiple video cameras to generate multi-view video sequences and their corresponding depth maps. In order to obtain the multi-view video-plus-depth data using the hybrid camera system, we capture multi-view videos using multiple video cameras and a single view depth video with the depth camera. After performing a three-dimensional (3-D) warping operation to obtain an initial depth map at each viewpoint, we refine the initial depth map using segment-based stereo matching. To reduce mismatched depth values along object boundaries, we detect the moving objects using color difference between frames and extract occlusion and disocclusion areas with the initial depth information. Finally, we recompute the depth value of each pixel in each segment using pairwise stereo matching with a proposed cost function. Experimental results show that the proposed hybrid camera system produces multi-view video sequences with more accurate depth maps, especially along the boundary of objects. In addition, it is suitable for generating more natural 3-D views for 3-D TV than previous works..

Nonlinear disparity mapping for stereoscopic 3D

This paper addresses the problem of remapping the disparity range of stereoscopic images and video. Such operations are highly important for a variety of issues arising from the production, live broadcast, and consumption of 3D content. Our work is motivated by the observation that the displayed depth and the resulting 3D viewing experience are dictated by a complex combination of perceptual, technological, and artistic constraints. We first discuss the most important perceptual aspects of stereo vision and their implications for stereoscopic content creation. We then formalize these insights into a set of basic disparity mapping operators. These operators enable us to control and retarget the depth of a stereoscopic scene in a nonlinear and locally adaptive fashion. To implement our operators, we propose a new strategy based on stereoscopic warping of the input video streams. From a sparse set of stereo correspondences, our algorithm computes disparity and image-based saliency estimates, and uses them to compute a deformation of the input views so as to meet the target disparities. Our approach represents a practical solution for actual stereo production and display that does not require camera calibration, accurate dense depth maps, occlusion handling, or inpainting. We demonstrate the performance and versatility of our method using examples from live action post-production, 3D display size adaptation, and live broadcast. An additional user study and ground truth comparison further provide evidence for the quality and practical relevance of the presented work.

A novel method for shape from focus in microscopy using Bezier surface approximation

In this article, we introduce a novel shape from focus method to compute 3D shape of microscopic objects, based on modified-pixel intensities and Bezier surface approximations. A new and simple but effective focus measure is proposed. In our focus measure, the original intensities of a sequence of small neighborhood are modified by subtracting the maximum of the values of first and last frames. An initial depth map is calculated by finding the maximum of the pixel's focused energy and its corresponding frame number. Missing information between two consecutive frames, false depth detection, and enhancement of noise related intensities may provide inaccurate depth map. To overcome these problems and to produce an accurate depth map, we proposed Bezier surface approximation. The proposed method is tested using synthetic and real image sequences. The comparative analysis demonstrates the effectiveness of the proposed method.