Depth Recovery Method Research Articles

Reflective Fourier ptychography-based depth-recovery & resolution-enhanced real scene hologram acquisition method

Recording interference holograms using coherent light is a complex process for real-world objects. Alternatively, holography based on integral imaging with natural illumination is considered an option. However, the field of view (FOV) of this technology is limited by the lenslet array of the integral imaging system. This paper proposes an integral imaging-based hologram recording method via a commercial plenoptic camera to increase the FOV for a conventional II system. Meanwhile, to address the problem of depth non-uniform compression caused by the main lens of the plenoptic camera, this paper proposed a depth recovery method combined with the camera’s metadata parameters for depth retargeting. Finally, we introduce a reflective Fourier-ptychography-based algorithm via an optimum overlap rate to acquire high-performance holograms. Compared to stereotypical integral imaging holography, this method produces a higher resolution reconstruction result. In addition, we conduct three sets of experiments on real scenes whose results also prove the proposed method embodies finer reconstruction results, and depth is recovered up to twice as much as the original outcome.

RGB-Guided Depth Map Recovery by Two-Stage Coarse-to-fine Dense CRF Models.

Depth maps generally suffer from large erroneous areas even in public RGB-Depth datasets. Existing learning-based depth recovery methods are limited by insufficient high-quality datasets and optimization-based methods generally depend on local contexts not to effectively correct large erroneous areas. This paper develops an RGB-guided depth map recovery method based on the fully connected conditional random field (dense CRF) model to jointly utilize local and global contexts of depth maps and RGB images. A high-quality depth map is inferred by maximizing its probability conditioned upon a low-quality depth map and a reference RGB image based on the dense CRF model. The optimization function is composed of redesigned unary and pairwise components, which constraint local structure and global structure of depth map, respectively, with the guidance of RGB image. In addition, the texture-copy artifacts problem is handled by two-stage dense CRF models in a coarse-to-fine way. A coarse depth map is first recovered by embedding RGB image in a dense CRF model in unit of 3×3 blocks. It is refined afterward by embedding RGB image in another model in unit of individual pixels and restricting the model mainly work in discontinued regions. Extensive experiments on six datasets verify that the proposed method considerably outperforms a dozen of baseline methods in correcting erroneous areas and diminishing texture-copy artifacts of depth maps.

Bispectral phasor imaging using continuous-wave time-of-flight camera for scattering-scene depth recovery.

In scattering scenes, depth measurements are greatly distorted due to light scattering for Time-of-flight imaging. We propose a bispectral Time-of-flight system and phasor-based depth-recovery method to improve the quality of depth maps in scattering scenes. We reveal that the amplitude of scattered light is wavelength dependent while the phase measured is wavelength independent. The method uses bispectral measurements to nullify the effects of scattering components by calculating the amplitude ratio of scattering phasors. Experimental results demonstrate that the proposed method has a significant improvement in depth recovery with robustness and low computational cost.

Accurate Depth Recovery Method Based on the Fusion of Time-of-Flight and Dot-Coded Structured Light

3D vision technology has been gradually applied to intelligent terminals ever since Apple Inc. introduced structured light on iPhoneX. At present, time-of-flight (TOF) and laser speckle-based structured light (SL) are two mainstream technologies applied to intelligent terminals, both of which are widely regarded as efficient dynamic technologies, but with low accuracy. This paper explores a new approach to achieve accurate depth recovery by fusing TOF and our previous work—dot-coded SL (DCSL). TOF can obtain high-density depth information, but its results may be deformed due to multi-path interference (MPI) and reflectivity-related deviations. In contrast, DCSL can provide high-accuracy and noise-clean results, yet only a limited number of encoded points can be reconstructed. This inspired our idea to fuse them to obtain better results. In this method, the sparse result provided by DCSL can work as accurate “anchor points” to keep the correctness of the target scene’s structure, meanwhile, the dense result from TOF can guarantee full-range measurement. Experimental results show that by fusion, the MPI errors of TOF can be eliminated effectively. Dense and accurate results can be obtained successfully, which has great potential for application in the 3D vision task of intelligent terminals in the future.

Robust illumination invariant depth recovery by combining local phase and intensity information of images

Both intensity and phase information of images have been the most important similarity measures in solving the general stereo matching problem. Intensity contains most of the imaging information of the scene/object, yet the phase information could reflect the local structure of images, which is more robust than the grayscale value. Plenty of work has been done in intensity-based or phase-based stereo matching methods. However, neither of them could work well enough when process images were taken under varied illuminations. A robust depth recovery method by making use of both intensity and phase information of stereo images properly is proposed. Firstly, 2D signal analysis is conducted by using the multiscale monogenic wavelet transform, from which local phase and intensity amplitude information are extracted into different scales. Secondly, disparity maps are estimated in different scales based on the intensity information. Thirdly, the optimal disparity is obtained by weighted-combining the disparity maps in different scales. The weighted coefficients are computed by making use of the phase information. Extensive experimental evaluation demonstrates the benefits of the proposed method.

Depth Sensing by Near-Infrared Light Absorption in Water.

This paper introduces a novel depth recovery method based on light absorption in water. Water absorbs light at almost all wavelengths whose absorption coefficient is related to the wavelength. Based on the Beer-Lambert model, we introduce a bispectral depth recovery method that leverages the light absorption difference between two near-infrared wavelengths captured with a distant point source and orthographic cameras. Through extensive analysis, we show that accurate depth can be recovered irrespective of the surface texture and reflectance, and introduce algorithms to correct for nonidealities of a practical implementation including tilted light source and camera placement, nonideal bandpass filters and the perspective effect of the camera with a diverging point light source. We construct a coaxial bispectral depth imaging system using low-cost off-the-shelf hardware and demonstrate its use for recovering the shapes of complex and dynamic objects in water. We also present a trispectral variant to further improve robustness to extremely challenging surface reflectance. Experimental results validate the theory and practical implementation of this novel depth recovery paradigm, which we refer to as shape from water.

Weighted Large Margin Nearest Center Distance-Based Human Depth Recovery With Limited Bandwidth Consumption.

This paper proposes a weighted large margin nearest center (WLMNC) distance-based human depth recovery method for tele-immersive video interaction systems with limited bandwidth consumption. In the remote stage, the proposed method highly compresses the depth data of the remote human into skeletal block structures by learning the WLMNC distance, which is equivalent to downsampling the human depth map at $64{\times}$ the sampling rate. In the local stage, the method first recovers a rough human depth map based on a WLMNC distance augmented clustering approach and then obtains a fine depth map based on a rough depth-guided autoregressive model to preserve the depth discontinuities and suppress texture copy artifacts. The proposed WLMNC distance is learned by the large margin clustering problem with a weighted hinge loss to balance the clustering accuracy and depth recovery accuracy and is verified to be able to preserve depth discontinuities between skeletal block structures with occlusion. A theoretical analysis is conducted to verify the effectiveness of using the weighted hinge loss. Furthermore, a novel data set containing various types of human postures with self-occlusion is built to benchmark the human depth recovery methods. The quantitative comparison with the state-of-the-art depth recovery methods on the introduced benchmark data set demonstrates the effectiveness of the proposed method for human depth recovery with such a high upsampling rate.

Using 3D face priors for depth recovery

For repairing inaccurate depth measurements from commodity RGB-D sensors, existing depth recovery methods primarily rely on low-level and rigid prior information. However, as the depth quality deteriorates, the recovered depth maps become increasingly unreliable, especially for non-rigid objects. Thus, additional high-level and non-rigid information is needed to improve the recovery quality. Taking as a starting point the human face that is the primary prior available in many high-level tasks, in this paper, we incorporate face priors into the depth recovery process. In particular, we propose a joint optimization framework that consists of two main steps: transforming the face model for better alignment and applying face priors for improved depth recovery. Face priors from both sparse and dense 3D face models are studied. By comparing with the baseline method on benchmark datasets, we demonstrate that the proposed method can achieve up to 23.8% improvement in depth recovery with more accurate face registrations, bringing inspirations to both non-rigid object modeling and analysis.

Color-Guided Depth Recovery via Joint Local Structural and Nonlocal Low-Rank Regularization

High-quality depth recovery from RGB-D data has received increasingly more attention in recent years due to their wide applications from depth-based image rendering to three-dimensional imaging and video. Sharp contrast between high-quality color images and low-quality depth maps presents severe challenges to the development of color-guided depth recovery techniques. Previous works have emphasized either locally varying characteristics of color-depth dependence or nonlocal similarities around the discontinuities of the scene geometry. Therefore, it is desirable to exploit both local and nonlocal structural constraints for optimizing the performance of color-guided depth recovery. In this work, we propose a unified variational approach via joint local and nonlocal regularization. The local regularization term consists of two complementary parts—one characterizing the color-depth dependence in the gradient domain and the other in the spatial domain; nonlocal regularization involves a low-rank constraint suitable for large-scale depth discontinuities. Extensive experimental results are reported to show that our approach outperforms several existing state-of-the-art depth recovery methods on both synthetic and real-world data sets.

A unified 3D face authentication framework based on robust local mesh SIFT feature

In this paper, we design a unified 3D face authentication system for practical use. First, we propose a facial depth recovery method to construct a facial depth map from stereoscopic videos. It effectively utilize prior facial information and incorporate the visibility term to classify static and dynamic pixels for robust depth estimation. Secondly, in order to make 3D face authentication more accurate and consistent, we present an intrinsic scale feature detection for interesting points on 3D facial mesh regions. Then, a novel feature descriptor is proposed, called Local Mesh Scale-Invariant Feature Transform (LMSIFT) to reflect the different face recognition abilities in different facial regions. Finally, the sparse optimization problem of visual codebook is used to 3D face learning. We evaluate our approach on publicly available 3D face databases and self-collected realistic scene databases. We also develop an interactive education system to investigate its performance in practice, which demonstrates the high performance of the proposed approach for accurate 3D face authentication. Compared with previous popular approaches, our system has consistently better performance in terms of effectiveness, robustness and universality.

Hallucinating Stereoscopy from a Single Image

AbstractWe introduce a novel method for enabling stereoscopic viewing of a scene from a single pre‐segmented image. Rather than attempting full 3D reconstruction or accurate depth map recovery, we hallucinate a rough approximation of the scene's 3D model using a number of simple depth and occlusion cues and shape priors. We begin by depth‐sorting the segments, each of which is assumed to represent a separate object in the scene, resulting in a collection of depth layers. The shapes and textures of the partially occluded segments are then completed using symmetry and convexity priors. Next, each completed segment is converted to a union of generalized cylinders yielding a rough 3D model for each object. Finally, the object depths are refined using an iterative ground fitting process. The hallucinated 3D model of the scene may then be used to generate a stereoscopic image pair, or to produce images from novel viewpoints within a small neighborhood of the original view. Despite the simplicity of our approach, we show that it compares favorably with state‐of‐the‐art depth ordering methods. A user study was conducted showing that our method produces more convincing stereoscopic images than existing semi‐interactive and automatic single image depth recovery methods.

Application of optical 3D measurement on thin film buckling to estimate interfacial toughness

The shape-from-focus (SFF) method has been widely studied as a passive depth recovery and 3D reconstruction method for digital images. An important step in SFF is the calculation of the focus level for different points in an image by using a focus measure. In this work, an image entropy-based focus measure is introduced into the SFF method to measure the 3D buckling morphology of an aluminum film on a polymethylmethacrylate (PMMA) substrate at a micro scale. Spontaneous film wrinkles and telephone-cord wrinkles are investigated after the deposition of a 300nm thick aluminum film onto the PMMA substrate. Spontaneous buckling is driven by the highly compressive stress generated in the Al film during the deposition process. The interfacial toughness between metal films and substrates is an important parameter for the reliability of the film/substrate system. The height profiles of different sections across the telephone-cord wrinkle can be considered a straight-sided model with uniform width and height or a pinned circular model that has a delamination region characterized by a sequence of connected sectors. Furthermore, the telephone-cord geometry of the thin film can be used to calculate interfacial toughness. The instability of the finite element model is introduced to fit the buckling morphology obtained by SFF. The interfacial toughness is determined to be 0.203J/m2 at a 70.4° phase angle from the straight-sided model and 0.105J/m2 at 76.9° from the pinned circular model.

Homography-based depth recovery with descent images

In planetary landing exploration task, the images captured by the landing camera are nearly along optical axis which results in multi-resolution images of same terrain surface. Recovering the surface shape of landing terrain from descent imagery is of great value for lander to choose safe landing area. In this paper, a homography-based depth recovery method with descent images is addressed. At first, the parallax and scale change in descent images are analyzed. Second, the camera motion is optimized with SIFT features correspondence constraints. For dense depth recovery, a set of virtual parallel planes is assumed to slice the terrain and each plane induces a homography to warp back the second image to first image plane. Zero-normalized cross-correlation score is chosen to compute the correlation score and the correlation curve is smoothed by two Gaussian filters. The depth for each pixel is determined by the plane which has highest correlation value. At the end, some experiments are conducted, including different correlation computation, depth recovery with different terrain, and the error tests. The results show that the discussed method is feasible to recover the depth information overall.

Analysis of focus measure operators for shape-from-focus

Shape-from-focus (SFF) has widely been studied in computer vision as a passive depth recovery and 3D reconstruction method. One of the main stages in SFF is the computation of the focus level for every pixel of an image by means of a focus measure operator. In this work, a methodology to compare the performance of different focus measure operators for shape-from-focus is presented and applied. The selected operators have been chosen from an extensive review of the state-of-the-art. The performance of the different operators has been assessed through experiments carried out under different conditions, such as image noise level, contrast, saturation and window size. Such performance is discussed in terms of the working principles of the analyzed operators.

Depth from automatic defocusing

This paper presents a depth recovery method that gives the depth of any scene from its defocused images. The method combines depth from defocusing and depth from automatic focusing techniques. Blur information in defocused images is utilised to measure depth in a way similar to determining depth from automatic focusing but without searching for sharp images of objects. The proposed method does not need special scene illumination and involves only a single camera. Therefore, there are no correspondence, occlusion and intrusive emissions problems. The paper gives experimental results which demonstrate the accuracy of the method.

Depth from motion and defocus blur

Finding the distance of an object in a scene from intensity images is an essential problem in many applications. In this work, we present a novel method for depth recovery from a single motion and defocus blurred image. Under the assumption of uniform lateral motion of the camera during finite exposure time, both the pinhole model and the camera with a finite aperture are considered. It is shown that the image blur produced by uniform linear motion of the camera is inversely proportional to the distance of the object. Furthermore, if the speed of the relative motion is known, the depth of the object can be acquired by identifying the blur parameters. An image blur model is formulated based on geometric optics. The blur extent is estimated by intensity profile analysis and focus measurement of the deblurred images. The proposed method is verified experimentally using different types of test patterns in an indoor environment.

Projection defocus analysis for scene capture and image display

In order to produce bright images, projectors have large apertures and hence narrow depths of field. In this paper, we present methods for robust scene capture and enhanced image display based on projection defocus analysis. We model a projector's defocus using a linear system. This model is used to develop a novel temporal defocus analysis method to recover depth at each camera pixel by estimating the parameters of its projection defocus kemel in frequency domain. Compared to most depth recovery methods, our approach is more accurate near depth discontinuities. Furthermore, by using a coaxial projector-camera system, we ensure that depth is computed at all camera pixels, without any missing parts. We show that the recovered scene geometry can be used for refocus synthesis and for depth-based image composition. Using the same projector defocus model and estimation technique, we also propose a defocus compensation method that filters a projection image in a spatially-varying, depth-dependent manner to minimize its defocus blur after it is projected onto the scene. This method effectively increases the depth of field of a projector without modifying its optics. Finally, we present an algorithm that exploits projector defocus to reduce the strong pixelation artifacts produced by digital projectors, while preserving the quality of the projected image. We have experimentally verified each of our methods using real scenes.

Grasping and Tracking Using Constant Curvature Dynamic Contours

In this paper we present our constant curvature dynamic contours (snakes) and three applications of these: visual servoing and grasping, occluding contour depth extraction, and localization of miniature mobile robots. For the first application, a novel deformable contour model is implemented for the automatic determination of plausible grasp axes of unknown objects using an eye-in-hand robotic system. The system finds potential grasp point pairs, ranks them based upon measurements taken from the contour, and executes a vision-guided grasp using the highest ranked grasp point pair to determine the gripper alignment. Our method is based upon statistical active deformable models. We have developed a new snake model that is applicable to real-time vision problems. The grasping method is experimentally verified using both simple and complex unknown grasping targets. These experiments demonstrate the effectiveness of using the proposed snakes to grasp previously unknown objects in minimally structured environments. We also present a novel method for active monocular depth recovery (second application of our snakes). It combines new, highly stable active deformable models with a structured camera motion along the optical axis to produce depth estimates for all snake control points. The method has a simple formulation and is suitable for real-time, vision-based robotic applications. Experiments with a variety of objects and depths demonstrate the practicality of the method. Finally, we present a novel method for localizing miniature mobile robots (Scouts) using dynamic contours. The miniature robot is tracked as it moves and jumps in the environment. The proposed dynamic contours are very effective in tracking the fast accelerations and decelerations of this small robot. We show initial experimental results emphasizing the task of monitoring a Scout's jumps.

Computation of a cross-section structure: a projection-based approach

While most of the existing depth recovery methods compute 2 1 2 -D depth information from 2-D intensity images, the method described in this paper recovers the cross-section of a scene by processing the line images of a line scan camera. This method has some advantages as compared to the existing stereo methods. Line images, which are combined to make a slit image, are taken by moving the line scan camera along a linear rail. The depth of a pixel is directly related to the slope of a straight line in the slit image. Thus one of the advantages is that correspondence becomes the problem of detecting straight lines in the slit image. Another advantage is that we can obtain very high resolution images if we use a line scan camera, since it is not difficult to find CCD sensors with 4 k or more linear arrays in the market. Hough transforms are used to find straight lines in the slit image. Instead of the number of points on a line, each cell in Hough space records the intensity variation of the pixels on a line. We derive the mathematical background of this method and show some experimental results with several real images. This scheme has a important potential for industrial and scientific application capability with greater refinement.

From NOMAD to explorer: active object recognition on mobile robots

This paper demonstrates the importance of active behavior in machine vision systems. It proposes using activity of the observer for object finding and recognition. The presented robotic system is based on active and qualitative vision approaches which are supplemented by a hierarchical, multiresolution image representation and a parallel implementation of the active vision algorithms. The system uses a hierarchical search strategy to locate and then identify objects in a three-dimensional space. The paper describes the recognition algorithm and its parallel implementation on the SFU hybrid pyramid vision machine.We also introduce a new method for depth recovery—Stereo from Head and Body Movements (SHABM). As shown in our analysis and experiment, it provides a practical and economic means of 3-D sensing on ordinary (wheeled) mobile robots.