Depth Map Estimation Research Articles

Low Power Depth Estimation of Rigid Objects for Time-of-Flight Imaging

Depth sensing is useful in a variety of applications that range from augmented reality to robotics. Time-of-flight (TOF) cameras are appealing because they obtain dense depth measurements with minimal latency. However, for many battery-powered devices, the illumination source of a TOF camera is power hungry and can limit the battery life of the device. To address this issue, we present an algorithm that lowers the power for depth sensing by reducing the usage of the TOF camera and estimating depth maps using concurrently collected images. Our technique also adaptively controls the TOF camera and enables it when an accurate depth map cannot be estimated. To ensure that the overall system power for depth sensing is reduced, we design our algorithm to run on a low power embedded platform, where it outputs $640\times 480$ depth maps at 30 frames per second. We evaluate our approach on several RGB-D datasets, where it produces depth maps with an overall mean relative error of 0.96% and reduces the usage of the TOF camera by 85%. When used with commercial TOF cameras, we estimate that our algorithm can lower the total power for depth sensing by up to 73%.

Depth estimation using modified cost function for occlusion handling

This paper presents a novel approach to the occlusion handling problem in depth estimation using three views. A solution based on modification of similarity cost function is proposed. During the depth estimation via optimization algorithms, like Graph Cuts, the similarity metric is constantly updated so that only non-occluded fragments in the side views are considered. At each iteration of the algorithm, non-occluded fragments are detected based on side view virtual depth maps synthesized from currently the best estimated depth map of the center view. Then, the similarity metric is updated for correspondence search only in non-occluded regions of the side views. The experimental results, conducted on well-known 3D video test sequences, show that the depth maps estimated with the proposed approach provide about 1.25 dB virtual view quality improvement in comparison with the virtual view synthesized based on depth maps generated with the use of the state-of-the-art MPEG Depth Estimation Reference Software.

Depth Map Estimation Using Defocus and Motion Cues

Significant recent developments in 3D display technology have focused on techniques for converting 2D media into 3D. Depth map is an integral part of 2D-to-3D conversion. Combining multiple depth cues results in a more accurate depth map as it compensates for the errors caused by one depth cue as well as its absence by other depth cues. In this paper, we present a novel framework to generate a more accurate depth map for video using defocus and motion cues. The moving objects present in the scene are the source of errors in both defocus and motion-based depth map estimation. The proposed method rectifies these errors in the depth map by integrating defocus blur and motion cues. In addition, it also corrects the errors in other parts of depth map caused by inaccurate estimation of defocus blur and motion. Since the proposed integration approach relies on the characteristics of point spread functions of defocus and motion blur along with their relations to camera parameters, it is more accurate and reliable.

Alternating minimization algorithm for hybrid regularized variational image dehazing

Imaging quality is often significantly degraded under hazy weather condition. The purpose of this paper is to recover the latent sharp image from its hazy version. It is well known that the accurate estimation of depth information could assist in improving dehazing performance. In this paper, a hybrid regularized variational framework was proposed to simultaneously estimate depth map and haze-free image. In particular, the second-order total generalized variation (TGV) regularizer was introduced to constrain the estimation of depth map, which commonly contains the piecewise smooth regions separated by sharp edges. To take full advantage of the similar properties between three different color channels, we proposed to exploit the multichannel total variation (MTV) regularizer to guarantee the robust restoration of haze-free image. The resulting nonsmooth optimization problem was effectively handled using the alternating direction method of multipliers (ADMM)-based numerical optimization algorithm. A two-step correction mechanism was further introduced to improve visual image quality in the presence of large sky region. Numerous experiments have been implemented based on both synthetic and realistic hazy images. Dehazing results demonstrated that our method was competitive with or even outperformed current state-of-the-art methods under different imaging conditions.

Face Depth Estimation With Conditional Generative Adversarial Networks

Depth map estimation and 3-D reconstruction from a single or a few face images is an important research field in computer vision. Many approaches have been proposed and developed over the last decade. However, issues like robustness are still to be resolved through additional research. With the advent of the GPU computational methods, convolutional neural networks are being applied to many computer vision problems. Later, conditional generative adversarial networks (CGAN) have attracted attention for its easy adaptation for many picture-to-picture problems. CGANs have been applied for a wide variety of tasks, such as background masking, segmentation, medical image processing, and superresolution. In this work, we developed a GAN-based method for depth map estimation from any given single face image. Many variants of GANs have been tested for the depth estimation task for this work. We conclude that conditional Wasserstein GAN structure offers the most robust approach. We have also compared the method with other two state-of-the-art methods based on deep learning and traditional approaches and experimentally shown that the proposed method offers great opportunities for estimation of face depth maps from face images.

Improving Monocular Depth Prediction in Ambiguous Scenes Using a Single Range Measurement

Depth maps are widely used in robotics, with numerous applications in agricultural tasks. Methods for estimating these from monocular images currently exist, but this is an ill posed problem which requires assumptions about object scale and camera focal length. These assumptions may not always be reasonable and how to deal with them has not been sufficiently explored in the current literature. For example, scenes in agriculture frequently violate the assumption of having a single scale per object class and represent a failure case for these methods. To avoid these assumptions when estimating depth maps, we present an approach where a single actual distance measurement is fused with a monocular image. Our results indicate that this method can outperform an image-only baseline, provided the distance measurement is sampled according to a projective model. We also found that a single measurement can significantly improve accuracy on simulated variable scale versions of two common public datasets. A hardware implementation of this approach was tested in an agricultural setting, though results were poor. Software and hardware designs are made available1.

Refining the Fusion of Pepper Robot and Estimated Depth Maps Method for Improved 3D Perception

As it is well known, some versions of the Pepper robot provide poor depth perception due to the lenses it has in front of the tridimensional sensor. In this paper, we present a method to improving that faulty 3D perception. Our proposal is based on a combination of the actual depth readings of Pepper and a deep learning-based monocular depth estimation. As shown, the combination of both of them provides a better 3D representation of the scene. In previous works we made an initial approximation of this fusion technique, but it had some drawbacks. In this paper we analyze the pros and cons of the Pepper readings, the monocular depth estimation method and our previous fusion method. Finally, we demonstrate that the proposed fusion method outperforms them all.

Inertial sensor aided multi-image nonuniform motion blur removal based on motion decomposition

Many factors lead to spatially varying blur kernels in a blurred image, such as camera shake, moving objects, and scene depth variation. The traditional camera shake removal methods either ignore the influence of varying depth values or object motion in dynamic scenes, while the methods not limited to removing camera shake always make simple assumptions about camera motion trajectory. We consider these factors in a unified framework, with the aid of an alternate-exposure capture strategy and simultaneously recorded inertial sensor readings. The inertial measurements relate the long-exposed blurred image to preceding and succeeding short-exposed noisy images. The special exposure arrangement effectively addresses the problem inherent in reconstructing camera motion from inertial measurements. In addition, the noisy image pair bracketing the blurred image is used for motion detection and initial depth map estimation, making the proposed method free of user interaction and additional expensive devices. Contrary to previous methods that individually parametrize the motion blur of the moving foreground layer and the static background layer, we exploit the fact that camera shake has a global influence to decompose the motion of the foreground layer such that a more tight constraint between the motion of layers is established. Given the motion and image data, we propose a single-energy model and minimize it using alternating optimization to estimate the spatially varying motion blur and the latent sharp image. Comparative experimental results demonstrate that our method outperforms conventional camera motion deblurring and object deblurring methods on both synthetic and real scenes.

Multifocus image fusion with enhanced linear spectral clustering and fast depth map estimation

Abstract One novel regional method is proposed in this paper to get more natural multifocus image fusion results. At first, the edge defocus degree propagation approach is combined with the linear-time guided filter to generate the depth information of all pixels efficiently. Then the fast linear spectral clustering (LSC) is enhanced by the depth information to produce superpixels as regions. Due to the embedding of depth information in the clustering algorithm, the resulting regions are more consistent with the distributions of pixels in source images with different focus levels. The flexibility in selecting the number of superpixels also offers the enhanced LSC a better chance to get more homogeneous segmentation, which is of great importance in deriving better fusion results. Finally a post-processing step based on multi-guided filtering and morphological operations is suggested to polish the decision map of fusion and further improve the fusion results. Experiments demonstrate that the proposed method can achieve state-of-the-art performance for multifocus image fusion problems.

Automatic Depth Prediction from 2D Videos Based on Non-parametric Learning and Bi-directional Depth Propagation

With progress in 3D video technology, 2D to 3D conversion has drawn great attention in recent years. Predicting perceptually reasonable depth information from traditional monocular videos is a challenging task in 2D to 3D conversion. For the purpose of generating convincing depth maps, an efficient depth estimation algorithm utilizing non-parametric learning and bi-directional depth propagation is proposed. First, global depth maps on key frames are generated on the basis of gradient samples and the gradient reconstruction method. Then, foreground objects are extracted and employed to refine global depth maps for producing more local depth details. Next, the depth information of key frames is propagated utilizing bi-directional motion compensation to recover the forward and backward depth information of non-key frames. In the end, a weighting fusion strategy is designed to integrate forward and backward depths for predicting the depth information of each non-key frame. The quality of estimated depth maps is assessed by leveraging objective and subjective quality evaluation criteria. The experimental results verify that the proposed depth prediction framework outperforms some depth estimation methods and is efficient at producing 3D views.

Segmentation-based Method of Increasing The Depth Maps Temporal Consistency

In this paper, a modification of the graph-based depth estimation is presented. The purpose of proposed modification is to increase the quality of estimated depth maps, reduce the time of the estimation, and increase the temporal consistency of depth maps. The modification is based on the image segmentation using superpixels, therefore in the first step of the proposed modification a segmentation of previous frames is used in the currently processed frame in order to reduce the overall time of the depth estimation. In the next step, a depth map from the previous frame is used in the depth map optimization as the initial values of a depth map estimated for the current frame. It results in the better representation of silhouettes of objects in depth maps and in the reduced computational complexity of the depth estimation process. In order to evaluate the performance of the proposed modification the authors performed the experiment for a set of multiview test sequences that varied in their content and an arrangement of cameras. The results of the experiments confirmed the increase of the depth maps quality — the quality of depth maps calculated with the proposed modification is higher than for the unmodified depth estimation method, apart from the number of the performed optimization cycles. Therefore, use of the proposed modification allows to estimate a depth of the better quality with almost 40% reduction of the estimation time. Moreover, the temporal consistency, measured through the reduction of the bitrate of encoded virtual views, was also considerably increased.

Automatic Depth Extraction from 2D Images Using a Cluster-Based Learning Framework.

There has been a significant increase in the availability of 3D players and displays in the last years. Nonetheless, the amount of 3D content has not experimented an increment of such magnitude. To alleviate this problem, many algorithms for converting images and videos from 2D to 3D have been proposed. Here, we present an automatic learning-based 2D-3D image conversion approach, based on the key hypothesis that color images with similar structure likely present a similar depth structure. The presented algorithm estimates the depth of a color query image using the prior knowledge provided by a repository of color + depth images. The algorithm clusters this database attending to their structural similarity, and then creates a representative of each color-depth image cluster that will be used as prior depth map. The selection of the appropriate prior depth map corresponding to one given color query image is accomplished by comparing the structural similarity in the color domain between the query image and the database. The comparison is based on a K-Nearest Neighbor framework that uses a learning procedure to build an adaptive combination of image feature descriptors. The best correspondences determine the cluster, and in turn the associated prior depth map. Finally, this prior estimation is enhanced through a segmentation-guided filtering that obtains the final depth map estimation. This approach has been tested using two publicly available databases, and compared with several state-of-the-art algorithms in order to prove its efficiency.

Total variation-based dense depth from multicamera array

Multi-Camera arrays are increasingly employed in both consumer and industrial applications, and various passive techniques are documented to estimate depth from such camera arrays. Current depth estimation methods provide useful estimations of depth in an imaged scene but are often impractical due to significant computational requirements. This paper presents a novel framework that generates a high-quality continuous depth map from multi-camera array/light field cameras. The proposed framework utilizes analysis of the local Epipolar Plane Image (EPI) to initiate the depth estimation process. The estimated depth map is then processed using Total Variation (TV) minimization based on the Fenchel-Rockafellar duality. Evaluation of this method based on a well-known benchmark indicates that the proposed framework performs well in terms of accuracy when compared to the top-ranked depth estimation methods and a baseline algorithm. The test dataset includes both photorealistic and non-photorealistic scenes. Notably, the computational requirements required to achieve an equivalent accuracy are significantly reduced when compared to the top algorithms. As a consequence, the proposed framework is suitable for deployment in consumer and industrial applications.

Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy

Colorectal cancer is the fourth leading cause of cancer deaths worldwide and the second leading cause in the United States. The risk of colorectal cancer can be mitigated by the identification and removal of premalignant lesions through optical colonoscopy. Unfortunately, conventional colonoscopy misses more than 20% of the polyps that should be removed, due in part to poor contrast of lesion topography. Imaging depth and tissue topography during a colonoscopy is difficult because of the size constraints of the endoscope and the deforming mucosa. Most existing methods make unrealistic assumptions which limits accuracy and sensitivity. In this paper, we present a method that avoids these restrictions, using a joint deep convolutional neural network-conditional random field (CNN-CRF) framework for monocular endoscopy depth estimation. Estimated depth is used to reconstruct the topography of the surface of the colon from a single image. We train the unary and pairwise potential functions of a CRF in a CNN on synthetic data, generated by developing an endoscope camera model and rendering over 200,000 images of an anatomically-realistic colon.We validate our approach with real endoscopy images from a porcine colon, transferred to a synthetic-like domain via adversarial training, with ground truth from registered computed tomography measurements. The CNN-CRF approach estimates depths with a relative error of 0.152 for synthetic endoscopy images and 0.242 for real endoscopy images. We show that the estimated depth maps can be used for reconstructing the topography of the mucosa from conventional colonoscopy images. This approach can easily be integrated into existing endoscopy systems and provides a foundation for improving computer-aided detection algorithms for detection, segmentation and classification of lesions.

Automated outdoor depth-map generation and alignment

Image enhancement tasks can highly benefit from depth information, but the direct estimation of outdoor depth maps is difficult due to vast object distances. This paper presents a fully automatic framework for model-based generation of outdoor depth maps and its applications to image enhancements. We leverage 3D terrain models and camera pose estimation techniques to render approximate depth maps without resorting to manual alignment. Potential local misalignments, resulting from insufficient model details and rough registrations, are eliminated with our novel free-form warping. We first align synthetic depth edges with photo edges using the as-rigid-as-possible image registration and further refine the shape of the edges using the tight trimap-based alpha matting. The resulting synthetic depth maps are accurate, calibrated in the absolute distance. We demonstrate their benefit in image enhancement techniques including reblurring, depth-of-field simulation, haze removal, and guided texture synthesis.

Depth Map Estimation based on Maximum a Posteriori Probability

In this paper, we consider depth map estimation expressed as an optimization problem. We focus on the fitness function, for which we present a theoretical derivation based on a maximum a posteriori probability (MAP) rule. This is then used to show relations of interest with commonly used similarity metrics: sum of absolute differences (SAD) and sum of squared differences (SSD). The original derivations are also used to propose a depth estimation method. The experimental results are obtained with an implementation made on the basis of Moving Picture Expert Group (MPEG) Depth Estimation Reference Software (DERS). We show that with the proposed approach, it is possible to estimate a depth that allows higher quality of view synthesis (up to 2.8 dB of PSNR – Peak Signal-to-Noise Ratio) versus the original unsupervised DERS, when sub-optimal control parameters are used. If the DERS control parameters are optimized manually, the attained gain is smaller (up to 0.08 dB PSNR) but still does not need manual selection of control parameters.

Analyzing the impact of the initialization for the nonnegative tensor factorization in tensor displays

Tensor display is an option in glasses-free three-dimensional (3-D) display technology. An initial solution has to be set to decompose the light-field information to be represented by the system. We have analyzed the impact of the initial guess on the multiplicative update rules in terms of peak signal-to-noise ratio, and proposed a method based on depth map estimation from an input light field. Results from simulations were obtained and compared with previous literature. In our sample, the initial values used have a large influence on results and convergence to a local minimum. The quality of the output stabilizes after a certain number of iterations, suggesting that a limit on such numbers should be imposed. We show that the proposed methods outperform the pre-existing ones.

Quantitative Performance Evaluation for Dehazing Algorithms on Synthetic Outdoor Hazy Images

Dehazing has emerged as a promising technology to recover the clear image from an input hazy image, such that the image quality can be significantly enhanced. However, it is still an open issue to quantitatively evaluate the performance of existing dehazing algorithms. To fill in this gap, we propose an effective performance evaluation method for image dehazing by exploiting synthetic outdoor hazy images dataset. To be specific, we first synthesize hazes on the original haze-free images to approximate the real-world hazy scenes. A dataset that contains original images, synthetic hazy images, estimated depth maps, and transmission maps of the same scene is then established. Due to the fact that the generation of the synthetic hazy images is based on the physical model which is strongly related to the depth information, an effective depth estimation method that combines the geometry and edge information is proposed. With the estimated depth map, we are able to create a corresponding hazy scene with high fidelity. Numerous experiments are conducted to verify the consistency of subjective and objective evaluation of the proposed method.

The Synthesis of Unpaired Underwater Images Using a Multistyle Generative Adversarial Network

Underwater image datasets are crucial in underwater vision research. Because of the strong absorption and scattering effects that occur underwater, some ground truth such as the depth map, which can be easily collected in-air, becomes a great challenge in underwater environments. To solve the issues associated with the lack of underwater ground truth, we propose a trainable end-to-end system of an underwater multistyle generative adversarial network (UMGAN) that takes advantage of a cycle-consistent adversarial network (CycleGAN) and conditional generative adversarial networks. This system can generate multiple realistic underwater images from in-air images using a hybrid adversarial system and an unpaired method. Moreover, our model can translate in-air images to underwater images that retain the main content and structural information of the in-air images under specified turbidities or water styles through a style classifier and a conditional vector. Furthermore, we define the color loss and include the structural similarity index measure loss for the system to preserve the content and structure of original in-air images while transferring the backgrounds of the images from air to water. Using UMGAN, we can take advantage of the in-air ground truth and convert the corresponding in-air images into an underwater dataset with multiple water color styles. Our experiments demonstrate that our synthesized underwater images have a high score on image assessment against CycleGAN, WaterGAN, StarGAN, AdaIN, and other state-of-the-art methods. We also show that our synthesized underwater images with in-air depths can be applied to real underwater image depth map estimation.

Efficient Light Field Images Compression Method Based on Depth Estimation and Optimization

Massive recent investigations from both industry and academia have been poured into autostereoscopic display. One of the emerging techniques, light field image (LFI), can provide more immersive perception by increasing the number of views and the spatial resolution. However, these advantages restrict the storage and transmission due to such dense-view image simultaneously. To solve this problem, we propose to compress the LFI using multiview video plus depth (MVD) coding architecture. In this paper, we preliminarily estimate the depth based on the concept of epipolar plane image. To achieve a depth value, we design an optimal slope decision algorithm to determine the best slope with the minimal cost. While this rough estimation produces some error points within initial depth map, therefore, we present a depth optimization algorithm using the characteristic of the associated texture image. Ultimately, a small number of selected viewpoint images are encoded with their corresponding depth maps using the MVD framework, and then, the unselected viewpoint images are synthesized by depth image-based rendering technique. To verify the validity of the proposed LFI compression scheme, extensive experiments are conducted. The simulated results demonstrate that the proposed depth map estimation algorithm is superior to other state-of-the-art methods for the LFI. Meanwhile, our LFI compression method outperforms other LFI compression algorithms significantly.