Single Image Depth Estimation Research Articles

Consistent video depth estimation

We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Unlike the ad-hoc priors in classical reconstruction, we use a learning-based prior, i.e., a convolutional neural network trained for single-image depth estimation. At test time, we fine-tune this network to satisfy the geometric constraints of a particular input video, while retaining its ability to synthesize plausible depth details in parts of the video that are less constrained. We show through quantitative validation that our method achieves higher accuracy and a higher degree of geometric consistency than previous monocular reconstruction methods. Visually, our results appear more stable. Our algorithm is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion. The improved quality of the reconstruction enables several applications, such as scene reconstruction and advanced video-based visual effects.

DEEP LEARNING FOR MONOCULAR DEPTH ESTIMATION FROM UAV IMAGES

Abstract. Depth is an essential component for various scene understanding tasks and for reconstructing the 3D geometry of the scene. Estimating depth from stereo images requires multiple views of the same scene to be captured which is often not possible when exploring new environments with a UAV. To overcome this monocular depth estimation has been a topic of interest with the recent advancements in computer vision and deep learning techniques. This research has been widely focused on indoor scenes or outdoor scenes captured at ground level. Single image depth estimation from aerial images has been limited due to additional complexities arising from increased camera distance, wider area coverage with lots of occlusions. A new aerial image dataset is prepared specifically for this purpose combining Unmanned Aerial Vehicles (UAV) images covering different regions, features and point of views. The single image depth estimation is based on image reconstruction techniques which uses stereo images for learning to estimate depth from single images. Among the various available models for ground-level single image depth estimation, two models, 1) a Convolutional Neural Network (CNN) and 2) a Generative Adversarial model (GAN) are used to learn depth from aerial images from UAVs. These models generate pixel-wise disparity images which could be converted into depth information. The generated disparity maps from these models are evaluated for its internal quality using various error metrics. The results show higher disparity ranges with smoother images generated by CNN model and sharper images with lesser disparity range generated by GAN model. The produced disparity images are converted to depth information and compared with point clouds obtained using Pix4D. It is found that the CNN model performs better than GAN and produces depth similar to that of Pix4D. This comparison helps in streamlining the efforts to produce depth from a single aerial image.

A self-supervised method of single-image depth estimation by feeding forward information using max-pooling layers

We propose an encoder–decoder CNN framework to predict depth from one single image in a self-supervised manner. To this aim, we design three kinds of encoder based on the recent advanced deep neural network and one kind of decoder which can generate multiscale predictions. Eight loss functions are designed based on the proposed encoder–decoder CNN framework to validate the performance. For training, we take rectified stereo image pairs as input of the CNN, which is trained by reconstructing image via learning multiscale disparity maps. For testing, the CNN can estimate the accurate depth information by inputting only one single image. We validate our framework on two public datasets in contrast to the state-of-the-art methods and our designed different variants, and the performance of different encoder–decoder architectures and loss functions is evaluated to obtain the best combination, which proves that our proposed method performs very well for single-image depth estimation without the supervision of ground truth.

S2DNet: Depth Estimation From Single Image and Sparse Samples

Depth prediction from single image is a challenging task due to the intra scale ambiguity and unavailability of prior information. The prediction of an unambiguous depth from single RGB image is very important aspect for computer vision applications. In this paper, an end-to-end sparse-to-dense network (S2DNet) is proposed for single image depth estimation (SIDE). The proposed network processes single image along with the additional sparse depth samples for depth estimation. The additional sparse depth sample are acquired either with a low-resolution depth sensor or calculated by visual simultaneous localization and mapping (SLAM) algorithms. In first stage, the proposed S2DNet estimates coarse-level depth map using sparse-to-dense coarse network (S2DCNet). In second stage, the estimated coarse-level depth map is concatenated with the input image and used as an input to the sparse-to-dense fine network (S2DFNet) for fine-level depth map estimation. The proposed S2DFNet comprises of attention map architecture which helps to estimate the prominent depth information. The quantitative and qualitative performance evaluation of the proposed network has been carried out using the error metrics. We perform complete evaluation of S2DNet on four publically available benchmark data sets i.e. NYU Depth-V2 indoor dataset [1] , KITTI odometry outdoor dataset [2] , KITTI depth completion test database [3] and SUN-RGB database [4] . Further, we have extended the proposed S2DNet for image de-hazing. The experimental analysis shows that the proposed S2DNet outperforms the existing state-of-the-art methods for both single image depth estimation and image de-hazing.

Comparison of monocular depth estimation methods using geometrically relevant metrics on the IBims-1 dataset

The task of predicting a dense depth map from a monocular RGB image, commonly known as single-image depth estimation (SIDE) or monocular depth estimation (MDE), is an active research topic in computer vision for decades. With the significant progress of deep models in recent years, new standards were set yielding remarkable results in capturing the 3D structure from a single image. However, established evaluation schemes of predicted depth maps are still limited, as they only consider global statistics of the depth residuals. In order to allow for a geometry-aware analysis, we propose a set of novel quality criteria addressing the preservation of depth discontinuities and planar regions, the depth consistency across the image, and a distance-related assessment. As current datasets do not fulfill the requirements of all proposed error metrics, we provide a new high-quality indoor RGB-D test dataset, acquired by a digital single-lens reflex (DSLR) camera together with a laser scanner. New insights into the performance of current state-of-the-art SIDE approaches, as well as subtle differences among them, could be unveiled by employing the proposed error metrics on our reference dataset. Additionally, investigations on the real-world applicability of SIDE methods by a series of experiments regarding different image augmentations, illumination changes and textured planar regions have shown current limitations in this research field.

MONOCULAR-DEPTH ASSISTED SEMI-GLOBAL MATCHING

Abstract. Reconstruction of dense photogrammetric point clouds is often based on depth estimation of rectified image pairs by means of pixel-wise matching. The main drawback lies in the high computational complexity compared to that of the relatively straightforward task of laser triangulation. Dense image matching needs oriented and rectified images and looks for point correspondences between them. The search for these correspondences is based on two assumptions: pixels and their local neighborhood show a similar radiometry and image scenes are mostly homogeneous, meaning that neighboring points in one image are most likely also neighbors in the second. These rules are violated, however, at depth changes in the scene. Optimization strategies tend to find the best depth estimation based on the resulting disparities in the two images. One new field in neural networks is the estimation of a depth image from a single input image through learning geometric relations in images. These networks are able to find homogeneous areas as well as depth changes, but result in a much lower geometric accuracy of the estimated depth compared to dense matching strategies. In this paper, a method is proposed extending the Semi-Global-Matching algorithm by utilizing a-priori knowledge from a monocular depth estimating neural network to improve the point correspondence search by predicting the disparity range from the single-image depth estimation (SIDE). The method also saves resources through path optimization and parallelization. The algorithm is benchmarked on Middlebury data and results are presented both quantitatively and qualitatively.

Aggregation of Rich Depth-Aware Features in a Modified Stacked Generalization Model for Single Image Depth Estimation

Estimating scene depth from a single monocular image is a crucial component in computer vision tasks, enabling many further applications such as robot vision, 3-D modeling, and above all, 2-D to 3-D image/video conversion. Since there are an infinite number of possible world scenes, that can produce a unique image, single image depth estimation is a highly challenging task. This paper tackles such an ambiguous problem by using the merits of both global and local information (structures) of a scene. To this end, we formulate single image depth estimation as a regression problem via (on) rich depth related features which describe effective monocular cues. Exploiting the relationship between these image features and depth values is adopted via a learning model which is inspired by modified stacked generalization scheme. The experiments demonstrate competitive results compared with existing data-driven approaches in both quantitative and qualitative analysis with a remarkably simpler approach than previous works.

Depth‐based end‐to‐end deep network for human action recognition

Recognition of human actions from videos can be improved if depth information is available. Depth information certainly helps in segregating foreground motion from the background. Single image depth estimation (SIDE) is a commonly used method for the analysis of weather degraded images. In this study, the idea of SIDE is extended to human action recognition (HAR) on datasets where depth information is not available. Several depth‐based HAR algorithms are available but all of them are using the depth information given with the dataset. Some other methods are using depth motion map which refers to the depth of motion in a temporal direction. Here, a new depth‐based end‐to‐end deep network is proposed for HAR in which the frame‐wise depth is estimated and this estimated depth is used for processing instead of RGB frame. As colour information is not required for estimating motion, a single channel depth map is used for estimating motion in the video. It makes the system computationally efficient. The proposed method is tested and verified on three benchmark datasets namely JHMDB, HMDB51 and UCF101. The proposed method outperforms the existing state‐of‐the‐art methods for HAR on all the three tested datasets.

Deep Monocular Depth Estimation via Integration of Global and Local Predictions.

Recent works on machine learning have greatly advanced the accuracy of single image depth estimation. However, the resulting depth images are still over-smoothed and perceptually unsatisfying. This paper casts depth prediction from single image as a parametric learning problem. Specifically, we propose a deep variational model that effectively integrates heterogeneous predictions from two convolutional neural networks (CNNs), named global and local networks. They have contrasting network architecture and are designed to capture depth information with complementary attributes. These intermediate outputs are then combined in the integration network based on the variational framework. By unrolling the optimization steps of Split Bregman (SB) iterations in the integration network, our model can be trained in an end-to-end manner. This enables one to simultaneously learn an efficient parameterization of the CNNs and hyper-parameter in the variational method. Finally, we offer a new dataset of 0.22 million RGB-D images captured by Microsoft Kinect v2. Our model generates realistic and discontinuity-preserving depth prediction without involving any low-level segmentation or superpixels. Intensive experiments demonstrate the superiority of the proposed method in a range of RGB-D benchmarks including both indoor and outdoor scenarios.

Single image depth estimation based on convolutional neural network and sparse connected conditional random field

Deep convolutional neural networks (DCNNs) have attracted significant interest in the computer vision community in the recent years and have exhibited high performance in resolving many computer vision problems, such as image classification. We address the pixel-level depth prediction from a single image by combining DCNN and sparse connected conditional random field (CRF). Owing to the invariance properties of DCNNs that make them suitable for high-level tasks, their outputs are generally not localized enough for detailed pixel-level regression. A multiscale DCNN and sparse connected CRF are combined to overcome this localization weakness. We have evaluated our framework using the well-known NYU V2 depth dataset, and the results show that the proposed method can improve the depth prediction accuracy both qualitatively and quantitatively, as compared to previous works. This finding shows the potential use of the proposed method in three-dimensional (3-D) modeling or 3-D video production from the given two-dimensional (2-D) images or 2-D videos.

GPU-Accelerated Single Image Depth Estimation with Color-Filtered Aperture

There are two major ways to implement depth estimation, multiple image depth estimation and single image depth estimation, respectively. The former has a high hardware cost because it uses multiple cameras but it has a simple software algorithm. Conversely, the latter has a low hardware cost but the software algorithm is complex. One of the recent trends in this field is to make a system compact, or even portable, and to simplify the optical elements to be attached to the conventional camera. In this paper, we present an implementation of depth estimation with a single image using a graphics processing unit (GPU) in a desktop PC, and achieve real-time application via our evolutional algorithm and parallel processing technique, employing a compute shader. The methods greatly accelerate the compute-intensive implementation of depth estimation with a single view image from 0.003 frames per second (fps) (implemented in MATLAB) to 53 fps, which is almost twice the real-time standard of 30 fps. In the previous literature, to the best of our knowledge, no paper discusses the optimization of depth estimation using a single image, and the frame rate of our final result is better than that of previous studies using multiple images, whose frame rate is about 20fps.