Coarse Depth Map Research Articles

An app for tree trunk diameter estimation from coarse optical depth maps

Trunk diameter is related to the overall health and level of carbon sequestration in a tree. Trunk diameter measurement, therefore, is a key task in both forest plot and urban settings. Unlike the traditional approach of manual measurement with a measuring tape or calipers, several recent approaches rely on sophisticated technologies such as LiDAR and time-of-flight cameras that provide fine-grain depth maps, which are used for depth-assisted image segmentation in downstream processing. These technologies are supported only on specialized devices or high-end smartphones. We present a mobile application that uses coarse-grain depth maps derived from an optical sensor, and so can be run on most common Android devices. Moreover, we use a state-of-the-art deep neural network to estimate trunk diameter from an image and its corresponding coarse depth map (RGB-D). We tested our app using a data set collected from four countries and under challenging conditions including occlusion, leaning trees, and irregular shapes and found that our algorithm has a MAE of 1.66 cm and an RMSE of 2.46 cm, which is comparable to accuracy from fine-grain depth maps. Moreover, diameter measurement using our app is >5 times faster than traditional manual surveying.

High-precision polarization imaging for three-dimensional reconstruction aided by a separate coarse depth map

For less-texture objects with highly reflective regions, traditional vision-based three-dimensional reconstruction techniques often fail to yield ideal results. Utilizing polarization information for reconstructing such objects is a convenient and effective method. However, relying solely on polarization information for three-dimensional reconstruction presents challenges such as ambiguity in surface normals and difficulties in normal integration. In this paper, we propose to resolve the ambiguity in polarization normals using the coarse depth as prior information under a perspective projection model. By fusing the disambiguated normal with the coarse depth, we avoid the need for normal integration. The experimental results demonstrate that our proposed algorithm improves the quality of polarization imaging, effectively restoring the details lost in the coarse depth and smoothing the areas with high reflectivity.

CSRNet: Focusing on critical points for depth completion

Depth completion is an effective method for generating dense depth maps from sparse ones. In recent studies, the majority of points, which we call standard points, often exhibit sub-optimal performance. This issue arises from the need to fit only very few points, termed as challenging points, which consist of noises and regions with discontinuous depth in the ground-truth. On the other hand, traditional evaluations can not recognize this situation, and are dominated by these limited challenging points, whose performance improvements may not significantly benefit related tasks. In contrast, standard points, which are critical for these tasks, are not effectively measured. This discrepancy highlights the need for a more targeted approach and evaluation method for depth completion. In order to solve the above problems, we propose a standard-point-enhancing learning paradigm. This paradigm aims to improve the performance on standard points, which consists of a Cascaded Segmentation-to-Regression Networks (CSRNet) and a Mining L1 loss. CSRNet includes two branches: DSNet and DRNet. DSNet uses segmentation to generate a coarse depth map, providing challenging-point-insensitive information. DRNet adopts a coarse-to-fine approach to learn the residual depth map between the coarse depth map and the ground-truth depth map. In addition, our Mining L1 loss leverages the segmentation results to filter out potential challenging points. This approach allows the network to concentrate more effectively on standard points. Lastly, we introduce the Minimum Error (ME) Curves as a new way to measure the performance of predicted depth maps in a flexible and comprehensive manner, irrespective of whether the points are standard or challenging. Experimental results on the KITTI and NYUDv2 datasets show that our approach significantly improves accuracy on the majority of points.

ARAI-MVSNet: A multi-view stereo depth estimation network with adaptive depth range and depth interval

Multi-View Stereo (MVS) is a fundamental problem in geometric computer vision which aims to reconstruct a scene using multi-view images with known camera parameters. However, the mainstream approaches represent the scene with a fixed all-pixel depth range and equal depth interval partition, which will result in inadequate utilization of depth planes and imprecise depth estimation. In this paper, we present a novel multi-stage coarse-to-fine framework to achieve adaptive all-pixel depth range and depth interval. We predict a coarse depth map in the first stage, then an Adaptive Depth Range Prediction module is proposed in the second stage to zoom in the scene by leveraging the reference image and the obtained depth map in the first stage and predict a more accurate all-pixel depth range for the following stages. In the third and fourth stages, we propose an Adaptive Depth Interval Adjustment module to achieve adaptive variable interval partition for pixel-wise depth range. The depth interval distribution in this module is normalized by Z-score, which can allocate dense depth hypothesis planes around the potential ground truth depth value and vice versa to achieve more accurate depth estimation. Extensive experiments on four widely used benchmark datasets (DTU, TnT, BlendedMVS, ETH 3D) demonstrate that our model achieves state-of-the-art performance and yields competitive generalization ability. Particularly, our method achieves the highest Acc and Overall on the DTU dataset, while attaining the highest Recall and F1-score on the Tanks and Temples intermediate and advanced dataset. Moreover, our method also achieves the lowest e1 and e3 on the BlendedMVS dataset and the highest Acc and F1-score on the ETH 3D dataset, surpassing all listed methods. Project website: https://github.com/zs670980918/ARAI-MVSNet

RGB-Guided Depth Map Recovery by Two-Stage Coarse-to-fine Dense CRF Models.

Depth maps generally suffer from large erroneous areas even in public RGB-Depth datasets. Existing learning-based depth recovery methods are limited by insufficient high-quality datasets and optimization-based methods generally depend on local contexts not to effectively correct large erroneous areas. This paper develops an RGB-guided depth map recovery method based on the fully connected conditional random field (dense CRF) model to jointly utilize local and global contexts of depth maps and RGB images. A high-quality depth map is inferred by maximizing its probability conditioned upon a low-quality depth map and a reference RGB image based on the dense CRF model. The optimization function is composed of redesigned unary and pairwise components, which constraint local structure and global structure of depth map, respectively, with the guidance of RGB image. In addition, the texture-copy artifacts problem is handled by two-stage dense CRF models in a coarse-to-fine way. A coarse depth map is first recovered by embedding RGB image in a dense CRF model in unit of 3×3 blocks. It is refined afterward by embedding RGB image in another model in unit of individual pixels and restricting the model mainly work in discontinued regions. Extensive experiments on six datasets verify that the proposed method considerably outperforms a dozen of baseline methods in correcting erroneous areas and diminishing texture-copy artifacts of depth maps.

Implicit neural refinement based multi-view stereo network with adaptive correlation

In this paper, we propose ACINR-MVSNet, an end-to-end trainable framework with adaptive group-wise correlation and implicit neural depth refinement for multi-view stereo (MVS). Previous learning-based MVS methods have demonstrated their outstanding performance, and most of them estimate depth maps in a coarse-to-fine manner. However, in a commonly used multi-stage cascaded framework, the previous wrong estimation might lead to error propagation. In contrast, we focus on another coarse-to-fine structure, i.e., one-stage MVS architecture followed by refinement modules. Inspired by implicit neural representation, we propose an implicit neural refinement module to refine the coarse depth map. Guided by the corresponding reference image, it can better recover finer details, especially those in boundary areas. To solve the visibility problem in complex scenarios while maintaining efficiency, we propose an adaptive group-wise correlation similarity measure for cost volume construction. Besides, we present a pyramid-based feature extraction network with a repeated top-down and bottom-up structure to gather more context-aware information, which can better meet the challenges in ill-posed regions. This novel feature extractor is also utilized to construct an enhanced Gauss-Newton refinement module for further upsampling and optimizing. Extensive experiments on the DTU, the Tanks & Temples and the BlendedMVS datasets demonstrate the effectiveness and generalization of our approach, which can achieve better or competitive results compared to state-of-the-art methods. The code will be available at https://github.com/BoyangSONG/ACINR-MVSNet .

ArthroNet: a monocular depth estimation technique with 3D segmented maps for knee arthroscopy

BackgroundLack of depth perception from medical imaging systems is one of the long-standing technological limitations of minimally invasive surgeries. The ability to visualize anatomical structures in 3D can improve conventional arthroscopic surgeries, as a full 3D semantic representation of the surgical site can directly improve surgeons’ ability. It also brings the possibility of intraoperative image registration with preoperative clinical records for the development of semi-autonomous, and fully autonomous platforms. This study aimed to present a novel monocular depth prediction model to infer depth maps from a single-color arthroscopic video frame. MethodsWe applied a novel technique that provides the ability to combine both supervised and self-supervised loss terms and thus eliminate the drawback of each technique. It enabled the estimation of edge-preserving depth maps from a single untextured arthroscopic frame. The proposed image acquisition technique projected artificial textures on the surface to improve the quality of disparity maps from stereo images. Moreover, following the integration of the attention-ware multi-scale feature extraction technique along with scene global contextual constraints and multiscale depth fusion, the model could predict reliable and accurate tissue depth of the surgical sites that complies with scene geometry. ResultsA total of 4,128 stereo frames from a knee phantom were used to train a network, and during the pre-trained stage, the network learned disparity maps from the stereo images. The fine-tuned training phase uses 12,695 knee arthroscopic stereo frames from cadaver experiments along with their corresponding coarse disparity maps obtained from the stereo matching technique. In a supervised fashion, the network learns the left image to the disparity map transformation process, whereas the self-supervised loss term refines the coarse depth map by minimizing reprojection, gradients, and structural dissimilarity loss. Together, our method produces high-quality 3D maps with minimum re-projection loss that are 0.0004132 (structural similarity index), 0.00036120156 (L1 error distance) and 6.591908 × 10−5 (L1 gradient error distance). ConclusionMachine learning techniques for monocular depth prediction is studied to infer accurate depth maps from a single-color arthroscopic video frame. Moreover, the study integrates segmentation model hence, 3D segmented maps are inferred that provides extended perception ability and tissue awareness.

Recovery of geometry, natural illumination, and BRDF from a single light field image.

We propose an inverse rendering model for light fields to recover surface normals, depth, reflectance, and natural illumination. Our setting is fully uncalibrated, with the reflectance modeled with a spatially constant Blinn-Phong bidirectional reflectance distribution function (BRDF) and illumination as an environment map. While previous work makes strong assumptions in this difficult scenario, focusing solely on specific types of objects such as faces or imposing very strong priors, our approach leverages only the light field structure, where a solution consistent across all subaperture views is sought. The optimization is based primarily on shading, which is sensitive to fine geometric details that are propagated to the initial coarse depth map. Despite the problem being inherently ill posed, we achieve encouraging results on synthetic as well as real-world data.

FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion

Depth completion aims to recover a dense depth map from a sparse depth map with the corresponding color image as input. Recent approaches mainly formulate the depth completion as a one-stage end-to-end learning task, which outputs dense depth maps directly. However, the feature extraction and supervision in one-stage frameworks are insufficient, limiting the performance of these approaches. To address this problem, we propose a novel end-to-end residual learning framework, which formulates the depth completion as a two-stage learning task, i.e., a sparse-to-coarse stage and a coarse-to-fine stage. First, a coarse dense depth map is obtained by a simple CNN framework. Then, a refined depth map is further obtained using a residual learning strategy in the coarse-to-fine stage with coarse depth map and color image as input. Specially, in the coarse-to-fine stage, a channel shuffle extraction operation is utilized to extract more representative features from color image and coarse depth map, and an energy based fusion operation is exploited to effectively fuse these features obtained by channel shuffle operation, thus leading to more accurate and refined depth maps. We achieve SoTA performance in RMSE on KITTI benchmark. Extensive experiments on other datasets future demonstrate the superiority of our approach over current state-of-the-art depth completion approaches.

Self-Supervised Learning of Monocular Depth Estimation Based on Progressive Strategy

Monocular depth estimation has been carried out with convolutional neural networks (CNNs). However, vast quantities of ground truth depth data are required as a supervising signal. Recently, self-supervised learning is explored for monocular depth estimation, which uses the minor loss of image reconstruction, rather than depth information, as the supervising signal in the training phase. However, the blurring problem often occurs on the surface of a near object when using the image reconstruction as a major loss function. Here, we propose a novel Progressive Strategy Network (PSNet) to overcome this problem, which optimizes the depth map from coarse to fine with several progressive modules. A progressive module estimates a coarse depth map with a color image by CNNs and outputs it into the next module for progressively accurate refinement. It can simplify the difficulty of estimating the depth map with a large size and reduce the blurring problem. Experiments demonstrate that the proposed approach consistently improves the quality of depth map on the surface of large objects and keeps edges clear. Our proposed method performs high-quality depth estimation on the KITTI dataset and achieves a strong generalization on the Make3D dataset. Besides, it can be flexibly used on the real scenes captured by mobile phones without extra training.

Visibility-Aware Point-Based Multi-View Stereo Network.

We introduce VA-Point-MVSNet, a novel visibility-aware point-based deep framework for multi-view stereo (MVS). Distinct from existing cost volume approaches, our method directly processes the target scene as point clouds. More specifically, our method predicts the depth in a coarse-to-fine manner. We first generate a coarse depth map, convert it into a point cloud and refine the point cloud iteratively by estimating the residual between the depth of the current iteration and that of the ground truth. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point. This point-based architecture allows higher accuracy, more computational efficiency and more flexibility than cost-volume-based counterparts. Furthermore, our visibility-aware multi-view feature aggregation allows the network to aggregate multi-view appearance cues while taking into account visibility. Experimental results show that our approach achieves a significant improvement in reconstruction quality compared with state-of-the-art methods on the DTU and the Tanks and Temples dataset. The code of VA-Point-MVSNet proposed in this work will be released at https://github.com/callmeray/PointMVSNet.

Guided, Fusion-Based, Large Depth-of-field 3D Imaging Using a Focal Stack.

Three dimensional (3D) imaging technology has been widely used for many applications, such as human–computer interactions, making industrial measurements, and dealing with cultural relics. However, existing active methods often require both large apertures of projector and camera to maximize light throughput, resulting in a shallow working volume in which projector and camera are simultaneously in focus. In this paper, we propose a novel method to extend the working range of the structured light 3D imaging system based on the focal stack. Specifically in the case of large depth variation scenes, we first adopted the gray code method for local, 3D shape measurement with multiple focal distance settings. Then we extracted the texture map of each focus position into a focal stack to generate a global coarse depth map. Under the guidance of the global coarse depth map, the high-quality 3D shape measurement of the overall scene was obtained by local, 3D shape-measurement fusion. To validate the method, we developed a prototype system that can perform high-quality measurements in the depth range of 400 mm with a measurement error of 0.08%.

Depth acquisition with the combination of structured light and deep learning stereo matching

The methods of structured light and deep learning are widely used in artificial vision to acquire a depth map of real-world scenes. In this paper, we propose a novel method of combining structured light and deep learning stereo matching to calculate the depth. To combat the problems with textureless areas of stereo matching, a pair of left and right side images with abundant structured light information is adopted to acquire a coarse depth map by unsupervised CNN networks. The coarse depth map is used for phase unwrapping, which is a bottleneck in the phase-based structured light method. Then, a fine and accurate depth map is obtained by phase matching. Compared with the traditional structured light method, the proposed method performs better on occluded and textureless areas. To evaluate the performance of our proposed method, an experimental platform is established and several experiments are conducted. Quantitative and qualitative experiments demonstrate that the proposed method can generate a high precision depth and relieve the occlusion in the structured light system.

A self-calibrated photo-geometric depth camera

Compared with geometric stereo vision based on triangulation principle, photometric stereo method has advantages in recovering per-pixel surface details. In this paper, we present a practical 3D imaging system by combining the near-light photometric stereo and the speckle-based stereo matching method. The system is compact in structure and suitable for multi-albedo targets. The parameters (including position and intensity) of the light sources can be self-calibrated. To realize the auto-calibration, we first use the distant lighting model to estimate the initial surface albedo map, and then with the estimated albedo map and the normal vector field fixed, the parameters of the near lighting model are optimized. Next, with the optimized lighting model, we use the near-light photometric stereo method to re-compute the surface normal and fuse it with the coarse depth map from stereo vision to achieve high-quality depth map. Experimental results show that our system can realize high-quality reconstruction in general indoor environments.

Depth Sensing Using Geometrically Constrained Polarization Normals

Analyzing the polarimetric properties of reflected light is a potential source of shape information. However, it is well-known that polarimetric information contains fundamental shape ambiguities, leading to an underconstrained problem of recovering 3D geometry. To address this problem, we use additional geometric information, from coarse depth maps, to constrain the shape information from polarization cues. Our main contribution is a framework that combines surface normals from polarization (hereafter polarization normals) with an aligned depth map. The additional geometric constraints are used to mitigate physics-based artifacts, such as azimuthal ambiguity, refractive distortion and fronto-parallel signal degradation. We believe our work may have practical implications for optical engineering, demonstrating a new option for state-of-the-art 3D reconstruction.

Depth Map Reconstruction for Underwater Kinect Camera Using Inpainting and Local Image Mode Filtering

Underwater optical cameras are widely used for security monitoring in ocean, such as earthquake prediction and tsunami alarming. Optical cameras recognize objects for autonomous underwater vehicles and provide security protection for sea-floor networks. However, there are many issues for underwater optical imaging, such as forward and backward scattering, light absorption, and sea snow. Many underwater image processing techniques have been proposed to overcome these issues. Among these techniques, the depth map gives important information for many applications of the post-processing. In this paper, we propose a Kinect-based underwater depth map estimation method that uses a captured coarse depth map by Kinect with the loss of depth information. To overcome the drawbacks of low accuracy of coarse depth maps, we propose a corresponding reconstruction architecture that uses the underwater dual channels prior dehazing model, weighted enhanced image mode filtering, and inpainting. Our proposed method considers the influence of mud sediments in water and performs better than the traditional methods. The experimental results demonstrated that, after inpainting, dehazing, and interpolation, our proposed method can create high-accuracy depth maps.

Merging Static and Dynamic Depth Cues with Optical-Flow Recovery for Creating Stereo Videos

A method for estimating the depth information of a general monocular image sequence and then creating a 3D stereo video is proposed. Distinguishing between foreground and background is possible without additional information, and then foreground pixels are moved to create the binocular image. The proposed depth estimation method is based on coarse-to-fine strategy. By applying the CID method in the spatial domain, the sharpness and the contrast of an image can be improved by the distance of the region based on its color. Then a coarse depth map of the image can be generated. An optical-flow method based on temporal information is then used to search and compare the block motion status between previous and current frames, and then the distance of the block can be estimated according to the amount of block motion. Finally, the static and motion depth information is integrated to create the fine depth map. By shifting foreground pixels based on the depth information, a binocular image pair can be created. A sense of 3D stereo can be obtained without glasses by an autostereoscopic 3D display.

Photorealistic rendering of rain streaks

Photorealistic rendering of rain streaks with lighting and viewpoint effects is a challenging problem. Raindrops undergo rapid shape distortions as they fall, a phenomenon referred to as oscillations. Due to these oscillations, the reflection of light by, and the refraction of light through, a falling raindrop produce complex brightness patterns within a single motion-blurred rain streak captured by a camera or observed by a human. The brightness pattern of a rain streak typically includes speckles, multiple smeared highlights and curved brightness contours. In this work, we propose a new model for rain streak appearance that captures the complex interactions between the lighting direction, the viewing direction and the oscillating shape of the drop. Our model builds upon a raindrop oscillation model that has been developed in atmospheric sciences. We have measured rain streak appearances under a wide range of lighting and viewing conditions and empirically determined the oscillation parameters that are dominant in raindrops. Using these parameters, we have rendered thousands of rain streaks to create a database that captures the variations in streak appearance with respect to lighting and viewing directions. We have developed an efficient image-based rendering algorithm that uses our streak database to add rain to a single image or a captured video with moving objects and sources. The rendering algorithm is very simple to use as it only requires a coarse depth map of the scene and the locations and properties of the light sources. We have rendered rain in a wide range of scenarios and the results show that our physically-based rain streak model greatly enhances the visual realism of rendered rain.

Robust ego-motion estimation and 3-D model refinement using surface parallax

We present an iterative algorithm for robustly estimating the ego-motion and refining and updating a coarse depth map using parametric surface parallax models and brightness derivatives extracted from an image pair. Given a coarse depth map acquired by a range-finder or extracted from a digital elevation map (DEM), ego-motion is estimated by combining a global ego-motion constraint and a local brightness constancy constraint. Using the estimated camera motion and the available depth estimate, motion of the three-dimensional (3-D) points is compensated. We utilize the fact that the resulting surface parallax field is an epipolar field, and knowing its direction from the previous motion estimates, estimate its magnitude and use it to refine the depth map estimate. The parallax magnitude is estimated using a constant parallax model (CPM) which assumes a smooth parallax field and a depth based parallax model (DBPM), which models the parallax magnitude using the given depth map. We obtain confidence measures for determining the accuracy of the estimated depth values which are used to remove regions with potentially incorrect depth estimates for robustly estimating ego-motion in subsequent iterations. Experimental results using both synthetic and real data (both indoor and outdoor sequences) illustrate the effectiveness of the proposed algorithm.

Integration of shape from shading and stereo

Stereo algorithms suffer from the lack of local surface texture due to smoothness of depth constraint, or local miss-matches in disparity estimates. Thus, most stereo methods only provide a coarse depth map which can be associated with a low pass image of the depth map. On the other hand, shape from shading algorithms generally produce better estimates of local surface areas, but some of them have problems with variable albedo and spherical surfaces. Thus, shape from shading methods produce better detailed depth information, and can be associated with the high pass image of the depth map. In order to compute a better depth map, we present a method for integrating the high frequncy information from the shape from shading and the low frequency information from stereo. The proposed algorithm is very simple, takes about 0.7 s for a 128 × 128 image on a Sun SparcStation-1, is non-iterative, and requires very little adjustment of parameters. The results obtained with a variety of synthetic and real images are discussed. The quality of depth obtained by integrating shading and stereo is compared with the ground truth (range image) using height error measure, and improvement ranging from 30 to 50% over stereo, and from 65 to 98% over shading is demonstrated.