Resolution Depth Map Research Articles

228 × 304 200-mW lidar based on a single-point global-depth d-ToF sensor and RGB-guided super-resolution neural network.

The cutting-edge imaging system exhibits low output resolution and high power consumption, presenting challenges for the RGB-D fusion algorithm. In practical scenarios, aligning the depth map resolution with the RGB image sensor is a crucial requirement. In this Letter, the software and hardware co-design is considered to implement a lidar system based on the monocular RGB 3D imaging algorithm. A 6.4 × 6.4-mm2 deep-learning accelerator (DLA) system-on-chip (SoC) manufactured in a 40-nm CMOS is incorporated with a 3.6-mm2 TX-RX integrated chip fabricated in a 180-nm CMOS to employ the customized single-pixel imaging neural network. In comparison to the RGB-only monocular depth estimation technique, the root mean square error is reduced from 0.48 m to 0.3 m on the evaluated dataset, and the output depth map resolution matches the RGB input.

Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

Unsupervised learning methods have achieved remarkable performance in monocular depth estimation and camera pose, which mostly solve the multi-task learning problem by using their inner geometry consistency as the self-supervision signal. While most existing approaches mostly adopt the generative model to obtain the depth map prediction, so in the resolution of depth map there is room for improvement. To this end, we present our unsupervised learning architecture based on adversarial learning model, which is used for unsupervised learning of high-resolution single view depth and camera pose. Specifically, we present a multi-scale deep convolutional Generative Adversarial Network (GAN) based learning system, which consists of three networks (pose estimation network PCNN, Generator-D and Discriminator-D for depth map prediction). Furthermore, in order to generate high-resolution depth map, we propose a multi-scale GAN model (MSGAN) to decompose the hard high-quality image generation problem into more manageable sub-problems through a coarse-to-fine process. Then, we modify the overall generation architecture of GAN model by changing the down-sampling and up-sampling components to improve the quality and accuracy of the depth map prediction. Finally, in order to improve the rate of convergence, we use the Least Square Error to increase the penalty for outliers. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI dataset show that the proposed method provides better results for both pose estimation and depth recovery.

CORNet: Context-Based Ordinal Regression Network for Monocular Depth Estimation

Monocular depth estimation, as one of the fundamental tasks of computer vision, plays a crucial role in three-dimensional (3D) scene understanding and perception. Usually, deep learning methods recover monocular depth maps using continuous regression manners by minimizing the errors between the ground-truth depth and the predicted depth. However, fine depth features may not be fully captured through layer-by-layer coding, which is prone to low spatial resolution depth maps and insufficient details. Furthermore, it usually converges slowly and suffers from unsatisfactory results. To tackle these issues, we propose a novel model, named context-based ordinal regression network (CORNet), to reconstruct monocular depth maps in the ordinal regression manner with context information in this paper. Firstly, we put forward a novel context-based encoder with a feature transformation (FT) module to learn context information and details from inputs, and output multi-scale feature maps. Then, we design a boundary enhancement module (BEM) with a spatial attention mechanism following each operation of feature fusion, which captures boundary features in the scene to enhance the border depth. Finally, a feature optimization module (FOM) is designed to fuse and optimize the multi-scale features and boundary features to strengthen depth learning. What’s more, we introduce an ordinal weighted inference to predict depth maps from probabilities and discretization values. Experiments and results on two challenging datasets, KITTI and NYU Depth V2, demonstrate that our proposed CORNet can estimate monocular depth maps effectively and obtain superior performance in capturing geometric features over existing methods.

Unsupervised multi-view stereo network based on multi-stage depth estimation

In current years, supervised learning multi-view stereo (MVS) methods have achieved impressive performance. However, these methods still suffer the limitation of hard to acquire large-scale depth supervision data, which hinders the generalization ability in never-seen-before scenarios. Recently, some unsupervised-learning methods have been proposed, which relieved the requirement of depth supervision data. However, the generated depth map with lower resolution since the memory consumption grows cubically. In this paper, we propose a novel unsupervised multi-view stereo network based on multi-stage depth estimation, which can increase depth map resolution and generate a dense 3D model with rich details without relying on depth supervision data. To reduce the 3D cost volume highly memory consumption, the progressive coarse-to-fine multiple stages are adopted. Besides, a multi-view group-wise correlation (MV-GwC) module is designed to introduce multi-view correlation prior, which can enhance network performance and further reduce memory consumption. Qualitative and quantitative experiment results show the effectiveness of our method. We outperform some previous supervised and unsupervised MVS methods on DTU and Tanks & Temples benchmarks.

A Real-Time Online Learning Framework for Joint 3D Reconstruction and Semantic Segmentation of Indoor Scenes

This letter presents a real-time online vision framework to jointly recover an indoor scene’s 3D structure and semantic label. Given noisy depth maps, a camera trajectory, and 2D semantic labels at train time, the proposed deep neural network based approach learns to fuse the depth over frames with suitable semantic labels in the scene space. Our approach exploits the joint volumetric representation of the depth and semantics in the scene feature space to solve this task. For a compelling online fusion of the semantic labels and geometry in real-time, we introduce an efficient vortex pooling block while dropping the use of routing network in online depth fusion to preserve high-frequency surface details. We show that the context information provided by the semantics of the scene helps the depth fusion network learn noise-resistant features. Not only that, it helps overcome the shortcomings of the current online depth fusion method in dealing with thin object structures, thickening artifacts, and false surfaces. Experimental evaluation on the Replica dataset shows that our approach can perform depth fusion at 37 and 10 frames per second with an average reconstruction F-score of 88% and 91%, respectively, depending on the depth map resolution. Moreover, our model shows an average IoU score of 0.515 on the ScanNet 3D semantic benchmark leaderboard. Code and example dataset information is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/suryanshkumar/online-joint-depthfusion-and-semantic</uri> .

Monocular Real Time Full Resolution Depth Estimation Arrangement with a Tunable Lens

This work introduces a real-time full-resolution depth estimation device, which allows integral displays to be fed with a real-time light-field. The core principle of the technique is a high-speed focal stack acquisition method combined with an efficient implementation of the depth estimation algorithm, allowing the generation of real time, high resolution depth maps. As the procedure does not depend on any custom hardware, if the requirements are met, the described method can turn any high speed camera into a 3D camera with true depth output. The concept was tested with an experimental setup consisting of an electronically variable focus lens, a high-speed camera, and a GPU for processing, plus a control board for lens and image sensor synchronization. The comparison with other state of the art algorithms shows our advantages in computational time and precision.

Depth Completion and Super-Resolution with Arbitrary Scale Factors for Indoor Scenes.

Depth sensing has improved rapidly in recent years, which allows for structural information to be utilized in various applications, such as virtual reality, scene and object recognition, view synthesis, and 3D reconstruction. Due to the limitations of the current generation of depth sensors, the resolution of depth maps is often still much lower than the resolution of color images. This hinders applications, such as view synthesis or 3D reconstruction, from providing high-quality results. Therefore, super-resolution, which allows for the upscaling of depth maps while still retaining sharpness, has recently drawn much attention in the deep learning community. However, state-of-the-art deep learning methods are typically designed and trained to handle a fixed set of integer-scale factors. Moreover, the raw depth map collected by the depth sensor usually has many depth data missing or misestimated values along the edges and corners of observed objects. In this work, we propose a novel deep learning network for both depth completion and depth super-resolution with arbitrary scale factors. The experimental results on the Middlebury stereo, NYUv2, and Matterport3D datasets demonstrate that the proposed method can outperform state-of-the-art methods.

3D reconstruction with auto-selected keyframes based on depth completion correction and pose fusion

Dense 3D reconstruction is required for robots to safely navigate or perform advanced tasks. The accurate depth information of the image and its pose are the basis of 3D reconstruction. The resolution of depth maps obtained by LIDAR and RGB-D cameras is limited, and traditional pose calculation methods are not accurate enough. In addition, if each image is used for dense 3D reconstruction, the dense point clouds will increase the amount of calculation. To address these issues, we propose a 3D reconstruction system. Specifically, we propose a depth network of contour and gradient attention, which is used to complete and correct depth maps to obtain high-resolution and high-quality depth maps. Then, we propose a method of fusion of traditional algorithms and deep learning for pose estimation to obtain accurate localization results. Finally, we adopt the method of autonomous selection of keyframes to reduce the number of keyframes, the surfel-based geometric reconstruction is performed to reconstruct the dense 3D environment. On the TUM RGB-D, ICL-NIUM, and KITTI datasets, our method significantly improves the quality of the depth maps, the localization results, and the effect of 3D reconstruction. At the same time, we have also accelerated the speed of 3D reconstruction.

Attention aware cost volume pyramid based multi-view stereo network for 3D reconstruction

We present an efficient multi-view stereo (MVS) network for 3D reconstruction from multi-view images. While previous learning based reconstruction approaches performed quite well, most of them estimate depth maps at a fixed resolution using plane sweep volumes with a fixed depth hypothesis at each plane, which requires densely sampled planes for desired accuracy and therefore is difficult to achieve high resolution depth maps. In this paper we introduce a coarse-to-fine depth inference strategy to achieve high resolution depth. This strategy first estimates the depth map at coarsest level, and the depth maps at finer levels are considered as the upsampled depth map from previous level with pixel-wise depth residual. Thus, we narrow the depth searching range with the priori information from previous level and construct new cost volumes from the pixel-wise depth residual to perform depth map refinement. Then the final depth map can be achieved iteratively since all the parameters are shared among different levels. At each level, the self-attention layer is introduced to the feature extraction block for capturing the important information in depth inference task, and the cost volume is generated using similarity measurement instead of the variance based methods used in previous work. Experiments were conducted on three diverse datasets including the DTU benchmark dataset, BlendedMVS dataset and the Tanks and Temples dataset. The results demonstrated that our proposed approach could outperform most state-of-the-arts (SOTA) methods.22The codebase of this project is at https://github.com/ArthasMil/AACVP-MVSNet.

Deep edge map guided depth super resolution

Accurate edge reconstruction is critical for depth map super resolution (SR). Therefore, many traditional SR methods utilize edge maps to guide depth SR. However, it is difficult to predict accurate edge maps from low resolution (LR) depth maps. In this paper, we propose a deep edge map guided depth SR method, which includes an edge prediction subnetwork and an SR subnetwork. The edge prediction subnetwork takes advantage of the hierarchical representation of color and depth images to produce accurate edge maps, which promote the performance of SR subnetwork. The SR subnetwork is a disentangling cascaded network to progressively upsample SR result, where every level is made up of a weight sharing module and an adaptive module. The weight sharing module extracts the general features in different levels, while the adaptive module transfers the general features to the specific features to adapt to different degraded inputs. Quantitative and qualitative evaluations on various datasets with different magnification factors demonstrate the effectiveness and promising performance of the proposed method. In addition, we construct a benchmark dataset captured by Kinect-v2 to facilitate research on real-world depth map SR.

A Real-Time Infrared Stereo Matching Algorithm for RGB-D Cameras’ Indoor 3D Perception

Low-cost, commercial RGB-D cameras have become one of the main sensors for indoor scene 3D perception and robot navigation and localization. In these studies, the Intel RealSense R200 sensor (R200) is popular among many researchers, but its integrated commercial stereo matching algorithm has a small detection range, short measurement distance and low depth map resolution, which severely restrict its usage scenarios and service life. For these problems, on the basis of the existing research, a novel infrared stereo matching algorithm that combines the idea of the semi-global method and sliding window is proposed in this paper. First, the R200 is calibrated. Then, through Gaussian filtering, the mutual information and correlation between the left and right stereo infrared images are enhanced. According to mutual information, the dynamic threshold selection in matching is realized, so the adaptability to different scenes is improved. Meanwhile, the robustness of the algorithm is improved by the Sobel operators in the cost calculation of the energy function. In addition, the accuracy and quality of disparity values are improved through a uniqueness test and sub-pixel interpolation. Finally, the BundleFusion algorithm is used to reconstruct indoor 3D surface models in different scenarios, which proved the effectiveness and superiority of the stereo matching algorithm proposed in this paper.

Windowed Bundle Adjustment Framework for Unsupervised Learning of Monocular Depth Estimation With U-Net Extension and Clip Loss

This letter presents a self-supervised framework for learning depth from monocular videos. In particular, the main contributions of this letter include: (1) We present a windowed bundle adjustment framework to train the network. Compared to most previous works that only consider constraints from consecutive frames, our framework increases the camera baseline and introduces more constraints to avoid overfitting. (2) We extend the widely used U-Net architecture by applying a Spatial Pyramid Net (SPN) and a Super Resolution Net (SRN). The SPN fuses information from an image spatial pyramid for the depth estimation, which addresses the context information attenuation problem of the original U-Net. The SRN learns to estimate a high resolution depth map from a low resolution image, which can benefit the recovery of details. (3) We adopt a clip loss function to handle moving objects and occlusions that were solved by designing complicated network or requiring extra information (such as segmentation mask [1] ) in previous works. Experimental results show that our algorithm provides state-of-the-art results on the KITTI benchmark.

Co-axial depth sensor with an extended depth range for AR/VR applications

BackgroundDepth sensor is an essential element in virtual and augmented reality devices to digitalize users' environment in real time. The current popular technologies include the stereo, structured light, and Time-of-Flight (ToF). The stereo and structured light method require a baseline separation between multiple sensors for depth sensing, and both suffer from a limited measurement range. The ToF depth sensors have the largest depth range but the lowest depth map resolution. To overcome these problems, we propose a co-axial depth map sensor which is potentially more compact and cost-effective than conventional structured light depth cameras. Meanwhile, it can extend the depth range while maintaining a high depth map resolution. Also, it provides a high-resolution 2D image along with the 3D depth map. MethodsThis depth sensor is constructed with a projection path and an imaging path. Those two paths are combined by a beamsplitter for a co-axial design. In the projection path, a cylindrical lens is inserted to add extra power in one direction which creates an astigmatic pattern. For depth measurement, the astigmatic pattern is projected onto the test scene, and then the depth information can be calculated from the contrast change of the reflected pattern image in two orthogonal directions. To extend the depth measurement range, we use an electronically focus tunable lens at the system stop and tune the power to implement an extended depth range without compromising depth resolution. ResultsIn the depth measurement simulation, we project a resolution target onto a white screen which is moving along the optical axis and then tune the focus tunable lens power for three depth measurement subranges, namely, near, middle and far. In each sub-range, as the test screen moves away from the depth sensor, the horizontal contrast keeps increasing while the vertical contrast keeps decreasing in the reflected image. Therefore, the depth information can be obtained by computing the contrast ratio between features in orthogonal directions. ConclusionsThe proposed depth map sensor could implement depth measurement for an extended depth range with a co-axial design.

Depth map super-resolution with cascaded pyramid structure

Due to the limitation of equipment, the resolution of depth map is low. Depth edges often become blurred when the low-resolution depth image is upsampled. In this paper, we present the pyramid dense residual network (PDRN) to efficiently reconstruct the high-resolution images. The network takes residual network as the main frame and adopts the cascaded pyramid structure for phased upsampling. At each pyramid level, the modified dense block is used to acquire high frequency residual, especially the edge features and the skip connection branch in the residual structure is used to deal with the low frequency information. The network directly uses the low-resolution depth image as the initial input of the network and the subpixel convolution layers is used for upsampling. It reduces the computational complexity. The experiments indicate that the proposed method effectively solves the problem of blurred edge and obtains great results both in qualitative and quantitative.

Multi-Direction Dictionary Learning Based Depth Map Super-Resolution With Autoregressive Modeling

3D depth cameras have become more and more popular in recent years. However, depth maps captured by these cameras can hardly be used in 3D reconstruction directly because they often suffer from low resolution and blurring depth discontinuities. Super resolution of depth maps is necessary. In depth maps, the edge areas play more important role and demonstrate distinct geometry directions compared with natural images. However, most existing super-resolution methods ignore this fact, and they can not handle depth edges properly. Motivated by this, we propose a compound method that combines multi-direction dictionary sparse representation and autoregressive (AR) models, so that the depth edges are presented precisely at different levels. In the patch level, the depth edge patches with geometry directions are well represented by the pre-trained multi-directional dictionaries. Compared with a universal dictionary, multiple dictionaries trained from different directional patches can represent the directional depth patch much better. In the finer pixel level, we utilize an adaptive AR model to represent the local correlation patterns in small areas. Extensive experimental results on both synthetic and real datasets demonstrate that, the proposed model outperforms state-of-the-art depth map super-resolution methods in terms of both quantitative metrics and subjective visual quality.

Lasers structured light with phase-shifting for dense depth perception

In this paper, a novel depth acquisition algorithm aiming to obtain high resolution of dense depth maps based on lasers is proposed. Which expands the projection range and improves the robustness. In addition, dithering projection and defocusing techniques are used by lasers with the improvement of the Diffraction Optical Elements (DOE) technology. Four patterns are projected onto the projects by the lasers and captured by camera, where three for three-step phase-shifting (TS-PS) patterns and the other one is for space stripe pattern. More important, the calibrated by the traditional methods are unavailable in the proposed devices. We propose an algorithm for obtaining intrinsic and extrinsic parameters of the system. A substantially different spatial stripe coding scheme is designed, which is utilized to assist in time coding (TS-PS) solving depth information. We proposed a novel phase unwrapping method to obtain accurate absolute phase map. In addition, to further boost the performance, we utilized Weighted Least Squares (WLS) strategy to optimize the detail of the depth map based on color image. Thereby high resolution, detail and dense depth map obtained. Compared with the current methods, the robustness of our method is improved which will be extended to more scenes. The quantitative and qualitative evaluation experiments demonstrate that the proposed algorithm consistently produces better quality dense and higher resolution depth maps, superior to the current depth sensor.

A Maximum Likelihood Approach for Depth Field Estimation Based on Epipolar Plane Images.

In this paper, a multi-resolution method for depth estimation from dense image arrays is presented. Recent progress in consumer electronics has enabled the development of low cost hand-held plenoptic cameras. In these systems, multiple views of a scene are captured in a single shot by means of a micro-lens array placed on the focal point of the first camera lens, in front of the imaging sensor. These views can be processed jointly to obtain accurate depth maps. In this contribution, to reduce the computational complexity associated to global optimization schemes based on match cost functions, we make a local estimate based on the maximization of the total log-likelihood spatial density aggregated along the epipolar lines corresponding to each view pair. This method includes the local maximum likelihood estimation of the depth field based on epipolar plane images. To face the potential accuracy losses associated to the ambiguity problem that arises in flat surface regions while preserving bandwidth in correspondence of the edges, we adopt a multi-resolution scheme. In practice, the depth map resolution is reduced in regions where maximizing the higher resolution functional is ill-conditioned. The main benefits of the proposed system are in a reduced computational complexity and a high accuracy of the estimated depth. Experimental results show that the proposed scheme represents a good tradeoff among accuracy, robustness, and discontinuities handling.

A Novel Segmentation Based Depth Map Up-Sampling

A novel color image segmentation-based depth map upsampling method is proposed in this paper. In this method, the color image is segmented into a certain number of connected regions first. Based on the segmentation result, the target pixels will be interpolated by the seed pixels <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">11</sup> The seed pixels are directly from the low resolution depth maps, i.e., the ones that have depth values. The targets are those without depth and to be interpolated. regionally. In the segmentation part, simple linear iterative clustering is introduced to generate superpixels in the first place. Then, the obtained superpixels will be judged whether they are correct-clustered or not, and the incorrect-clustered ones will be subdivided with an adaptive region-growing strategy. Third, the regions that have no seed will be constantly merged into their nearest neighbors, until seed pixel can be found in each independent region. Finally, adjacent regions that have quite small depth gaps will be united as one. The proposed color image segmentation strictly follows the guidance of the depth; therefore, the segmented regions adhere to the depth boundary well. In the interpolation part, the targets will be interpolated with their surrounding seeds weighted by a joint trilateral filter (JTF). The JTF is constructed by three terms: the color term, the distance term, and the region term, which are driven by the previous segmentation result. Experimental results indicate that our method greatly reduces depth bleeding and depth confusion artifacts, and leads to clear depth boundary in the up-sampled image. Comparisons with the state of art verify the advantages of the proposed method in both visual experience and quantitative evaluations.

Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution.

Rapid development of affordable and portable consumer depth cameras facilitates the use of depth information in many computer vision tasks such as intelligent vehicles and 3D reconstruction. However, depth map captured by low-cost depth sensors (e.g., Kinect) usually suffers from low spatial resolution, which limits its potential applications. In this paper, we propose a novel deep network for depth map super-resolution (SR), called DepthSR-Net. The proposed DepthSR-Net automatically infers a high resolution (HR) depth map from its low resolution (LR) version by hierarchical features driven residual learning. Specifically, DepthSR-Net is built on a residual U-Net deep network architecture. Given LR depth map, we first obtain the desired HR by bicubic interpolation upsampling, and then construct an input pyramid to achieve multiple level receptive fields. Next, we extract hierarchical features from the input pyramid, intensity image, and encoder-decoder structure of UNet. Finally, we learn the residual between the interpolated depth map and the corresponding HR one using the rich hierarchical features. The final HR depth map is achieved by adding the learned residual to the interpolated depth map. We conduct an ablation study to demonstrate the effectiveness of each component in the proposed network. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art methods. Additionally, the potential usage of the proposed network in other low-level vision problems is discussed.

High‐order Markov random field for single depth image super‐resolution

Although there is an increasing interest in employing the depth data in computer vision applications, the spatial resolution of depth maps is still limited compared with typical visible‐light images. A novel method is proposed to synthetically improve the spatial resolution of a single depth image. It integrates the higher‐order terms into the Markov random field (MRF) formulation of example‐based methods in order to improve the representational power of those methods. The inference is performed by approximately minimising the higher‐order multi‐label MRF energies. In addition, to improve the efficiency of the inference algorithm, a hierarchical scheme on the number of MRF states is proposed. First, a large number of states are used to obtain an initial labelling by solving the minimisation problem of inference for only the first‐order energies. Then, the problem is solved for the higher‐order energies in a smaller number of states. Performance comparisons show that proposed method improves the results of first‐order approaches that are based on simple four‐connected MRF graph structure, both qualitatively and quantitatively.