High-resolution Depth Map Research Articles

Neural single-shot GHz FMCW correlation imaging

Depth sensing is essential for 3D environmental perception across application domains, including autonomous driving, topographical mapping, and augmented and virtual reality (AR/VR). Traditional correlation time-of-flight (ToF) methods, while are able to produce dense high-resolution depth maps, are plagued by phase wrapping artifacts which limit their effective depth range. Though multi-frequency methods can help reduce this problem by simultaneously solving for phase wrap counts in multiple wavelengths, this requires multiple measurements per pixel, necessitating additional hardware and imaging time. We introduce a 3D imaging method that requires a single per-point measurement by combining frequency-modulated continuous wave (FMCW) operation, all-optical correlation ToF imaging, and a specialized frequency-decoding network. Our system performs all-optical correlation imaging at GHz rates. The method is validated through both simulations and real-world experiments, comparing favorably to existing methods in all experiments.

Digging into Depth and Color Spaces: A Mapping Constraint Network for Depth Super-Resolution

Scene depth super-resolution (DSR) poses an inherently ill-posed problem due to the extremely large space of one-to-many mapping functions from a given low-resolution (LR) depth map, which possesses limited depth information, to multiple plausible high-resolution (HR) depth maps. This characteristic renders the task highly challenging, as identifying an optimal solution becomes significantly intricate amidst this multitude of potential mappings. While simplistic constraints have been proposed to address the DSR task, the relationship between LR and HR depth maps and the color image has not been thoroughly investigated. In this paper, we introduce a novel mapping constraint network (MCNet) that incorporates additional constraints derived from both LR depth maps and color images. This integration aims to optimize the space of mapping functions and enhance the performance of DSR. Specifically, alongside the primary DSR network (DSRNet) dedicated to learning LR-to-HR mapping, we have developed an auxiliary degradation network (ADNet) that operates in reverse, generating the LR depth map from the reconstructed HR depth map to obtain depth features in LR space. To enhance the learning process of DSRNet in LR-to-HR mapping, we introduce two mapping constraints in LR space: 1) the cycle-consistent constraint, which offers additional supervision by establishing a closed loop between LR-to-HR and HR-to-LR mappings, and 2) the region-level contrastive constraint, aimed at reinforcing region-specific HR representations by explicitly modeling the consistency between LR and HR spaces. To leverage the color image effectively, we introduce a feature screening module (FSM) to adaptively fuse color features at different layers, which can simultaneously maintain strong structural context and suppress texture distraction through subspace generation and image projection. Comprehensive experimental results across synthetic and real-world benchmark datasets unequivocally demonstrate the superiority of our proposed method over state-of-the-art DSR methods. Our MCNet achieves an average MAD reduction of 3.7% and 7.5% over state-of-the-art DSR method for \(\times\) 8 and \(\times\) 16 cases on Milddleburry dataset, respectively, without incurring additional costs during inference.

Learning Complementary Correlations for Depth Super-Resolution With Incomplete Data in Real World.

Depth information is a significant ingredient to visually perceive the physical world. However, mainstream depth sensors, e.g., time-of-flight (ToF) cameras, often measure incomplete and low-resolution depth data, resulting in low-quality visual perception. In this article, we try to address a potentially valuable task, i.e., depth super-resolution (DSR) with incomplete data, which recovers dense and high-resolution depth map from incomplete and low-resolution one. To tackle this task, we introduce a novel incomplete DSR (IDSR) framework, including a primary branch for DSR to recover high-frequency details, and an auxiliary branch for depth completion (DC) to fill missing pixels. More importantly, we propose two modules, joint correlation learning (JCL) and iterative-cross (IC), to enhance the learning of complementary information flows between the two branches. The former module aims to learn the correlative relationships of the two branches, whilst the latter module adequately fuses higher level representations for more precise predictions. Extensive experiments show that our framework is effective and achieves the state-of-the-art performance on the real-world RGB-D-D and the synthetic NYUv2 datasets.

Improving Shape Transformations for RGB Cameras Using Photometric Stereo

The emergence of low-cost red, green, and blue (RGB) cameras has significantly impacted various computer vision tasks. However, these cameras often produce depth maps with limited object details, noise, and missing information. These limitations can adversely affect the quality of 3D reconstruction and the accuracy of camera trajectory estimation. Additionally, existing depth refinement methods struggle to distinguish shape from complex albedo, leading to visible artifacts in the refined depth maps. In this paper, we address these challenges by proposing two novel methods based on the theory of photometric stereo. The first method, the RGB ratio model, tackles the nonlinearity problem present in previous approaches and provides a closed-form solution. The second method, the robust multi-light model, overcomes the limitations of existing depth refinement methods by accurately estimating shape from imperfect depth data without relying on regularization. Furthermore, we demonstrate the effectiveness of combining these methods with image super-resolution to obtain high-quality, high-resolution depth maps. Through quantitative and qualitative experiments, we validate the robustness and effectiveness of our techniques in improving shape transformations for RGB cameras.

DSRNet: Depth Super-Resolution Network guided by blurry depth and clear intensity edges

Although high resolution (HR) depth images are required in many applications such as virtual reality and autonomous navigation, their resolution and quality generated by consumer depth cameras fall short of the requirements. Existing depth upsampling methods focus on extracting multiscale features of HR color image to guide low resolution (LR) depth upsampling, thus causing blurry and inaccurate edges in depth. In this paper, we propose a depth super-resolution (SR) network guided by blurry depth and clear intensity edges, called DSRNet. DSRNet differentiates effective edges from a number of HR edges with the guidance of blurry depth and clear intensity edges. First, we perform global residual estimation based on an encoder–decoder architecture to extract edge structure from HR color image for depth SR. Then, we distinguish effective edges from HR edges in the decoder side with the guidance of LR depth upsampling. To maintain edges for depth SR, we use intensity edge guidance that extracts clear intensity edges from HR image. Finally, we use residual loss to generate accurate high frequency (HF) residual and reconstruct HR depth maps. Experimental results show that DSRNet successfully reconstructs depth edges in SR results as well as outperforms the state-of-the-art methods in terms of visual quality and quantitative measurements.11The proposed model with some test image pairs are available in https://github.com/lanhui-123/DSRNet.

Depth super-resolution from explicit and implicit high-frequency features

Guided depth super-resolution aims at using a low-resolution depth map and an associated high-resolution RGB image to recover a high-resolution depth map. However, restoring precise and sharp edges near depth discontinuities and fine structures is still challenging for state-of-the-art methods. To alleviate this issue, we propose a novel multi-stage depth super-resolution network, which progressively reconstructs HR depth maps from explicit and implicit high-frequency information. We introduce an efficient transformer to obtain explicit high-frequency information. The shape bias and global context of the transformer allow our model to focus on high-frequency details between objects, i.e., depth discontinuities, rather than texture within objects. Furthermore, we project the input color images into the frequency domain for additional implicit high-frequency cues extraction. Finally, to incorporate the structural details, we develop a fusion strategy that combines depth features and high-frequency information in the multi-stage-scale framework. Exhaustive experiments on the main benchmarks show that our approach establishes a new state-of-the-art. Code will be publicly available at https://github.com/wudiqx106/DSR-EI.

Guided Depth Map Super-Resolution: A Survey

Guided depth map super-resolution (GDSR), which aims to reconstruct a high-resolution depth map from a low-resolution observation with the help of a paired high-resolution color image, is a longstanding and fundamental problem that has attracted considerable attention from computer vision and image processing communities. Myriad novel and effective approaches have been proposed recently, especially with powerful deep learning techniques. This survey is an effort to present a comprehensive survey of recent progress in GDSR. We start by summarizing the problem of GDSR and explaining why it is challenging. Next, we introduce some commonly used datasets and image quality assessment methods. In addition, we roughly classify existing GDSR methods into three categories: filtering-based methods, prior-based methods, and learning-based methods. In each category, we introduce the general description of the published algorithms and design principles, summarize the representative methods, and discuss their highlights and limitations. Moreover, depth-related applications are introduced. Furthermore, we conduct experiments to evaluate the performance of some representative methods based on unified experimental configurations, so as to offer a systematic and fair performance evaluation to readers. Finally, weconclude this survey with possible directions and open problems for further research. All related materials can be found at https://github.com/zhwzhong/Guided-Depth-Map-Super-resolution-A-Survey .

Depth Map Super-Resolution Reconstruction Based on Multi-Channel Progressive Attention Fusion Network

Depth maps captured by traditional consumer-grade depth cameras are often noisy and low-resolution. Especially when upsampling low-resolution depth maps with large upsampling factors, the resulting depth maps tend to suffer from vague edges. To address these issues, we propose a multi-channel progressive attention fusion network that utilizes a pyramid structure to progressively recover high-resolution depth maps. The inputs of the network are the low-resolution depth image and its corresponding color image. The color image is used as prior information in this network to fill in the missing high-frequency information of the depth image. Then, an attention-based multi-branch feature fusion module is employed to mitigate the texture replication issue caused by incorrect guidance from the color image and inconsistencies between the color image and the depth map. This module restores the HR depth map by effectively integrating the information from both inputs. Extensive experimental results demonstrate that our proposed method outperforms existing methods.

DSC-MVSNet: attention aware cost volume regularization based on depthwise separable convolution for multi-view stereo

Deep learning has recently been proven to deliver excellent performance in multi-view stereo (MVS). However, it is difficult for deep learning-based MVS approaches to balance their efficiency and effectiveness. Towards this end, we propose the DSC-MVSNet, a novel coarse-to-fine and end-to-end framework for more efficient and more accurate depth estimation in MVS. In particular, we propose an attention aware 3D UNet-shape network, which first uses the depthwise separable convolutions for cost volume regularization. This mechanism enables effective aggregation of information and significantly reduces the model parameters and computation by transforming the ordinary convolution on cost volume as depthwise convolution and pointwise convolution. Besides, a 3D-Attention module is proposed to alleviate the feature mismatching problem in cost volume regularization and aggregate the important information of cost volume in three dimensions (i.e. channel, space, and depth). Moreover, we propose an efficient Feature Transfer Module to upsample the low-resolution (LR) depth map to a high-resolution (HR) depth map to achieve higher accuracy. With extensive experiments on two benchmark datasets, i.e. DTU and Tanks & Temples, we demonstrate that the parameters of our model are significantly reduced to 25% of the state-of-the-art model MVSNet. Besides, our method outperforms or maintains on par accuracy with the state-of-the-art models. Our source code is available at https://github.com/zs670980918/DSC-MVSNet.

MONOCULAR DEPTH ESTIMATION OF GOOGLE EARTH IMAGES USING CONVOLUTIONAL NEURAL NETWORKS

Abstract. Depth estimation from images is an important task using scene understanding and reconstruction. Recently, encoder-decoder type fully convolutional architectures have gained great success in the area of depth estimation. Depth extraction from aerial and satellite images is one of the important topics in photogrammetry and remote sensing. This is usually done using image pairs, or more than two images. Solving this problem using a single image is still a challenging problem and has not been completely solved. Several convolutional neural networks have been proposed to extract depth from a single image, which act as encoders and decoders. In this article, we use one of these networks, which has performed well for depth estimation, in order to extract height from aerial and satellite images. Our main goal is to investigate the performance of Google Earth satellite data in order to produce a digital elevation model. At first, we extracted the digital model of the target area using ISPRS benchmark data, then we did the same thing using Google Earth satellite images. The paper presents a convolutional neural network for computing a high-resolution depth map given a single RGB Google Earth image. The results show the proper performance of Google Earth satellite images for height extraction. We achieved values of 2.07 m and 0.36 m for the RMS and REL metrics, respectively, which are very comparable and acceptable to the values of 2.04 m and 0.39 m obtained from the ISPRS benchmark images.

Gm-APD LiDAR Single-Source Data Self-Guided: Obtaining High-Resolution Depth Map

Gm-APD LiDAR Single-Source Data Self-Guided: Obtaining High-Resolution Depth Map

Applicability of alpine snow depth estimation based on multitemporal UAV-LiDAR data: A case study in the Maxian Mountains, Northwest China

Unmanned aerial vehicle (UAV) LiDAR has emerged as a viable approach to generate high-resolution snow depth (SD) maps. However, implementing this technology in high-elevation mountains is difficult, and few studies have examined the SD on mountains in Asia until now. Here, we used UAV-LiDAR to estimate the SD on the Maxian Mountains along the eastern edge of High Mountain Asia to assess the reliability of this technology for obtaining SD distributions on a typical alpine periglacial geomorphology. Using LiDAR data from different periods (i.e., 2020/10/19, 2020/11/25 and 2021/04/08) and evenly-distributed in situ measurements, we obtained and validated SDs for both shallow (mean SD less than 15 cm) and deep (mean SD ≥ 15 cm) snow conditions. Our results indicated that the quality of the LiDAR point cloud data met the requirements of the relevant specifications, and the SD distribution map captured the spatial heterogeneity caused by the hummocks. The multitemporal SDs derived from LiDAR were all overestimated, but the accuracy for shallow snow was better than that for deep snow. The root mean square error (RMSE) and mean absolute error (MAE) are 2.23 cm and 1.57 cm, respectively, for deep snow, which were almost three times those of shallow snow. The seasonal deformation of the hummock and manual sampling errors caused by the ice layer over the soil were the main factors affecting accuracy. In contrast, snow surface conditions played a minor role in the error estimators. This study helps refine the use of UAV-LiDAR for SD estimation in similar environments.

Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

Unsupervised learning methods have achieved remarkable performance in monocular depth estimation and camera pose, which mostly solve the multi-task learning problem by using their inner geometry consistency as the self-supervision signal. While most existing approaches mostly adopt the generative model to obtain the depth map prediction, so in the resolution of depth map there is room for improvement. To this end, we present our unsupervised learning architecture based on adversarial learning model, which is used for unsupervised learning of high-resolution single view depth and camera pose. Specifically, we present a multi-scale deep convolutional Generative Adversarial Network (GAN) based learning system, which consists of three networks (pose estimation network PCNN, Generator-D and Discriminator-D for depth map prediction). Furthermore, in order to generate high-resolution depth map, we propose a multi-scale GAN model (MSGAN) to decompose the hard high-quality image generation problem into more manageable sub-problems through a coarse-to-fine process. Then, we modify the overall generation architecture of GAN model by changing the down-sampling and up-sampling components to improve the quality and accuracy of the depth map prediction. Finally, in order to improve the rate of convergence, we use the Least Square Error to increase the penalty for outliers. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI dataset show that the proposed method provides better results for both pose estimation and depth recovery.

Depth Map Super-Resolution via Cascaded Transformers Guidance

Depth information captured by affordable depth sensors is characterized by low spatial resolution, which limits potential applications. Several methods have recently been proposed for guided super-resolution of depth maps using convolutional neural networks to overcome this limitation. In a guided super-resolution scheme, high-resolution depth maps are inferred from low-resolution ones with the additional guidance of a corresponding high-resolution intensity image. However, these methods are still prone to texture copying issues due to improper guidance by the intensity image. We propose a multi-scale residual deep network for depth map super-resolution. A cascaded transformer module incorporates high-resolution structural information from the intensity image into the depth upsampling process. The proposed cascaded transformer module achieves linear complexity in image resolution, making it applicable to high-resolution images. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art techniques for guided depth super-resolution.

Monocular Real Time Full Resolution Depth Estimation Arrangement with a Tunable Lens

This work introduces a real-time full-resolution depth estimation device, which allows integral displays to be fed with a real-time light-field. The core principle of the technique is a high-speed focal stack acquisition method combined with an efficient implementation of the depth estimation algorithm, allowing the generation of real time, high resolution depth maps. As the procedure does not depend on any custom hardware, if the requirements are met, the described method can turn any high speed camera into a 3D camera with true depth output. The concept was tested with an experimental setup consisting of an electronically variable focus lens, a high-speed camera, and a GPU for processing, plus a control board for lens and image sensor synchronization. The comparison with other state of the art algorithms shows our advantages in computational time and precision.

DTS-Depth: Real-Time Single-Image Depth Estimation Using Depth-to-Space Image Construction.

As most of the recent high-resolution depth-estimation algorithms are computationally so expensive that they cannot work in real time, the common solution is using a low-resolution input image to reduce the computational complexity. We propose a different approach, an efficient and real-time convolutional neural network-based depth-estimation algorithm using a single high-resolution image as the input. The proposed method efficiently constructs a high-resolution depth map using a small encoding architecture and eliminates the need for a decoder, which is typically used in the encoder–decoder architectures employed for depth estimation. The proposed algorithm adopts a modified MobileNetV2 architecture, which is a lightweight architecture, to estimate the depth information through the depth-to-space image construction, which is generally employed in image super-resolution. As a result, it realizes fast frame processing and can predict a high-accuracy depth in real time. We train and test our method on the challenging KITTI, Cityscapes, and NYUV2 depth datasets. The proposed method achieves low relative absolute error (0.028 for KITTI, 0.167 for CITYSCAPES, and 0.069 for NYUV2) while working at speed reaching 48 frames per second on a GPU and 20 frames per second on a CPU for high-resolution test images. We compare our method with the state-of-the-art methods on depth estimation, showing that our method outperforms those methods. However, the architecture is less complex and works in real time.

High-Resolution Depth Maps Imaging via Attention-Based Hierarchical Multi-Modal Fusion.

Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging issue for guided DSR is how to correctly select consistent structures and propagate them, and properly handle inconsistent ones. In this paper, we propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided DSR. Specifically, to effectively extract and combine relevant information from LR depth and HR guidance, we propose a multi-modal attention based fusion (MMAF) strategy for hierarchical convolutional layers, including a feature enhancement block to select valuable features and a feature recalibration block to unify the similarity metrics of modalities with different appearance characteristics. Furthermore, we propose a bi-directional hierarchical feature collaboration (BHFC) module to fully leverage low-level spatial information and high-level structure information among multi-scale features. Experimental results show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.

Method for power reduction of demodulation driver circuit in indirect time-of-flight CMOS image sensor.

To improve the depth accuracy of an indirect time-of-flight CMOS image sensor, high modulation frequency is often adopted. It will result in high power consumption of an on-chip demodulation driver, and this problem will be much more serious when the resolution of the sensor is much higher. In this paper, a power reduction method that can lower the power consumption of the demodulation driver circuit during the integration time while obtaining accurate high-resolution depth maps is proposed and analyzed theoretically. The method decreases the number of driven pixels at a high-modulation frequency by a programmable resolution adjustment circuit to obtain an accurate low-resolution depth map. A low-depth accuracy high-resolution depth map is obtained at a low modulation frequency, and then a modified super-resolution algorithm is used to obtain an accurate high resolution solution depth map. To demonstrate the effectiveness of the proposed method, a model is established based on the actual indirect time-of-flight sensor architecture, then the depth error and power consumption are analyzed by the simulation results of the model. In the simulation, 25 MHz and 100 MHz are used as the low modulation frequency and high modulation frequency, respectively. With the best scenario in depth accuracy kept, average power consumption decreases 38.47% and peak power consumption decreases 49.83% while the depth error that is represented by RMSE merely increases 8.08%.

A Depth Map Generation Method for Path Planning of Mobile Robot

Recently, the unmanned mobile robots have received broad applications, such as industrial and security inspection, disinfection and epidemic prevention, warehousing logistics, agricultural picking, etc. In order to drive autonomously from departure to destination, an unmanned mobile robot mounts different sensors to collect information around it and further understand its surrounding environment based on the perceptions. Here we proposed a method to generate high-resolution depth map for given sparse LiDAR point cloud. Our method fits the point cloud into a 3D curve and projects LiDAR data onto the curve surface, and then we make appropriate interpolations of the curve and finally implement the Delaunay triangulation algorithm to all the data points on the 3D curve. The experimental results show that our approach can effectively improve the resolution of depth maps from sparse LiDAR measurements.

3D reconstruction with auto-selected keyframes based on depth completion correction and pose fusion

Dense 3D reconstruction is required for robots to safely navigate or perform advanced tasks. The accurate depth information of the image and its pose are the basis of 3D reconstruction. The resolution of depth maps obtained by LIDAR and RGB-D cameras is limited, and traditional pose calculation methods are not accurate enough. In addition, if each image is used for dense 3D reconstruction, the dense point clouds will increase the amount of calculation. To address these issues, we propose a 3D reconstruction system. Specifically, we propose a depth network of contour and gradient attention, which is used to complete and correct depth maps to obtain high-resolution and high-quality depth maps. Then, we propose a method of fusion of traditional algorithms and deep learning for pose estimation to obtain accurate localization results. Finally, we adopt the method of autonomous selection of keyframes to reduce the number of keyframes, the surfel-based geometric reconstruction is performed to reconstruct the dense 3D environment. On the TUM RGB-D, ICL-NIUM, and KITTI datasets, our method significantly improves the quality of the depth maps, the localization results, and the effect of 3D reconstruction. At the same time, we have also accelerated the speed of 3D reconstruction.