Depth Map Research Articles

A2 GSTran: Depth Map Super-Resolution via Asymmetric Attention With Guidance Selection

A² GSTran: Depth Map Super-Resolution via Asymmetric Attention With Guidance Selection

Efficiency-Accuracy Trade-Off in Light Field Estimation with Cost Volume Construction and Aggregation.

The Rich spatial and angular information in light field images enables accurate depth estimation, which is a crucial aspect of environmental perception. However, the abundance of light field information also leads to high computational costs and memory pressure. Typically, selectively pruning some light field information can significantly improve computational efficiency but at the expense of reduced depth estimation accuracy in the pruned model, especially in low-texture regions and occluded areas where angular diversity is reduced. In this study, we propose a lightweight disparity estimation model that balances speed and accuracy and enhances depth estimation accuracy in textureless regions. We combined cost matching methods based on absolute difference and correlation to construct cost volumes, improving both accuracy and robustness. Additionally, we developed a multi-scale disparity cost fusion architecture, employing 3D convolutions and a UNet-like structure to handle matching costs at different depth scales. This method effectively integrates information across scales, utilizing the UNet structure for efficient fusion and completion of cost volumes, thus yielding more precise depth maps. Extensive testing shows that our method achieves computational efficiency on par with the most efficient existing methods, yet with double the accuracy. Moreover, our approach achieves comparable accuracy to the current highest-accuracy methods but with an order of magnitude improvement in computational performance.

Self-supervised monocular depth estimation with self-distillation and dense skip connection

Monocular depth estimation (MDE) is crucial in a wide range of applications, including robotics, autonomous driving and virtual reality. Self-supervised monocular depth estimation has emerged as a promising MDE approach without requiring hard-to-obtain depth labels during training, and multi-scale photometric loss is widely used for self-supervised monocular depth estimation as the self-supervised signal. However, multi-photometric loss is a weak training signal and might disturb the good intermediate features representation. In this paper, we propose a successive depth map self-distillation(SDM-SD) loss, which combines with the single-scale photometric loss to replace the multi-scale photometric loss. Moreover, considering that multi-stage feature representations are essential for dense prediction tasks such as depth estimation, we also propose a dense skip connection, which can efficiently fuse the intermediate features of the encoder and fully utilize them in each stage of the decoder in our encoder–decoder architecture. By applying successive depth map self-distillation loss and dense skip connection, our proposed method can achieve state-of-the-art performance on the KITTI benchmark, and exhibit the best generalization ability on the challenging indoor dataset NYUv2 dataset.

Enhancing the field-of-view of spectral-scanning FMCW LiDAR by multipass configuration with an echelle grating.

We present a spectral-scanning frequency-modulated continuous wave (FMCW) 3D imaging system capable of producing high-resolution depth maps with an extended field of view (FOV). By employing a multipass configuration with an echelle grating, the system achieves an FOV of 5.5° along the grating axis. The resulting depth maps have a resolution of 70 × 40 pixels, with a depth resolution of 5.1 mm. The system employs an echelle grating for beam steering and leverages the multipass configuration for angular FOV magnification. Quantitative depth measurements and 3D imaging results of a static 3D-printed depth variation target are demonstrated. The proposed approach offers a promising solution for enhancing the FOV of spectral-scanning FMCW LiDAR systems within a limited wavelength-swept range, thereby reducing system complexity and cost, paving the way for improved 3D imaging applications.

Improving RGB-D salient object detection by addressing inconsistent saliency problems

RGB-D salient object detection (SOD) models based on a two-stream structure have achieved good performance in single-object scenes. In multi-object scenes, there is an inconsistent saliency problem between RGB modality and depth modality, which deteriorates the accuracy of subsequent fusion results. Inconsistent saliency is caused by the following issues: firstly, artifacts, missing depth values, and confusion in depth maps render depth modality unreliable, leading to increased reliance on RGB modality for results. Secondly, RGB modality and depth modality lack guidance in salient object detection. Thirdly, there is a lack of interaction between modalities. To address these issues, we first propose a depth recovery (DR) block to mitigate the negative effects of both the original and estimated depth maps. Next, we design the saliency detection (SD) block, which effectively guides each modality to focus on salient objects using semantic information. Meanwhile, SD combines multi-scale information to enhance the ability to detect multi-scale objects in each modality. Finally, a specific fusion block (SFB) is designed to fuse salient object information obtained from RGB and depth modalities. Quantitative and qualitative experiments demonstrate that our method achieves state-of-the-art (SOTA) performance among 10 methods.

All-in-Focus Three-Dimensional Reconstruction Based on Edge Matching for Artificial Compound Eye

An artificial compound eye consists of multiple apertures that allow for a large field of view (FOV) while maintaining a small size. Each aperture captures a sub-image, and multiple sub-images are needed to reconstruct the full FOV. The reconstruction process is depth-related due to the parallax between adjacent apertures. This paper presents an all-in-focus 3D reconstruction method for a specific type of artificial compound eye called the electronic cluster eye (eCley). The proposed method uses edge matching to address the edge blur and large textureless areas existing in the sub-images. First, edges are extracted from each sub-image, and then a matching operator is applied to match the edges based on their shape context and intensity. This produces a sparse matching result that is then propagated to the whole image. Next, a depth consistency check and refinement method is performed to refine the depth of all sub-images. Finally, the sub-images and depth maps are merged to produce the final all-in-focus image and depth map. The experimental results and comparative analysis demonstrate the effectiveness of the proposed method.

Lightweight and Error-Tolerant Stereo Matching with a Stochastic Computing Processor

Stereo matching, utilized in diverse fields, poses a challenge to systems in resource-constrained environments due to the significant growth of computational load with image resolution. The challenge is crucial for the systems because fields utilizing stereo matching require short operational time for real-time applications and low power architecture. Stochastic computing (SC) is able to be a valuable approach to address the challenge by reducing the computational load by representing binary numbers with stochastic sequences, which are encoded as a probability value, and by leveraging the concept of mathematical probability. Also, it is possible for a system to be error-tolerant by utilizing the characteristics of stochastic computing. Therefore, in this paper, we propose an approach for lightweight and error-tolerant stereo matching with a hardware-implemented stochastic computing processor. To verify the feasibility and error tolerance of the proposed system, we implemented the proposed system and conducted experiments comparing depth maps with or without stochastic computing by calculating similarities. According to the experimental results, the proposed system indicated no significant differences in output depth maps and achieved an improvement in the depth maps from error-injected input images by an average of 58.95%. Therefore, we demonstrated that stereo matching with stochastic computing is feasible and error-tolerant.

Multimodal Fusion Network for 3-D Lane Detection.

3-D lane detection is a challenging task due to the diversity of lanes, occlusion, dazzle light, and so on. Traditional methods usually use highly specialized handcrafted features and carefully designed postprocessing to detect them. However, these methods are based on strong assumptions and single modal so that they are easily scalable and have poor performance. In this article, a multimodal fusion network (MFNet) is proposed through using multihead nonlocal attention and feature pyramid for 3-D lane detection. It includes three parts: multihead deformable transformation (MDT) module, multidirectional attention feature pyramid fusion (MA-FPF) module, and top-view lane prediction (TLP) ones. First, MDT is presented to learn and mine multimodal features from RGB images, depth maps, and point cloud data (PCD) for achieving optimal lane feature extraction. Then, MA-FPF is designed to fuse multiscale features for presenting the vanish of lane features as the network deepens. Finally, TLP is developed to estimate 3-D lanes and predict their position. Experimental results on the 3-D lane synthetic and ONCE-3DLanes datasets demonstrate that the performance of the proposed MFNet outperforms the state-of-the-art methods in both qualitative and quantitative analyses and visual comparisons.

Dual-Frequency Lidar for Compressed Sensing 3D Imaging Based on All-Phase Fast Fourier Transform

Lidar, with its advantages of high measurement accuracy, fine angular resolution, and strong anti-interference capability, plays a pivotal role in the field of scene depth information acquisition. Traditional approaches to achieving lateral spatial resolution in imaging include raster scanning and array detectors. The former necessitates frequent scanning to acquire depth maps, resulting in time consumption and instability. The latter encounters challenges such as high dark count rates, pixel crosstalk, and excessive costs for obtaining high-resolution images using array detectors. The introduction of compressed sensing (CS) offers a novel perspective on realizing non-scanning three-dimensional imaging. In this context, we propose a novel three-dimensional imaging system that combines CS with coherent dual-frequency continuous-wave lidar and utilizes the all-phase fast Fourier transform to extract both amplitude and phase information. This system requires only M measurements, and through a reconstruction algorithm, it achieves the inversion of depth information for N-pixel scenes (M << N). Integrating cost-effective components such as digital micromirrors and single-point detectors, this affordable system accomplishes three-dimensional imaging of a single target. Notably, it significantly reduces the required number of measurements while concurrently ensuring enhanced eye safety and signal-to-noise ratio.

Self-supervised 3D vehicle detection based on monocular images

The deep learning-based 3D object detection literature on monocular images has been dominated by methods that require supervision in the form of 3D bounding box annotations for training. However, obtaining sufficient 3D annotations is expensive, laborious and prone to introducing errors. To address this problem, we propose a monocular self-supervised approach towards 3D object detection relying solely on observed RGB data rather than 3D bounding boxes for training. We leverage differentiable rendering to apply visual alignment to depth maps, instance masks and point clouds for self-supervision. Furthermore, considering the complexity of autonomous driving scenes, we introduce a point cloud filter to reduce noise impact and design an automatic training set pruning strategy suitable for the self-supervised framework to further improve network performance. We provide detailed experiments on the KITTI benchmark and achieve competitive performance with existing self-supervised methods as well as some fully supervised methods.

AFNet: Asymmetric fusion network for monocular panorama depth estimation

Panoramic depth estimation is a crucial technique in 360°perception, which has gained great popularity in recent years due to its ability to provide complete visual coverage of the scene. Panoramic images are commonly presented in the form of equirectangular projection. However, this projection exhibits significant distortions towards the poles, making it unsuitable to directly apply traditional depth estimation frameworks for perspective imaging. The cubemap projection offers distortion-free images on each face but exhibits obvious discontinuities between adjacent faces. Therefore, according to the respective advantages of these two projection methods, we propose an asymmetric fusion network (AFNet), where the fusion information is only fed back to the equirectangular projection branch during the encoding stage to guide the subsequent encoding. We also design a dual projection feature fusion (DPFF) module in our framework to better combine the strengths of both projections. Furthermore, we introduce an edge information enhancement (EIE) module to enhance the edge contours of the final depth map. Experiments on four representative datasets (including both synthetic and real-world scenarios) show that our proposed AFNet achieves favorable performance compared with existing methods.

A preliminary assessment of regional CO2 storage potential in the onshore northern Perth Basin

The Geological Survey of Western Australia (GSWA) is creating a new carbon dioxide (CO2) geological storage atlas for Western Australia. The atlas will incorporate the latest state-wide datasets to provide a regional assessment of the potential for CO2 sequestration across the Perth, Southern Carnarvon, Northern Carnarvon (onshore and State waters), Canning and Officer basins. Multi-1D modelling of subsurface temperature in the northern Perth Basin incorporates newly collated temperature data, updated regional depth maps, and stratigraphic revisions. This modelling captures lateral subsurface temperature variations that are incorporated into isothermal depth and temperature maps identifying reservoir units within the optimum temperature–pressure window for CO2 storage. Isopach maps of associated sealing units indicate where they are of adequate thickness to contain CO2. Modelled present-day temperature maps of seven key reservoir units reveal promising CO2 storage prospectivity, with all postulated reservoir units having significant extents within optimum CO2 storage temperature and pressure ranges. Whereas older reservoir units are prospective for CO2 storage towards the margins of the basin, younger reservoir units in the shallower part of the succession are only prospective in or near the Dandaragan Trough, the deepest part of the basin. The critical upper temperature limit of 31.1°C is shallower in the north and west of the basin, where the modelled geothermal gradient is higher. All state-wide raw and interpreted datasets including temperature and depth maps will be published and available for download on the GSWA Western Australian Petroleum and Geothermal Information Management System website.

Learning content-aware feature fusion for guided depth map super-resolution

RGB-D data including paired RGB color images and depth maps is widely used in downstream computer vision tasks. However, compared with the acquisition of high-resolution color images, the depth maps captured by consumer-level sensors are always in low resolution. Within decades of research, the most state-of-the-art (SOTA) methods of depth map super-resolution cannot adaptively tune the guidance fusion for all feature positions by channel-wise feature concatenation with spatially sharing convolutional kernels. This paper proposes JTFNet to resolve this issue, which simulates the traditional Joint Trilateral Filter (JTF). Specifically, a novel JTF block is introduced to adaptively tune the fusion pattern between the color features and the depth features for all feature positions. Moreover, based on the variant of JTF block whose target features and guidance features are in the cross-scale shape, the fusion for depth features is performed in a bi-directional way. Therefore, the error accumulation along scales can be effectively mitigated by iteratively HR feature guidance. Compared with the SOTA methods, the sufficient experiment is conducted on the mainstream synthetic datasets and real datasets, i.e., Middlebury, NYU and ToF-Mark, which shows remarkable improvement of our JTFNet.

Unsupervised Domain Adaptation Depth Estimation Based on Self-attention Mechanism and Edge Consistency Constraints

In the unsupervised domain adaptation (UDA) (Akada et al. Self-supervised learning of domain invariant features for depth estimation, in: 2022 IEEE/CVF winter conference on applications of computer vision (WACV), pp 3377–3387 (2022). 10.1109/WACV51458.2022.00107) depth estimation task, a new adaptive approach is to use the bidirectional transformation network to transfer the style between the target and source domain inputs, and then train the depth estimation network in their respective domains. However, the domain adaptation process and the style transfer may result in defects and biases, often leading to depth holes and instance edge depth missing in the target domain’s depth output. To address these issues, We propose a training network that has been improved in terms of model structure and supervision constraints. First, we introduce a edge-guided self-attention mechanism in the task network of each domain to enhance the network’s attention to high-frequency edge features, maintain clear boundaries and fill in missing areas of depth. Furthermore, we utilize an edge detection algorithm to extract edge features from the input of the target domain. Then we establish edge consistency constraints between inter-domain entities in order to narrow the gap between domains and make domain-to-domain transfers easier. Our experimental demonstrate that our proposed method effectively solve the aforementioned problem, resulting in a higher quality depth map and outperforming existing state-of-the-art methods.

CSRNet: Focusing on critical points for depth completion

Depth completion is an effective method for generating dense depth maps from sparse ones. In recent studies, the majority of points, which we call standard points, often exhibit sub-optimal performance. This issue arises from the need to fit only very few points, termed as challenging points, which consist of noises and regions with discontinuous depth in the ground-truth. On the other hand, traditional evaluations can not recognize this situation, and are dominated by these limited challenging points, whose performance improvements may not significantly benefit related tasks. In contrast, standard points, which are critical for these tasks, are not effectively measured. This discrepancy highlights the need for a more targeted approach and evaluation method for depth completion. In order to solve the above problems, we propose a standard-point-enhancing learning paradigm. This paradigm aims to improve the performance on standard points, which consists of a Cascaded Segmentation-to-Regression Networks (CSRNet) and a Mining L1 loss. CSRNet includes two branches: DSNet and DRNet. DSNet uses segmentation to generate a coarse depth map, providing challenging-point-insensitive information. DRNet adopts a coarse-to-fine approach to learn the residual depth map between the coarse depth map and the ground-truth depth map. In addition, our Mining L1 loss leverages the segmentation results to filter out potential challenging points. This approach allows the network to concentrate more effectively on standard points. Lastly, we introduce the Minimum Error (ME) Curves as a new way to measure the performance of predicted depth maps in a flexible and comprehensive manner, irrespective of whether the points are standard or challenging. Experimental results on the KITTI and NYUDv2 datasets show that our approach significantly improves accuracy on the majority of points.

Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images

Colonoscopy is the choice procedure to diagnose, screening, and treat the colon and rectum cancer, from early detection of small precancerous lesions (polyps), to confirmation of malign masses. However, the high variability of the organ appearance and the complex shape of both the colon wall and structures of interest make this exploration difficult. Learned visuospatial and perceptual abilities mitigate technical limitations in clinical practice by proper estimation of the intestinal depth. This work introduces a novel methodology to estimate colon depth maps in single frames from monocular colonoscopy videos. The generated depth map is inferred from the shading variation of the colon wall with respect to the light source, as learned from a realistic synthetic database. Briefly, a classic convolutional neural network architecture is trained from scratch to estimate the depth map, improving sharp depth estimations in haustral folds and polyps by a custom loss function that minimizes the estimation error in edges and curvatures. The network was trained by a custom synthetic colonoscopy database herein constructed and released, composed of 248400 frames (47 videos), with depth annotations at the level of pixels. This collection comprehends 5 subsets of videos with progressively higher levels of visual complexity. Evaluation of the depth estimation with the synthetic database reached a threshold accuracy of 95.65%, and a mean-RMSE of 0.451cm, while a qualitative assessment with a real database showed consistent depth estimations, visually evaluated by the expert gastroenterologist coauthoring this paper. Finally, the method achieved competitive performance with respect to another state-of-the-art method using a public synthetic database and comparable results in a set of images with other five state-of-the-art methods. Additionally, three-dimensional reconstructions demonstrated useful approximations of the gastrointestinal tract geometry. Code for reproducing the reported results and the dataset are available at https://github.com/Cimalab-unal/ColonDepthEstimation.

A concise but high-performing network for image guided depth completion in autonomous driving

Depth completion is a crucial task in autonomous driving, aiming to convert a sparse depth map into a dense depth prediction. The sparse depth map serves as a partial reference for the actual depth, and the fusion of RGB images is frequently employed to augment the completion process owing to its inherent richness in semantic information. Image-guided depth completion confronts three principal challenges: (1) the effective fusion of the two modalities; (2) the enhancement of depth information recovery; and (3) the realization of real-time predictive capabilities requisite for practical autonomous driving scenarios. In response to these challenges, we propose a concise but high-performing network, named CHNet, to achieve high-performance depth completion with an elegant and straightforward architecture. Firstly, we use a fast guidance module to fuse the two sensor features, harnessing abundant auxiliary information derived from the color space. Unlike the prevalent complex guidance modules, our approach adopts an intuitive and cost-effective strategy. In addition, we find and analyze the optimization inconsistency problem for observed and unobserved positions. To mitigate this challenge, we introduce a decoupled depth prediction head, tailored to better discern and predict depth values for both valid and invalid positions, incurring minimal additional inference time. Capitalizing on the dual-encoder and single-decoder architecture, the simplicity of CHNet facilitates an optimal balance between accuracy and computational efficiency. In benchmark evaluations on the KITTI depth completion dataset, CHNet demonstrates competitive performance metrics and inference speeds relative to contemporary state-of-the-art methodologies. To assess the generalizability of our approach, we extend our evaluations to the indoor NYUv2 dataset, where CHNet continues to yield impressive outcomes. The code of this work will be available at https://github.com/lmomoy/CHNet.

Saliency-Guided Point Cloud Compression for 3D Live Reconstruction

3D modeling and reconstruction are critical to creating immersive XR experiences, providing realistic virtual environments, objects, and interactions that increase user engagement and enable new forms of content manipulation. Today, 3D data can be easily captured using off-the-shelf, specialized headsets; very often, these tools provide real-time, albeit low-resolution, integration of continuously captured depth maps. This approach is generally suitable for basic AR and MR applications, where users can easily direct their attention to points of interest and benefit from a fully user-centric perspective. However, it proves to be less effective in more complex scenarios such as multi-user telepresence or telerobotics, where real-time transmission of local surroundings to remote users is essential. Two primary questions emerge: (i) what strategies are available for achieving real-time 3D reconstruction in such systems? and (ii) how can the effectiveness of real-time 3D reconstruction methods be assessed? This paper explores various approaches to the challenge of live 3D reconstruction from typical point cloud data. It first introduces some common data flow patterns that characterize virtual reality applications and shows that achieving high-speed data transmission and efficient data compression is critical to maintaining visual continuity and ensuring a satisfactory user experience. The paper thus introduces the concept of saliency-driven compression/reconstruction and compares it with alternative state-of-the-art approaches.

FusionVision: A Comprehensive Approach of 3D Object Reconstruction and Segmentation from RGB-D Cameras Using YOLO and Fast Segment Anything.

In the realm of computer vision, the integration of advanced techniques into the pre-processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth maps, as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color RGB and depth D channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information in order to improve post-processes such as object 6D pose estimation, Simultanious Localization and Mapping (SLAM) operations, accurate 3D dataset extraction, etc. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation.

Self-supervised learning for RGB-D object tracking

Recently, there has been a growing interest in RGB-D object tracking thanks to its promising performance achieved by combining visual information with auxiliary depth cues. However, the limited volume of annotated RGB-D tracking data for offline training has hindered the development of a dedicated end-to-end RGB-D tracker design. Consequently, the current state-of-the-art RGB-D trackers mainly rely on the visual branch to support the appearance modelling, with the depth map utilised for elementary information fusion or failure reasoning of online tracking. Despite the achieved progress, the current paradigms for RGB-D tracking have not fully harnessed the inherent potential of depth information, nor fully exploited the synergy of vision-depth information. Considering the availability of ample unlabelled RGB-D data and the advancement in self-supervised learning, we address the problem of self-supervised learning for RGB-D object tracking. Specifically, an RGB-D backbone network is trained on unlabelled RGB-D datasets using masked image modelling. To train the network, the masking mechanism creates a selective occlusion of the input visible image to force the corresponding aligned depth map to help with discerning and learning vision-depth cues for the reconstruction of the masked visible image. As a result, the pre-trained backbone network is capable of cooperating with crucial visual and depth features of the diverse objects and background in the RGB-D image. The intermediate RGB-D features output by the pre-trained network can effectively be used for object tracking. We thus embed the pre-trained RGB-D network into a transformer-based tracking framework for stable tracking. Comprehensive experiments and the analysis of the results obtained on several RGB-D tracking datasets demonstrate the effectiveness and superiority of the proposed RGB-D self-supervised learning framework and the following tracking approach.