Quality Of Depth Map Research Articles

On Alpha-Expansion-Based Graph-Cut Optimization for Decoder-Side Depth Estimation

In order to achieve high realism an acceptable level of user experience in immersive videos, it is crucial to provide both the best possible quality of depth maps and minimize computational time. In this paper, we propose a new approach to the decoder-side depth estimation (DSDE) problem, which uses the hierarchical alpha-expansion algorithm with additional improvements for the estimation designed to be more robust to compressed video artifacts and limited computational resources. As shown by the experimental results, the proposal simultaneously results in reduction of computation time of the estimation process (by almost 40%) and an improvement of quality of estimated depth maps. The increased quality is demonstrated by more than 6% Bjøntegaard delta gain compared to the Moving Picture Experts Group (MPEG) immersive video state-of-the-art DSDE method.

Multi-view depth estimation based on multi-feature aggregation for 3D reconstruction

Robust 3D reconstruction depends critically on the accuracy of depth maps, and multi-view depth estimate is greatly aided by Multi-View Stereo (MVS). Nevertheless, challenges still exist, particularly in accurately estimating depth in regions with low texture and occlusion edges. We introduce MFA-MVSNet, a novel Multi-Feature Aggregation Multi-View Stereo Network that aggregates diverse features in order to address these issues. The Local Feature Extractor (LFE) is incorporated into MFA-MVSNet to adaptively capture local features across various receptive fields. It also incorporates a Multi-feature Fusion Module (MFM) to combine local features with global features extracted by the Global Feature Extractor (GFE). We utilize neural edge information in a CNN-based Depth Refinement Module (DRM) to iteratively filter the depth map to address depth errors in low-textured regions and occlusion edges. To further improve network robustness and training efficiency, we also introduce a Region Perception Loss (RPL) to lessen the impact of both easily matched and mismatched areas. On both the DTU and BlendedMVS datasets, experimental results demonstrate that MFA-MVSNet outperforms recent advanced methods in terms of depth map quality. Additionally, we incorporate MFA-MVSNet into a multi-view 3D reconstruction pipeline, utilizing its depth maps to enhance downstream tasks such as point cloud reconstruction and neural radiance fields (NeRF), leading to improved 3D reconstruction performance.

Unsupervised Domain Adaptation Depth Estimation Based on Self-attention Mechanism and Edge Consistency Constraints

In the unsupervised domain adaptation (UDA) (Akada et al. Self-supervised learning of domain invariant features for depth estimation, in: 2022 IEEE/CVF winter conference on applications of computer vision (WACV), pp 3377–3387 (2022). 10.1109/WACV51458.2022.00107) depth estimation task, a new adaptive approach is to use the bidirectional transformation network to transfer the style between the target and source domain inputs, and then train the depth estimation network in their respective domains. However, the domain adaptation process and the style transfer may result in defects and biases, often leading to depth holes and instance edge depth missing in the target domain’s depth output. To address these issues, We propose a training network that has been improved in terms of model structure and supervision constraints. First, we introduce a edge-guided self-attention mechanism in the task network of each domain to enhance the network’s attention to high-frequency edge features, maintain clear boundaries and fill in missing areas of depth. Furthermore, we utilize an edge detection algorithm to extract edge features from the input of the target domain. Then we establish edge consistency constraints between inter-domain entities in order to narrow the gap between domains and make domain-to-domain transfers easier. Our experimental demonstrate that our proposed method effectively solve the aforementioned problem, resulting in a higher quality depth map and outperforming existing state-of-the-art methods.

Lifelong-MonoDepth: Lifelong Learning for Multidomain Monocular Metric Depth Estimation.

With the rapid advancements in autonomous driving and robot navigation, there is a growing demand for lifelong learning (LL) models capable of estimating metric (absolute) depth. LL approaches potentially offer significant cost savings in terms of model training, data storage, and collection. However, the quality of RGB images and depth maps is sensor-dependent, and depth maps in the real world exhibit domain-specific characteristics, leading to variations in depth ranges. These challenges limit existing methods to LL scenarios with small domain gaps and relative depth map estimation. To facilitate lifelong metric depth learning, we identify three crucial technical challenges that require attention: 1) developing a model capable of addressing the depth scale variation through scale-aware depth learning; 2) devising an effective learning strategy to handle significant domain gaps; and 3) creating an automated solution for domain-aware depth inference in practical applications. Based on the aforementioned considerations, in this article, we present 1) a lightweight multihead framework that effectively tackles the depth scale imbalance; 2) an uncertainty-aware LL solution that adeptly handles significant domain gaps; and 3) an online domain-specific predictor selection method for real-time inference. Through extensive numerical studies, we show that the proposed method can achieve good efficiency, stability, and plasticity, leading the benchmarks by 8%-15%. The code is available at https://github.com/FreeformRobotics/Lifelong-MonoDepth.

Quality-Aware Selective Fusion Network for V-D-T Salient Object Detection.

Depth images and thermal images contain the spatial geometry information and surface temperature information, which can act as complementary information for the RGB modality. However, the quality of the depth and thermal images is often unreliable in some challenging scenarios, which will result in the performance degradation of the two-modal based salient object detection (SOD). Meanwhile, some researchers pay attention to the triple-modal SOD task, namely the visible-depth-thermal (VDT) SOD, where they attempt to explore the complementarity of the RGB image, the depth image, and the thermal image. However, existing triple-modal SOD methods fail to perceive the quality of depth maps and thermal images, which leads to performance degradation when dealing with scenes with low-quality depth and thermal images. Therefore, in this paper, we propose a quality-aware selective fusion network (QSF-Net) to conduct VDT salient object detection, which contains three subnets including the initial feature extraction subnet, the quality-aware region selection subnet, and the region-guided selective fusion subnet. Firstly, except for extracting features, the initial feature extraction subnet can generate a preliminary prediction map from each modality via a shrinkage pyramid architecture, which is equipped with the multi-scale fusion (MSF) module. Then, we design the weakly-supervised quality-aware region selection subnet to generate the quality-aware maps. Concretely, we first find the high-quality and low-quality regions by using the preliminary predictions, which further constitute the pseudo label that can be used to train this subnet. Finally, the region-guided selective fusion subnet purifies the initial features under the guidance of the quality-aware maps, and then fuses the triple-modal features and refines the edge details of prediction maps through the intra-modality and inter-modality attention (IIA) module and the edge refinement (ER) module, respectively. Extensive experiments are performed on VDT-2048 dataset, and the results show that our saliency model consistently outperforms 13 state-of-the-art methods with a large margin. Our code and results are available at https://github.com/Lx-Bao/QSFNet.

Depth map super-resolution via learned nonlocal model and enhanced local regularization

The appearance of consumer depth cameras makes it easy to obtain depth maps of a scene. However, the quality of depth maps is low and cannot satisfy the practical application. This paper proposes a novel depth map super-resolution method, which combines the advantages of the learning-based method and the reconstruction-based method, and incorporates both the learned nonlocal self-similarity prior and enhanced local geometry prior. On one hand, the nonlocal self-similarity prior is learned from the external examples by the patch group-based Gaussian mixture model (PG-GMM) and then used to explore the nonlocal self-similarity of depth maps. On the other hand, the local geometry prior is explored in both the gradient and spatial domain. For the gradient domain, the hyper-Laplacian prior is introduced to model the gradient distribution. For the spatial domain, a structure similarity-based auto-regressive model is proposed to acquire local correlation in a small area of a depth map. Finally, the learned nonlocal and enhanced local prior are cast into a unified optimization framework to acquire an estimation of the high-resolution image. Compared with state-of-the-art methods, experimental results indicate that our method can achieve better reconstruction performance in terms of both quantitative and qualitative evaluations.

Scene-aware refinement network for unsupervised monocular depth estimation in ultra-low altitude oblique photography of UAV

Using depth estimation joint target detection networks to locate targets in the UAV field of view is a novel application in the depth estimation research field. The presence of more depth variations and low-texture regions in the ultra-low altitude oblique photographic images make them trickier to train for an excellent depth estimation network compared to autonomous driving scenarios. This presents a challenge in achieving optimal training. This study investigates the problem of unsupervised monocular depth estimation for ultra-low altitude oblique photography images. It aims to make subsequent advanced vision tasks better benefit from excellent depth estimation results in terms of overcoming complex scenes. The lack of effective back-projection directionality in training using adjacent frames is attributed to the extensive low-textured areas contained in the training data for complex ultra-low altitude oblique photography. We propose a self-supervised scene-aware refinement learning architecture from the perspective of enhancing feature perception to deal with such problems. The architecture consists of a multi-resolution feature fusion depth network and a perceptual refinement network (PRNet), together with a pose network to enhance regional differences in complex environments from a refined feature context perspective to obtain higher quality depth maps. We rethink the problem of depth information recovery and design the edge information aggregation (EIA) module, which is configured in the decoder section to refine the local region depth detail representation. We design several loss terms to constrain the training of the network in order to improve the quality of depth estimation. Our method is compared with six state-of-the-art self-supervised monocular depth estimation methods on three datasets (UAVid 2020, WildUAV, UAV ula). The experimental results demonstrate that our model achieves the best performance in most scenarios. The code and the private dataset (UAV ula) can be publicly available at https://github.com/takisu0916/MRFEDepth.

Multiscale and multidirection depth map super resolution with semantic inference

AbstractDepth map super resolution has been paid much attention in 3D applications due to the limitation of depth sensors. Few textures in objects with clear contours along them is the most important characteristic of depth map. An efficient image representation should be directional, multiscale and anisotropic. From this we propose a novel multiscale and multidirection depth map super resolution framework with semantic inference to improve the quality of depth maps. In this framework, a multiscale and multidirection depth map contour fusion scheme captures and assembles intrinsic geometrical structures through a multiview non‐subsampled contourlet transform manner. This scheme not only isolates the discontinuities of contours but retains the smoothness along the contours. The semantic inference is also utilized to segment and label the depth map into objects/backgrounds‐level which are coplanar. Furthermore, a semantic‐aware label refinement strategy is introduced to correct the rarely inaccurate labels of the label map for upscaling the target pixel with pixels in the same object or background. Experimental results on benchmark depth map dataset demonstrate that the proposed multiscale and multidirection depth map super resolution framework with semantic inference has a significant improvement than the state‐of‐the‐art algorithms both visually and quantitatively.

Immersive Video Postprocessing for Efficient Video Coding

This paper describes two methods for increasing the efficiency of the MPEG Immersive Video (MIV) coding standard. The methods manipulate the MIV-formatted atlas videos by considering the coding principles of the widely used video encoders, enhancing the encoding efficiency for the immersive video content. The first method, patch average color modification removes the constant component of all YCbCr components of each patch within atlases, resulting in the reduction of the number and magnitude of edges within a texture atlas video. The second proposed method changes the dynamic range of the geometry (depth) atlas, adapting to the quality of input depth maps. Both methods proposed by the authors of this paper were included into the Test Model for MPEG Immersive Video (TMIV), which is the reference implementation of the MIV codec. Moreover, the metadata syntax relating to the first proposed method was adopted to the ISO/IEC 23090-12 standard.

Polarimetric monocular leaf normal estimation model for plant phenotyping

We develop in this paper an accurate, pixel-level, plant leaf normal estimation model by a single polarization image. Though traditional sensing can generate leaf 3-D surface information, these techniques are limited in application because of sensor cost and acquisition-time-related considerations. To achieve the detailed and monocular estimation capacity, we propose a novel surface polarization reflection model that considers a mixture of diffuse or specular reflections and describes the actual reflection process more accurately than prevailing models. Our model also directly corresponds with the recorded polarization states, allowing for direct implementation, no additional computational cost, and no requirement for prior knowledge. We also propose a new strategy to disambiguate the normal solution associated with polarization-based imaging. In contrast to existing methods, which use auxiliary sensory information for the disambiguation, we derive a coarse normal map directly from our image data using an off-the-shelf convolutional neural network. Consequently, we facilitate instantaneous data acquisition, which is essential when modeling dynamic non-rigid objects. Using the coarse normal map as a constraint and optimizing optical smoothness properties makes our estimated outcome more accurate than state-of-the-art results. Experiments show that our median normal angular error is 5.6°, offering a threefold improvement to current polarimetric methods and equivalent or better than what SfM-MVS methods provide, yet using only a single image. Our leaf orientation map is also more detailed than existing methods while exhibiting robustness to polarization image noise and the guiding depth map quality. Hence, by a single polarization image, we obtain high-quality surface normal data with no additional aid.

DDL-MVS: Depth Discontinuity Learning for Multi-View Stereo Networks

We propose an enhancement module called depth discontinuity learning (DDL) for learning-based multi-view stereo (MVS) methods. Traditional methods are known for their accuracy but struggle with completeness. While recent learning-based methods have improved completeness at the cost of accuracy, our DDL approach aims to improve accuracy while retaining completeness in the reconstruction process. To achieve this, we introduce the joint estimation of depth and boundary maps, where the boundary maps are explicitly utilized for further refinement of the depth maps. We validate our idea by integrating it into an existing learning-based MVS pipeline where the reconstruction depends on high-quality depth map estimation. Extensive experiments on various datasets, namely DTU, ETH3D, “Tanks and Temples”, and BlendedMVS, show that our method improves reconstruction quality compared to our baseline, Patchmatchnet. Our ablation study demonstrates that incorporating the proposed DDL significantly reduces the depth map error, for instance, by more than 30% on the DTU dataset, and leads to improved depth map quality in both smooth and boundary regions. Additionally, our qualitative analysis has shown that the reconstructed point cloud exhibits enhanced quality without any significant compromise on completeness. Finally, the experiments reveal that our proposed model and strategies exhibit strong generalization capabilities across the various datasets.

EGA-Net: Edge feature enhancement and global information attention network for RGB-D salient object detection

With the supplement of texture and geometry cues in depth maps, salient object detection (SOD) shifts from 2D to 3D, aiming to detect the most attractive object in a pair of color and depth images. Previous work primarily focused on regional integrity. Few methods are used to improve the edge quality of prediction results, resulting in a final prediction with a complete structure but blurred edges. Moreover, due to the complexity of real-life scenarios, the problem of effectively separating the salient object from complex background has become a hot potato. Aiming to address these issues, we propose a novel network, EGA-Net, to improve the edge quality and highlight the main features of the salient object. Specifically, in the EGA-Net, we propose a feature interaction (FI) module and an edge feature enhancement (EFE) module, respectively. Among them, the FI module is used to remove unimodal feature redundancy, capture multi-modal feature complementarity, and reduce the contamination of low-quality depth maps. The EFE is used to improve the edge quality of the final salient object prediction results. Furthermore, a Global Information Guide Integration (GIGI) module has been proposed to suppress the background noise and effectively highlight the salient objects’ main features. It uses interleaving and fusion methods to automatically select and enhance the vital information in the original input features under the guidance of global features. We put the training of EGA-Net under the supervision of a new hybrid loss function that can simultaneously take global pixel point, foreground, and depth map quality into account. Quantitative and qualitative experiment results demonstrate that our method outperforms the 19 advanced methods on eight publicly available RGB-D salient object detection datasets with five evaluation metrics. You can find the code and results of our method athttps://github.com/guanyuzong/EGA-Net.

Depth Map Prediction of Occluded Objects Using Structure Tensor with Gain Regularization

The creation of the 3D rendering model involves the prediction of an accurate depth map for the input images. A proposed approach of a modified semi-global block matching algorithm with variable window size and the gradient assessment of objects predicts the depth map. 3D modeling and view synthesis algorithms could effectively handle the obtained disparity maps. This work uses the consistency check method to find an accurate depth map for identifying occluded pixels. The prediction of the disparity map by semi-global block matching has used the benchmark dataset of Middlebury stereo for evaluation. The improved depth map quality within a reasonable processing time outperforms the other existing depth map prediction algorithms. The experimental results have shown that the proposed depth map predictioncould identify the inter-object boundaryeven with the presence ofocclusion with less detection error and runtime.We observed that the Middlebury stereo dataset has very few images with occluded objects, which made the attainment of gain cumbersome. Considering this gain, we have created our dataset with occlusion using the structured lighting technique. The proposed regularization term as an optimization process in the graph cut algorithm handles occlusion for different smoothing coefficients. The experimented results demonstrated that our dataset had outperformed the Tsukuba dataset regarding the percentage of occluded pixels.

3D Vision Using Multiple Structured Light-Based Kinect Depth Cameras

Real-time 3D scanning of a scene or object using multiple depth cameras is often required in many applications but is still a challenging task for the computer vision community, especially when the object or scene is partially occluded and dynamic. If active depth sensors are used in this case, their resulting depth map quality gets degraded due to interference between active radiations from each depth sensor. Passive 3D sensors like stereo cameras can avoid the issue of interference as they do not emit any active radiation, but they face correspondence problems. Since releasing the commodity depth sensor Microsoft Kinect, researchers are getting more interested in active depth-sensing. However, Kinect sensors have some easily noticeable limitations concerning 3D reconstruction such as: they can provide depth maps for a limited range, their field of view is restricted and holes are observed in the depth map due to occlusion. The above-mentioned limitations can be overcome if multiple Kinect sensors are used simultaneously instead of a single Kinect sensor. Still, the challenge here is to avoid interference between these sensors. We present a comprehensive review of possible solutions to avoid interference between multiple Kinect sensors. Furthermore, we introduce the Kinect technology in detail along with applications where multiple Kinect sensors are used in the literature. We expect that this paper will be helpful to the researchers who want to use multiple Kinect sensors in sharing the workplace in their research.

Dynamic Knowledge Distillation with Noise Elimination for RGB-D Salient Object Detection.

RGB-D salient object detection (SOD) demonstrates its superiority in detecting in complex environments due to the additional depth information introduced in the data. Inevitably, an independent stream is introduced to extract features from depth images, leading to extra computation and parameters. This methodology sacrifices the model size to improve the detection accuracy which may impede the practical application of SOD problems. To tackle this dilemma, we propose a dynamic knowledge distillation (DKD) method, along with a lightweight structure, which significantly reduces the computational burden while maintaining validity. This method considers the factors of both teacher and student performance within the training stage and dynamically assigns the distillation weight instead of applying a fixed weight on the student model. We also investigate the issue of RGB-D early fusion strategy in distillation and propose a simple noise elimination method to mitigate the impact of distorted training data caused by low quality depth maps. Extensive experiments are conducted on five public datasets to demonstrate that our method can achieve competitive performance with a fast inference speed (136FPS) compared to 12 prior methods.

MONOCULAR DEPTH PREDICTION IN PHOTOGRAMMETRIC APPLICATIONS

Abstract. Despite the recent success of learning-based monocular depth estimation algorithms and the release of large-scale datasets for training, the methods are limited to depth map prediction and still struggle to yield reliable results in the 3D space without additional scene cues. Indeed, although state-of-the-art approaches produce quality depth maps, they generally fail to recover the 3D structure of the scene robustly. This work explores supervised CNN architectures for monocular depth estimation and evaluates their potential in 3D reconstruction. Since most available datasets for training are not designed toward this goal and are limited to specific indoor scenarios, a new metric, large-scale synthetic benchmark (ArchDepth) is introduced that renders near real-world scenarios of outdoor scenes. A encoder-decoder architecture is used for training, and the generalization of the approach is evaluated via depth inference in unseen views in synthetic and real-world scenarios. The depth map predictions are also projected in the 3D space using a separate module. Results are qualitatively and quantitatively evaluated and compared with state-of-the-art algorithms for single image 3D scene recovery.

OutCast: Outdoor Single‐image Relighting with Cast Shadows

AbstractWe propose a relighting method for outdoor images. Our method mainly focuses on predicting cast shadows in arbitrary novel lighting directions from a single image while also accounting for shading and global effects such the sun light color and clouds. Previous solutions for this problem rely on reconstructing occluder geometry, e. g., using multi‐view stereo, which requires many images of the scene. Instead, in this work we make use of a noisy off‐the‐shelf single‐image depth map estimation as a source of geometry. Whilst this can be a good guide for some lighting effects, the resulting depth map quality is insufficient for directly ray‐tracing the shadows. Addressing this, we propose a learned image space ray‐marching layer that converts the approximate depth map into a deep 3D representation that is fused into occlusion queries using a learned traversal. Our proposed method achieves, for the first time, state‐of‐the‐art relighting results, with only a single image as input. For supplementary material visit our project page at: dgriffiths.uk/outcast.

Energy minimization for image focus volume in shape from focus

In shape from focus (SFF) methods, the quality of depth map is mainly dependent on the accuracy level of image focus volume. Most of the SFF techniques optimize focus volume without incorporating any prior or additional structural information about the scene and thus resultant depth maps are deteriorated. We mitigate this deficiency by proposing to optimize focus volume through energy minimization. The proposed energy function contains smoothness and structural similarity along with data term. Smoothness constraint enforces spatial coherence while structural similarity constraint tries to preserve structures which are consistent with image sequence. This results in an optimized focus volume that imitates the underlying scene accurately. For the implementation of our 3D objective function, we employ an efficient technique that decomposes the problem into a sequence of 1D simple sub-problems. Experiments conducted on synthetic and real image sequences from a variety of datasets demonstrate that the proposed method optimizes the focus volume effectively and thus provides improved depth maps.

PDR-Net: Progressive depth reconstruction network for color guided depth map super-resolution

Low spatial resolution is a common problem for depth maps captured by low-cost consumer depth cameras. Depth map super-resolution (DMSR) can improve the quality of depth maps, but it is an ill-posed problem with many challenges. This paper proposes a progressive depth reconstruction network (PDR-Net) to further enhance the performance of DMSR. Specifically, we design an adaptive feature recombination module to recombine depth and color guidance features. We generate sufficient information from the recombined features with the proposed multi-scale feature fusion module, in which multi-scale feature distillation and joint attention mechanism are employed. We learn high frequency compensations for each up-interpolating and reconstruct corresponding high resolution depth maps in the proposed progressive depth reconstruction module. Experimental results with benchmark datasets verified the proposed method’s superiority over the state-of-the-art DMSR methods.

RGBD salient object detection based on depth feature enhancement

The depth map contains spatial location information, which has been proven to be beneficial for RGBD salient object detection. However, the quality of the depth map will directly affect the accuracy of RGBD saliency detection. In order to solve the problem of different quality of depth maps, a detection model based on depth enhancement is designed to enhance the available depth information and filter low-quality depth maps. By comparing four evaluation indicators with five advanced models, the experimental results show that the model in this paper is advanced.