Depth Map Prediction Research Articles

Enhancing Automated Loading and Unloading of Ship Unloaders through Dynamic 3D Coordinate System with Deep Learning

This paper proposes a deep learning approach for accurate pose estimation in ship unloaders, improving grasping accuracy by reconstructing 3D coordinates. A convolutional neural network optimizes depth map prediction from RGB images, further enhanced by a conditional generative adversarial network to refine quality. Evaluation of simulated ship unloading tasks showed over 90% grasping success rate, outperforming baseline methods. This research offers valuable insights into advanced visual perception and deep learning for next-generation automated cargo handling.

Virtual reality modeling application based on multi perspective and deep learning in the new media presentation and brand building of Dongguan City memory

AbstractTo help the new media presentation and brand building of Dongguan City memory, a virtual reality modeling model based on multi‐perspective and deep learning is proposed. First, in order to address the issue of imbalanced input and output information in depth map prediction tasks, as well as poor accuracy of predicted depth map boundaries, a depth map prediction model based on multiple perspectives and deep learning is built. Then, a single perspective modeling framework is proposed to address the scarcity of perspectives in practical situations, and a dynamic fusion model is built for single‐view virtual reality scenes based on multi‐view generation networks. The results indicated that the mean square error, average relative error, and average logarithmic error were minimized at 0.52, 0.138, and 0.068, respectively. The multi‐threshold accuracy index demonstrated peak values at 0.772, 0.8821, and 0.947, respectively. The grid simplification algorithm exhibited the shortest running times at 1.57, 2.52, 3.91, and 6.53 s, respectively. Moreover, the single‐view modeling frame displayed the smallest angle distance at 0.1449, while the overlap degree of the point cloud scene reached the highest levels at 77.47, 79.49, 83.5, and 84.47, respectively. To sum up, the model has a good application effect in virtual reality modeling and positively affects virtual reality technology development.

SRNSD: Structure-Regularized Night-Time Self-Supervised Monocular Depth Estimation for Outdoor Scenes.

Deep CNNs have achieved impressive improvements for night-time self-supervised depth estimation form a monocular image. However, the performance degrades considerably compared to day-time depth estimation due to significant domain gaps, low visibility, and varying illuminations between day and night images. To address these challenges, we propose a novel night-time self-supervised monocular depth estimation framework with structure regularization, i.e., SRNSD, which incorporates three aspects of constraints for better performance, including feature and depth domain adaptation, image perspective constraint, and cropped multi-scale consistency loss. Specifically, we utilize adaptations of both feature and depth output spaces for better night-time feature extraction and depth map prediction, along with high- and low-frequency decoupling operations for better depth structure and texture recovery. Meanwhile, we employ an image perspective constraint to enhance the smoothness and obtain better depth maps in areas where the luminosity jumps change. Furthermore, we introduce a simple yet effective cropped multi-scale consistency loss that utilizes consistency among different scales of depth outputs for further optimization, refining the detailed textures and structures of predicted depth. Experimental results on different benchmarks with depth ranges of 40m and 60m, including Oxford RobotCar dataset, nuScenes dataset and CARLA-EPE dataset, demonstrate the superiority of our approach over state-of-the-art night-time self-supervised depth estimation approaches across multiple metrics, proving our effectiveness.

LS-MVSNet: Lightweight self-supervised multi-view stereo

In recent years, significant progress has been made in multi-view stereo through the adoption of learning-based methods. However, current state-of-the-art approaches exhibit high computational and memory costs, owing to their reliance on semantic segmentation networks, neural rendering, and other advanced technologies for enhancing the accuracy of depth map prediction. To improve the efficiency of resource utilization and maintain accuracy in depth prediction, we present a self-supervised multi-view stereo framework that is lightweight and combines depth feature extraction modules, edge feature extraction modules, and data augmentation techniques. Specifically, the depth feature extraction module is employed to adaptively extract significant information from the input images for depth estimation. The edge feature extraction module is utilized to extract object contour features from the input images to augment the accuracy of depth inference at the edges. Experimental results on the DTU dataset show that our proposed lightweight self-supervised multi-view stereo framework, referred to as LS-MVSNet, outperforms the most advanced self-supervised MVS methods in terms of accuracy. Furthermore, extensive experiments on the Tanks and Temples dataset demonstrate the effective generalization ability of our model. The code will be released.

Conditional Generative Adversarial Network for Monocular Image Depth Map Prediction

Deep map prediction plays a crucial role in comprehending the three-dimensional structure of a scene, which is essential for enabling mobile robots to navigate autonomously and avoid obstacles in complex environments. However, most existing depth estimation algorithms based on deep neural networks rely heavily on specific datasets, resulting in poor resistance to model interference. To address this issue, this paper proposes and implements an optimized monocular image depth estimation algorithm based on conditional generative adversarial networks. The goal is to overcome the limitations of insufficient training data diversity and overly blurred depth estimation contours in current monocular image depth estimation algorithms based on generative adversarial networks. The proposed algorithm employs an enhanced conditional generative adversarial network model with a generator that adopts a network structure similar to UNet and a novel feature upsampling module. The discriminator uses a multi-layer patchGAN conditional discriminator and incorporates the original depth map as input to effectively utilize prior knowledge. The loss function combines the least squares loss function and the L1 loss function. Compared to traditional depth estimation algorithms, the proposed optimization algorithm can effectively restore image contour information and enhance the visualization capability of depth prediction maps. Experimental results demonstrate that our method can expedite the convergence of the model on NYU-V2 and Make3D datasets, and generate predicted depth maps that contain more details and clearer object contours.

Emulating Rainfall-Runoff-Inundation Model using Deep Neural Network with Dimensionality Reduction

Abstract Predicting the spatial distribution of maximum inundation depth (depth-MAP) is important for the mitigation of hydrological disasters induced by extreme precipitation. However, physics-based rainfall-runoff-inundation (RRI) models, which are used operationally to predict hydrological disasters in Japan, require massive computational resources for numerical simulations. Here, we aimed at developing a computationally inexpensive deep learning model (Rain2Depth) that emulates an RRI model. Our study focused on the Omono River (Akita Prefecture, Japan) and predicted the depth-MAP from spatial and temporal rainfall data for individual events. Rain2Depth was developed based on a convolutional neural network (CNN), and predicts depth-MAP from 7-day successive hourly rainfall at 13 rain gauge stations in the basin. For training the Rain2Depth, we simulated the depth-MAP by the RRI model forced by 50-ensembles of 30-year data from large-ensemble weather/climate predictions. Instead of using the input and output data directly, we extracted important features from input and output data with two dimensionality reduction techniques (principal component analysis (PCA) and the CNN approach) prior to training the network. This dimensionality reduction aimed to avoid overfitting caused by insufficient training data. The nonlinear CNN approach was superior to the linear PCA for extracting features. Finally, Rain2Depth was architected by connecting the extracted features between input and output data through a neural network. Rain2Depth-based predictions were more accurate than predictions from our previous model (K20), which used ensemble learning of multiple regularized regressions for a specific station. Whereas the K20 can predict maximum inundation depth only at stations, our study achieved depth-MAP prediction by training only the single model Rain2Depth.

Depth Map Prediction of Occluded Objects Using Structure Tensor with Gain Regularization

The creation of the 3D rendering model involves the prediction of an accurate depth map for the input images. A proposed approach of a modified semi-global block matching algorithm with variable window size and the gradient assessment of objects predicts the depth map. 3D modeling and view synthesis algorithms could effectively handle the obtained disparity maps. This work uses the consistency check method to find an accurate depth map for identifying occluded pixels. The prediction of the disparity map by semi-global block matching has used the benchmark dataset of Middlebury stereo for evaluation. The improved depth map quality within a reasonable processing time outperforms the other existing depth map prediction algorithms. The experimental results have shown that the proposed depth map predictioncould identify the inter-object boundaryeven with the presence ofocclusion with less detection error and runtime.We observed that the Middlebury stereo dataset has very few images with occluded objects, which made the attainment of gain cumbersome. Considering this gain, we have created our dataset with occlusion using the structured lighting technique. The proposed regularization term as an optimization process in the graph cut algorithm handles occlusion for different smoothing coefficients. The experimented results demonstrated that our dataset had outperformed the Tsukuba dataset regarding the percentage of occluded pixels.

A Critical Review of Deep Learning-Based Multi-Sensor Fusion Techniques

In this review, we provide a detailed coverage of multi-sensor fusion techniques that use RGB stereo images and a sparse LiDAR-projected depth map as input data to output a dense depth map prediction. We cover state-of-the-art fusion techniques which, in recent years, have been deep learning-based methods that are end-to-end trainable. We then conduct a comparative evaluation of the state-of-the-art techniques and provide a detailed analysis of their strengths and limitations as well as the applications they are best suited for.

Attention-Based Background/Foreground Monocular Depth Prediction Model Using Image Segmentation

While many monocular depth estimation methods have been proposed, determining depth variations in outdoor scenes remains challenging. Accordingly, this paper proposes an image segmentation-based monocular depth estimation model with attention mechanisms that can address outdoor scene variations. The segmentation model segments images into foreground and background regions and individually predicts depth maps. Moreover, attention mechanisms are also adopted to extract meaningful features from complex scenes to improve foreground and background depth map prediction via a multi-scale decoding scheme. From our experimental results, we observed that our proposed model outperformed previous methods by 27.5% on the KITTI dataset.

Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

Unsupervised learning methods have achieved remarkable performance in monocular depth estimation and camera pose, which mostly solve the multi-task learning problem by using their inner geometry consistency as the self-supervision signal. While most existing approaches mostly adopt the generative model to obtain the depth map prediction, so in the resolution of depth map there is room for improvement. To this end, we present our unsupervised learning architecture based on adversarial learning model, which is used for unsupervised learning of high-resolution single view depth and camera pose. Specifically, we present a multi-scale deep convolutional Generative Adversarial Network (GAN) based learning system, which consists of three networks (pose estimation network PCNN, Generator-D and Discriminator-D for depth map prediction). Furthermore, in order to generate high-resolution depth map, we propose a multi-scale GAN model (MSGAN) to decompose the hard high-quality image generation problem into more manageable sub-problems through a coarse-to-fine process. Then, we modify the overall generation architecture of GAN model by changing the down-sampling and up-sampling components to improve the quality and accuracy of the depth map prediction. Finally, in order to improve the rate of convergence, we use the Least Square Error to increase the penalty for outliers. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI dataset show that the proposed method provides better results for both pose estimation and depth recovery.

MONOCULAR DEPTH PREDICTION IN PHOTOGRAMMETRIC APPLICATIONS

Abstract. Despite the recent success of learning-based monocular depth estimation algorithms and the release of large-scale datasets for training, the methods are limited to depth map prediction and still struggle to yield reliable results in the 3D space without additional scene cues. Indeed, although state-of-the-art approaches produce quality depth maps, they generally fail to recover the 3D structure of the scene robustly. This work explores supervised CNN architectures for monocular depth estimation and evaluates their potential in 3D reconstruction. Since most available datasets for training are not designed toward this goal and are limited to specific indoor scenarios, a new metric, large-scale synthetic benchmark (ArchDepth) is introduced that renders near real-world scenarios of outdoor scenes. A encoder-decoder architecture is used for training, and the generalization of the approach is evaluated via depth inference in unseen views in synthetic and real-world scenarios. The depth map predictions are also projected in the 3D space using a separate module. Results are qualitatively and quantitatively evaluated and compared with state-of-the-art algorithms for single image 3D scene recovery.

Mobile-Based 3D Modeling: An In-Depth Evaluation for the Application in Indoor Scenarios.

Indoor environment modeling has become a relevant topic in several application fields, including augmented, virtual, and extended reality. With the digital transformation, many industries have investigated two possibilities: generating detailed models of indoor environments, allowing viewers to navigate through them; and mapping surfaces so as to insert virtual elements into real scenes. The scope of the paper is twofold. We first review the existing state-of-the-art (SoA) of learning-based methods for 3D scene reconstruction based on structure from motion (SFM) that predict depth maps and camera poses from video streams. We then present an extensive evaluation using a recent SoA network, with particular attention on the capability of generalizing on new unseen data of indoor environments. The evaluation was conducted by using the absolute relative (AbsRel) measure of the depth map prediction as the baseline metric.

Machine Learning Based Computer Vision Application for Visually Disabled People

With the easy availability of technology, smartphones are playing an important role in every person’s life. Also, with the advancements in computer vision based research, Automatic Driving cars, Object Recognition, Depth Map Prediction, Object Distance Estimation, have reached commendable levels of intelligence and accuracy. Combining the research and technological advancements, we can be hopeful in creating a computer vision based mobile-application which will help guide visually disabled people in performing their day to day tasks with easily available mobile applications. With our study, the visually disabled can perform simple tasks like outdoor/indoor navigation without encountering obstacles, also they can avoid accidental collisions with objects in their surroundings. Currently, there are very few applications which provide the same assistance to the visually impaired. Using physical tools like sticks is a very common practice when it comes to avoiding obstacles in a visually disabled person’s path. Our study will be focused on object detection and depth estimation techniques- two of the most popular and advanced fields in Intelligent Computer vision studies. We have explored more on the traditional challenges and future hopes of incorporating these techniques on embedded devices.

SimVODIS: Simultaneous Visual Odometry, Object Detection, and Instance Segmentation.

Intelligent agents need to understand the surrounding environment to provide meaningful services to or interact intelligently with humans. The agents should perceive geometric features as well as semantic entities inherent in the environment. Contemporary methods in general provide one type of information regarding the environment at a time, making it difficult to conduct high-level tasks. Moreover, running two types of methods and associating two resultant information requires a lot of computation and complicates the software architecture. To overcome these limitations, we propose a neural architecture that simultaneously performs both geometric and semantic tasks in a single thread: simultaneous visual odometry, object detection, and instance segmentation (SimVODIS). SimVODIS is built on top of Mask-RCNN which is trained in a supervised manner. Training the pose and depth branches of SimVODIS requires unlabeled video sequences and the photometric consistency between input image frames generates self-supervision signals. The performance of SimVODIS outperforms or matches the state-of-the-art performance in pose estimation, depth map prediction, object detection, and instance segmentation tasks while completing all the tasks in a single thread. We expect SimVODIS would enhance the autonomy of intelligent agents and let the agents provide effective services to humans.

Real-time Depth Estimation Using Recurrent CNN with Sparse Depth Cues for SLAM System

Depth map has been utilized for refinement of geometric information in a variety of fields such as 3D reconstruction and pose estimation in SLAM system where ill-posed problems are occurred. Currently, as learning-based approaches are successfully introduced throughout many problems of vision-based fields, several depth estimation algorithms based on CNN are suggested, which only conduct training of spatial information. Since an image sequence or video used for SLAM system tends to have temporal information, this paper proposes a recurrent CNN architecture for SLAM system to estimate depth map by exploring not only spatial but also temporal information by using convolutional GRU cell, which is constructed to remember weights of past convolutional layers. Furthermore, this paper proposes using additional layers that preserve structure of scenes by utilizing sparse depth cues obtained from SLAM system. The sparse depth cues are produced by projecting reconstructed 3D map into each camera frame, and the sparse cues help to predict accurate depth map avoiding ambiguity of depth map generation of untrained structures in latent space. Despite accuracy of depth cues according to monocular SLAM system degrades than stereo SLAM system, the proposed masking approach, which takes the confidence of depth cues with regard to a relative camera pose between current frame and previous frame, retains the performance of the proposed system with the proposed adaptive regularization in loss function. In the training phase, by preprocessing exponential quantization of ground-truth depth map to eliminate the ill-effects of the captured large distances, the depth map prediction of the proposed system improves more than other baseline methods with accomplishment of real-time system. We expect that this proposed system can be used in SLAM system to refine geometric information for more accurate 3D reconstruction and pose estimation, which are essential parts for robust navigation system of robots.

Peeking behind objects: Layered depth prediction from a single image

While conventional depth estimation can infer the geometry of a scene from a single RGB image, it fails to estimate scene regions that are occluded by foreground objects. This limits the use of depth prediction in augmented and virtual reality applications, that aim at scene exploration by synthesizing the scene from a different vantage point, or at diminished reality. To address this issue, we shift the focus from conventional depth map prediction to the regression of a specific data representation called Layered Depth Image (LDI), which contains information about the occluded regions in the reference frame and can fill in occlusion gaps in case of small view changes. We propose a novel approach based on Convolutional Neural Networks (CNNs) to jointly predict depth maps and foreground separation masks used to condition Generative Adversarial Networks (GANs) for hallucinating plausible color and depths in the initially occluded areas. We demonstrate the effectiveness of our approach for novel scene view synthesis from a single image.

Multi-stage cascaded deconvolution for depth map and surface normal prediction from single image

Understanding the 3D perspective of a scene is imperative in improving the precision of intelligent autonomous systems. The difficulty in understanding is compounded when only one image of the scene is available at disposal. In this regard, we propose a fully convolutional deep framework for predicting the depth map and surface normal from a single RGB image in a common architecture. The DenseNet CNN architecture is employed to learn the complex mapping between an input RGB image and its corresponding 3D primitives. We introduce a novel approach of multi-stage cascaded deconvolution, where the output feature maps of one dense block are reused by concatenating with the feature maps of the corresponding deconvolution block. These combined feature maps are progressed along the deep network in a pre-activated manner to construct the final output. The network is trained separately for estimating depth and surface normal while keeping the architecture same. The suggested architecture, compared to the counterparts, uses fewer training samples and model parameters. Exhaustive experiments on benchmark dataset not only reveal the efficacy of the proposed multi-stage scheme over the one-way sequential deconvolution but also outperform the state-of-the-art methods.

Real-time Depth Map Prediction and Optimization Based on Adaptive Image Segmentation

Real-time Depth Map Prediction and Optimization Based on Adaptive Image Segmentation