Depth Map Generation Research Articles

Polarimetric 3D integral imaging profilometry under degraded environmental conditions

We propose polarimetric three-dimensional (3D) integral imaging profilometry and investigate its performance under degraded environmental conditions in terms of the accuracy of object depth acquisition. Integral imaging based profilometry provides depth information by capturing and utilizing multiple perspectives of the observed object. However, the performance of depth map generation may degrade due to light condition, partial occlusions, and object surface material. To improve the accuracy of depth estimation in these conditions, we propose to use polarimetric profilometry. Our experiments indicate that the proposed approach may result in more accurate depth estimation under degraded environmental conditions. We measure a number of metrics to evaluate the performance of the proposed polarimetric profilometry methods for generating the depth map under degraded conditions. Experimental results are presented to evaluate the robustness of the proposed method under degraded environment conditions and compare its performance with conventional integral imaging. To the best of our knowledge, this is the first report on polarimetric 3D integral imaging profilometry, and its performance under degraded environments.

Self-supervised monocular depth estimation based on improved dense network and wavelet decomposition

Traditional self-supervised monocular depth estimation models do not adequately extract and fuse shallow features, which can easily lead to problems such as missed detection of small objects and blurred object edges. To address the above problems, this paper proposes a self-supervised monocular depth estimation model based on improved dense network and wavelet decomposition. The model follows the structure of U-net, in which the encoder adopts an improved dense connection method to improve the feature extraction and fusion capabilities of the encoder; at the same time, a detail enhancement module is added to the jump connection to further refine and integrate the multi-scale features output by the encoder; finally, the wavelet decomposition method is introduced in the decoder, forcing the decoder to pay more attention to high-frequency information and achieve refined processing of image edges. Experimental results show that the depth estimation model proposed in this paper has a stronger ability to capture small object features, and the edges of the generated depth map are clearer and more accurate.

Research on a Matching Method for Vehicle-Borne Laser Point Cloud and Panoramic Images Based on Occlusion Removal

Vehicle-borne mobile mapping systems (MMSs) have been proven as an efficient means of photogrammetry and remote sensing, as they simultaneously acquire panoramic images, point clouds, and positional information along the collection route from a ground-based perspective. Obtaining accurate matching results between point clouds and images is a key issue in data application from vehicle-borne MMSs. Traditional matching methods, such as point cloud projection, depth map generation, and point cloud coloring, are significantly affected by the processing methods of point clouds and matching logic. In this study, we propose a method for generating matching relationships based on panoramic images, utilizing the raw point cloud map, a series of trajectory points, and the corresponding panoramic images acquired using a vehicle-borne MMS as input data. Through a point-cloud-processing workflow, irrelevant points in the point cloud map are removed, and the point cloud scenes corresponding to the trajectory points are extracted. A collinear model based on spherical projection is employed during the matching process to project the point cloud scenes to the panoramic images. An algorithm for vectorial angle selection is also designed to address filtering out the occluded point cloud projections during the matching process, generating a series of matching results between point clouds and panoramic images corresponding to the trajectory points. Experimental verification indicates that the method generates matching results with an average pixel error of approximately 2.82 pixels, and an average positional error of approximately 4 cm, thus demonstrating efficient processing. This method is suitable for the data fusion of panoramic images and point clouds acquired using vehicle-borne MMSs in road scenes, provides support for various algorithms based on visual features, and has promising applications in fields such as navigation, positioning, surveying, and mapping.

An Improved MonoDepth2 Algorithm for Vehicle Monocular Depth Estimation

Depth estimation algorithms are of great significance for autonomous vehicle ranging. In order to solve the problems of large errors, fuzzy image generation and detail loss in depth estimation algorithms, this paper proposes an improved algorithm of the MonoDepth2 algorithm. First, the feature extraction network of the original algorithm is changed to the ResNeXt block to improve the running speed and enrich feature information. Secondly, the Convolutional Block Attention Module attention mechanism is added to the depth estimation network to improve the algorithm accuracy and promote detail detection effect. Finally, the Atrous Spatial Pyramid Pooling network structure is added between the encoder and decoder of the depth estimation network to optimize feature fusion at different scales and extract more detailed features. Compared with the original MonoDepth2 algorithm, the improved algorithm reduces the Absolute Relative Error by 0.4 %, the Squared Relative Error by 6.1 %, the Root Mean Square Error by 15 %, and the accuracy(thr=1.25), increased by 0.4 %, The generated depth map recovers deeper image details and a more detailed target contour.

Attention‐optimized 3D segmentation and reconstruction system for sewer pipelines employing multi‐view images

AbstractExisting deep learning‐based defect inspection results on images lack depth information to fully demonstrate the sewer, despite their high accuracy. To address this limitation, a novel attention‐optimized three‐dimensional (3D) segmentation and reconstruction system for sewer pipelines is presented. First, a real‐time sewer segmentation method called AM‐Pipe‐SegNet is developed to inspect defects (i.e., misalignment, obstacle, and fracture) efficiently. Attention mechanisms (AMs) are introduced to improve the performance of segmentation. Second, an attention‐optimized and sparse‐initialized depth estimation network called AM‐Pipe‐DepNet is presented to generate depth maps from multi‐view images. Third, a 2D‐to‐3D mapping algorithm is proposed to remove noise and transform the sewer segmentation results into 3D spaces. Comparison experiments reveal that incorporating AMs into the network significantly enhances pipe segmentation and 3D reconstruction performance. Finally, two digital replicas of real sewer pipes are built based on photos taken by probes, providing valuable insights for sewer maintenance.

Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images

Colonoscopy is the choice procedure to diagnose, screening, and treat the colon and rectum cancer, from early detection of small precancerous lesions (polyps), to confirmation of malign masses. However, the high variability of the organ appearance and the complex shape of both the colon wall and structures of interest make this exploration difficult. Learned visuospatial and perceptual abilities mitigate technical limitations in clinical practice by proper estimation of the intestinal depth. This work introduces a novel methodology to estimate colon depth maps in single frames from monocular colonoscopy videos. The generated depth map is inferred from the shading variation of the colon wall with respect to the light source, as learned from a realistic synthetic database. Briefly, a classic convolutional neural network architecture is trained from scratch to estimate the depth map, improving sharp depth estimations in haustral folds and polyps by a custom loss function that minimizes the estimation error in edges and curvatures. The network was trained by a custom synthetic colonoscopy database herein constructed and released, composed of 248400 frames (47 videos), with depth annotations at the level of pixels. This collection comprehends 5 subsets of videos with progressively higher levels of visual complexity. Evaluation of the depth estimation with the synthetic database reached a threshold accuracy of 95.65%, and a mean-RMSE of 0.451cm, while a qualitative assessment with a real database showed consistent depth estimations, visually evaluated by the expert gastroenterologist coauthoring this paper. Finally, the method achieved competitive performance with respect to another state-of-the-art method using a public synthetic database and comparable results in a set of images with other five state-of-the-art methods. Additionally, three-dimensional reconstructions demonstrated useful approximations of the gastrointestinal tract geometry. Code for reproducing the reported results and the dataset are available at https://github.com/Cimalab-unal/ColonDepthEstimation.

DepthGAN: GAN-based depth generation from semantic layouts

AbstractExisting GAN-based generative methods are typically used for semantic image synthesis. We pose the question of whether GAN-based architectures can generate plausible depth maps and find that existing methods have difficulty in generating depth maps which reasonably represent 3D scene structure due to the lack of global geometric correlations. Thus, we propose DepthGAN, a novel method of generating a depth map using a semantic layout as input to aid construction, and manipulation of well-structured 3D scene point clouds. Specifically, we first build a feature generation model with a cascade of semantically-aware transformer blocks to obtain depth features with global structural information. For our semantically aware transformer block, we propose a mixed attention module and a semantically aware layer normalization module to better exploit semantic consistency for depth features generation. Moreover, we present a novel semantically weighted depth synthesis module, which generates adaptive depth intervals for the current scene. We generate the final depth map by using a weighted combination of semantically aware depth weights for different depth ranges. In this manner, we obtain a more accurate depth map. Extensive experiments on indoor and outdoor datasets demonstrate that DepthGAN achieves superior results both quantitatively and visually for the depth generation task.

DPF-Nutrition: Food Nutrition Estimation via Depth Prediction and Fusion.

A reasonable and balanced diet is essential for maintaining good health. With advancements in deep learning, an automated nutrition estimation method based on food images offers a promising solution for monitoring daily nutritional intake and promoting dietary health. While monocular image-based nutrition estimation is convenient, efficient and economical, the challenge of limited accuracy remains a significant concern. To tackle this issue, we proposed DPF-Nutrition, an end-to-end nutrition estimation method using monocular images. In DPF-Nutrition, we introduced a depth prediction module to generate depth maps, thereby improving the accuracy of food portion estimation. Additionally, we designed an RGB-D fusion module that combined monocular images with the predicted depth information, resulting in better performance for nutrition estimation. To the best of our knowledge, this was the pioneering effort that integrated depth prediction and RGB-D fusion techniques in food nutrition estimation. Comprehensive experiments performed on Nutrition5k evaluated the effectiveness and efficiency of DPF-Nutrition.

A comprehensive system for 3D display: From image capture to autostereoscopic playback

The production and playback processes involved in rendering 3D content exhibit a notable degree of intricacy and inefficiency. This article presents a comprehensive array of remedies aimed at addressing challenges to the acquisition, processing, and display of such content. The proposed framework is strategically compartmentalized into three key modules: the depth map generation and optimization module, the multi-viewpoint generation and optimization module, and the 3D content encoding and display module. Within the domain of depth map generation and optimization, we introduce a self-supervised convolutional network that builds upon the PSMNet(Pyramid Stereo Matching Network) architecture. This network serves the purpose of extracting accurate depth data. In the realm of multi-viewpoint generation and optimization, we put forth a novel partial convolutional neural network founded upon the Edge-PUnet framework. This network is specifically designed to enhance virtual viewpoint images. The efficacy of the depth data generation algorithm is substantiated by achieving an Out-Noc value of 2.27% on the KITTI-2012 dataset. Furthermore, the virtual viewpoint optimization algorithm yields notable outcomes, as evidenced by attained PSNR and SSIM values of 32.688 and 0.932, respectively, in the context of a street dance sequence. In summation, the outcomes of our study suggest that the implemented system holds promise for elevating the efficiency of 3D content production while concurrently curbing associated costs.

AUGMENTED REALITY-BASED NASAL ENDOSCOPE VIDEO RECONSTRUCTION AND REGISTRATION

Neuroendoscopic surgery is a minimally invasive surgical technique commonly used to remove a tumor through the patient’s mouth or nose, which requires the surgeon to avoid important neural and vascular structures. Augmented reality-based surgical navigation can provide surgeons with more information and visual aids, clarifying the structure and details of the surgical area. It requires real-time registration of intraoperative endoscopic video with pre-operative CT models. However, 3D reconstruction and camera tracking from nasal endoscopic video are challenging due to the narrow nasal cavity and the lack of texture. Besides, the nasal endoscopic datasets that can be used for deep learning-based depth estimation are scarce and hard to elaborate. To this end, a style-augmented module (SAM) is proposed in this study to minimize discrepancies between different endoscopic datasets. An unsupervised approach was trained to generate depth maps and camera motion paths, which were used to reconstruct the nasal scene’s 3D model. The obtained reconstruction was aligned with a pre-operative CT model for augmented reality-based surgical navigation. The results showed that the proposed SAM improved the generalization of the depth estimation model on nasal endoscopy data. The proposed approach to combining augmented reality technology with surgical navigation is considered instrumental in furnishing surgeons with richer information on the nasal endoscopic surgical scene.

Robust 3D breast reconstruction based on monocular images and artificial intelligence for robotic guided oncological interventions.

Breast cancer is a global public health concern. For women with suspicious breast lesions, the current diagnosis requires a biopsy, which is usually guided by ultrasound (US). However, this process is challenging due to the low quality of the US image and the complexity of dealing with the US probe and the surgical needle simultaneously, making it largely reliant on the surgeon's expertise. Some previous works employing collaborative robots emerged to improve the precision of biopsy interventions, providing an easier, safer, and more ergonomic procedure. However, for these equipment to be able to navigate around the breast autonomously, 3D breast reconstruction needs to be available. The accuracy of these systems still needs to improve, with the 3D reconstruction of the breast being one of the biggest focuses of errors. The main objective of this work is to develop a method to obtain a robust 3D reconstruction of the patient's breast, based on RGB monocular images, which later can be used to compute the robot's trajectories for the biopsy. To this end, depth estimation techniques will be developed, based on a deep learning architecture constituted by a CNN, LSTM, and MLP, to generate depth maps capable of being converted into point clouds. After merging several from multiple points of view, it is possible to generate a real-time reconstruction of the breast as a mesh. The development and validation of our method was performed using a previously described synthetic dataset. Hence, this procedure takes RGB images and the cameras' position and outputs the breasts' meshes. It has a mean error of 3.9 mm and a standard deviation of 1.2 mm. The final results attest to the ability of this methodology to predict the breast's shape and size using monocular images.Clinical Relevance- This work proposes a method based on artificial intelligence and monocular RGB images to obtain the breast's volume during robotic guided breast biopsies, improving their execution and safety.

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display.

This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

AI-Driven Permanent Cuttings Monitoring Enables Safer and Faster Drilling

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 211759, “Safer and Faster Drilling Through AI-Driven Permanent Cuttings Monitoring: An Operator’s Approach,” by Georg Ripperger, SPE, and Jörg Peisker, OMV, and Patrick Oberschmidleitner, Montanuniversität Leoben, et al. The paper has not been peer reviewed. _ The project outlined in the complete paper demonstrates the feasibility of using deep-learning and machine-learning (ML) approaches to introduce camera-based solids monitoring to the drilling industry. Despite a short development time, the project proved that it was possible to recognize cuttings, cavings, and anomalies in the solids output using proprietary ML models and regular off-the-shelf hardware. Computer-Vision Techniques for Cuttings Detection Several technologies, such as 2D vision, stereo vision, structured light, or time-of-flight, can be used to detect objects such as drill cuttings on a shale shaker. Depending on the physics behind them, these techniques can recognize individual objects and their dimensions or generate depth maps by measuring the physical distance between the sensor and a defined point on the image. Single 2D vision is the simplest and most cost-effective approach but has limitations in measuring depth. Such cameras do not require any special sensors and can be directly installed close to the area of interest. The captured 2D images are usually converted into optical arrays and further segmented. The stereo-vision setup uses two cameras placed at a defined distance to one another. Both cameras capture an image, which is then processed by changing the resolution and filtering out the noise. The resulting depth map might require higher processing times for higher resolutions. Structured light makes use of predefined patterns of light to detect objects and requires active illumination of the scene. Structured-light techniques can be classified into single-shot and multishot. The multishot technique can be used for static objects and may result in more-accurate results. The time-of-flight technique requires only one device, which must be equipped with a light emitter (usually infrared) and a receiver. During the procedure, the object is irradiated with photons and the time that elapses until these photons are reflected is measured. For each pixel, the distance is measured and a depth map is generated. One challenge with this technique is that the sensor also responds to sunlight, which requires the isolation of the camera and the shaker to ensure reliability. Methodology The first point of contact for a human observer with the drilled cuttings is at the outlet of the shale shakers. At this point, the cuttings are distributed as evenly as possible and mostly free of mud. For a camera-based system, this poses the perfect opportunity to detect individual cuttings (and cavings, if present). If an algorithm can identify individual cuttings or cavings, further geometric parameters such as size and shape can be analyzed. A training data set of video footage filmed in different conditions is divided into subsets used for training, verification, and testing of ML models. As outlined in Fig. 1, the procedure consists of taking video footage from the shale shaker, followed by the extraction of video frames and preprocessing. The data are then fed into the ML algorithm, where the model is trained. After sufficient training, such a model can detect and analyze cuttings and cavings in real time.

A Comparison of Deep Neural Networks for Monocular Depth Map Estimation in Natural Environments Flying at Low Altitude

Currently, the use of Unmanned Aerial Vehicles (UAVs) in natural and complex environments has been increasing, because they are appropriate and affordable solutions to support different tasks such as rescue, forestry, and agriculture by collecting and analyzing high-resolution monocular images. Autonomous navigation at low altitudes is an important area of research, as it would allow monitoring parts of the crop that are occluded by their foliage or by other plants. This task is difficult due to the large number of obstacles that might be encountered in the drone's path. The generation of high-quality depth maps is an alternative for providing real-time obstacle detection and collision avoidance for autonomous UAVs. In this paper, we present a comparative analysis of four supervised learning deep neural networks and a combination of two for monocular depth map estimation considering images captured at low altitudes in simulated natural environments. Our results show that the Boosting Monocular network is the best performing in terms of depth map accuracy because of its capability to process the same image at different scales to avoid loss of fine details.

Automatic translation of sign language with multi-stream 3D CNN and generation of artificial depth maps

Sign languages play an essential role in the cognitive and social development of the deaf, consisting of a natural form of communication and being a symbol of identity and culture. However, hearing loss has a severe social impact due to an existing communication barrier, preventing access to essential services such as education and health. A bi-directional sign language translation may be the solution to bridging the communication gap between the deaf and the listener, completing a two-way communication cycle. Virtual personal assistants can benefit from this technology by extending how users interact with the intelligent system. With this idea, in this work we develop a multi-stream deep learning model to recognize signs of Brazilian (BSL), Indian (ISL), and Korean (KSL) Sign Languages. We combine different types of information for the classification task, using single-stream and multi-stream 3D Convolutional Neural Networks. In addition, considering the largest source of sign data globally – the internet – we propose a depth sensor-free classification method, with depth maps artificially generated through Generative Adversarial Networks. In order to consider the main parameters that encode sign languages, the final architecture is composed of a multi-stream network that receives the segmented hands, the faces, the distances and speeds of the points of articulation, and the RGB frames associated with artificial depth maps. Finally, we provide a visual explanation to understand which regions were important for model decision-making. The best models were obtained using the multi-stream network, presenting an accuracy of 0.91 ± 0.07, and f1-score of 0.90 ± 0.08 on publicly available BSL data set. The results suggest that the multi-stream network with artificially generated depth maps is suitable for the task of sign recognition in different languages.

Autonomous Reinforcement Control of Visual Underwater Vehicles: Real-Time Experiments Using Computer Vision

Swift decision-making based on visual environment perception is crucial for autonomous control of visual underwater vehicles (VUVs) during underwater missions. However, learning perception and decision models individually might result in weak robustness of overall control system as the mismatched state extraction and control decision making are asynchronous. As a remedy, we will introduce in this paper an end-to-end monocular autonomous reinforcement control (MARC) framework for autonomous control of VUVs, which is performed in two cascaded procedures, i.e., 1) perception, where a geometric network (GeoNet) is designed based on a convolutional encoder-decoder network to generate depth maps from input environmental videos; 2) decision, where with depth maps as input, a reinforcement control network (CtrlNet) integrates a convolutional neural network into a deep deterministic policy gradient network and outputs action decisions, which are refined by reinforcement learning algorithm for obstacle-avoiding based autonomous control. Numerical and experimental results demonstrate that the proposed MARC exhibits high-quality depth prediction and is capable of conducting obstacle-avoiding navigation and autonomous control of VUVs with high accuracy and strong robustness.

MODELLING COLOUR ABSORPTION OF UNDERWATER IMAGES USING SFM-MVS GENERATED DEPTH MAPS

Abstract. The problem of colour correction of underwater images concerns not only surveyors, who primarily use images for photogrammetric purposes, but also archaeologists, marine biologists, and many other domains experts whose aim is to study objects and lifeforms underwater. Different methods exist in the literature; some of them provide outstanding results but works involving physical models that take into account additional information and variables (light conditions, depths, camera to objects distances, water properties) that are not always available or can be measured using expensive equipment or calculated using more complicated models. Some other methods have the advantages of working with basically all kinds of dataset, but without considering any geometric information, therefore applying corrections that work only in very generic conditions that most of the time differs from the real-world applications.This paper presents an easy and fast method for restoring the colour information on images captured underwater. The compelling idea is to model light backscattering and absorption variation according to the distance of the surveyed object. This information is always obtainable in photogrammetric datasets, as the model utilises the scene's 3D geometry by creating and using SfM-MVS generated depth maps, which are crucial for implementing the proposed methodology. The results presented visually and quantitatively are promising since they are an excellent compromise to provide a straightforward and easily adaptable workflow to restore the colour information in underwater images.

An unsupervised monocular image depth prediction algorithm using Fourier domain analysis

Aiming at the problems of high cost and low accuracy of scene details during the depth map generation in 3D reconstruction, we propose an unsupervised monocular image depth prediction algorithm based on Fourier domain analysis. Generally speaking, small-scale images can better display depth details, while large-scale images can more reliably display the depth distribution value of the entire image. In order to take advantage of these complementary properties, we crop the input image with different cropped image ratios to generate multiple disparity map candidates, and then use Fourier frequency domain analysis algorithms to fuse disparity mapping candidates into left and right disparity maps. At the same time, we propose a loss function based on MSSIM to compensate the difference between left and right views and realize unsupervised monocular image depth prediction model training. Experimental results show that our method has good performance on the KITTI dataset.

Depth map generationfor mobile navigation systemsbased onobjects localization in images

This paperis aimedto develop a method for a depth map generation based on objects localization in images, obtained througha stereopair. The proposed solution describes theobjects by the following informative elements: contours, interestpoints (points of the greatest curvature of the contour), center of mass of the object. Moreover, to describe the contour of the image, it is proposed to use methods with adjustable detailing, basedonthe wavelet transform, which has frequency-selective properties. The novelty of this method is the possibility of obtaining an approximate depth map by simplifying the calculation of stereo image difference values, which is traditionally used to generate a depth map. Software was developed based on the proposed solutions. Modeling confirmed the effectiveness of the proposed approach. The proposed method makes it possible to significantly reduce the number of computational operations and, consequently, improve depth map generation performance and recommend the proposed method for mobile navigation systems operating in conditions of limited computing and energy resources. The method provides object detection and spatial positioning, makes it possible to obtain reliable information about the distance to objects for other subsystems that use technical vision in their operation, for example, navigation systems for visually impaired people, robotic devices, etc.

Unsupervised multi-view stereo network based on multi-stage depth estimation

In current years, supervised learning multi-view stereo (MVS) methods have achieved impressive performance. However, these methods still suffer the limitation of hard to acquire large-scale depth supervision data, which hinders the generalization ability in never-seen-before scenarios. Recently, some unsupervised-learning methods have been proposed, which relieved the requirement of depth supervision data. However, the generated depth map with lower resolution since the memory consumption grows cubically. In this paper, we propose a novel unsupervised multi-view stereo network based on multi-stage depth estimation, which can increase depth map resolution and generate a dense 3D model with rich details without relying on depth supervision data. To reduce the 3D cost volume highly memory consumption, the progressive coarse-to-fine multiple stages are adopted. Besides, a multi-view group-wise correlation (MV-GwC) module is designed to introduce multi-view correlation prior, which can enhance network performance and further reduce memory consumption. Qualitative and quantitative experiment results show the effectiveness of our method. We outperform some previous supervised and unsupervised MVS methods on DTU and Tanks & Temples benchmarks.