Noisy Depth Maps Research Articles

AI-BASED 3D DETECTION OF PARKED VEHICLES ON A MOBILE MAPPING PLATFORM USING EDGE COMPUTING

Abstract. In this paper we present an edge-based hardware and software framework for the 3D detection and mapping of parked vehicles on a mobile mapping platform for the use case of on-street parking statistics. First, we investigate different point cloud-based 3D object detection methods on our extremely dense and noisy depth maps obtained from low-cost RGB-D sensors to find a suitable object detector and determine the optimal preparation of our data. We then retrain the chosen object detector to detect all types of vehicles, rather than standard cars only. Finally, we design and develop a software framework integrating the newly trained object detector. By repeating the parking statistics of our previous work (Nebiker et al., 2021), our software is tested regarding the detection accuracy. With our edge-based framework, we achieve a precision and recall of 100% and 98% respectively on any parking configuration and vehicle type, outperforming all other known work on on-street parking statistics. Furthermore, our software is evaluated in terms of processing speed and volume of generated data. While the processing speed reaches only 1.9 frames per second due to limited computing resources, the amount of data generated is just 0.25 KB per frame.

A Real-Time Online Learning Framework for Joint 3D Reconstruction and Semantic Segmentation of Indoor Scenes

This letter presents a real-time online vision framework to jointly recover an indoor scene’s 3D structure and semantic label. Given noisy depth maps, a camera trajectory, and 2D semantic labels at train time, the proposed deep neural network based approach learns to fuse the depth over frames with suitable semantic labels in the scene space. Our approach exploits the joint volumetric representation of the depth and semantics in the scene feature space to solve this task. For a compelling online fusion of the semantic labels and geometry in real-time, we introduce an efficient vortex pooling block while dropping the use of routing network in online depth fusion to preserve high-frequency surface details. We show that the context information provided by the semantics of the scene helps the depth fusion network learn noise-resistant features. Not only that, it helps overcome the shortcomings of the current online depth fusion method in dealing with thin object structures, thickening artifacts, and false surfaces. Experimental evaluation on the Replica dataset shows that our approach can perform depth fusion at 37 and 10 frames per second with an average reconstruction F-score of 88% and 91%, respectively, depending on the depth map resolution. Moreover, our model shows an average IoU score of 0.515 on the ScanNet 3D semantic benchmark leaderboard. Code and example dataset information is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/suryanshkumar/online-joint-depthfusion-and-semantic</uri> .

Robust Depth Estimation for Light Field Microscopy.

Light field technologies have seen a rise in recent years and microscopy is a field where such technology has had a deep impact. The possibility to provide spatial and angular information at the same time and in a single shot brings several advantages and allows for new applications. A common goal in these applications is the calculation of a depth map to reconstruct the three-dimensional geometry of the scene. Many approaches are applicable, but most of them cannot achieve high accuracy because of the nature of such images: biological samples are usually poor in features and do not exhibit sharp colors like natural scene. Due to such conditions, standard approaches result in noisy depth maps. In this work, a robust approach is proposed where accurate depth maps can be produced exploiting the information recorded in the light field, in particular, images produced with Fourier integral Microscope. The proposed approach can be divided into three main parts. Initially, it creates two cost volumes using different focal cues, namely correspondences and defocus. Secondly, it applies filtering methods that exploit multi-scale and super-pixels cost aggregation to reduce noise and enhance the accuracy. Finally, it merges the two cost volumes and extracts a depth map through multi-label optimization.

Hybrid Multi-level Regularizations with Sparse Representation for Single Depth Map Super-Resolution

Limited spatial resolution and varieties of degradations are the main restrictions of today’s captured depth map by active 3D sensing devices. Typical restrictions limit the direct use of the obtained depth maps in most of 3D applications. In this paper, we present a single depth map upsampling approach in contrast to the common work of using the corresponding combined color image to guide the upsampling process. The proposed approach employs a multi-level decomposition to convert the depth upsampling process to a classification-based problem via a multi-level classification-based learning algorithm. Hence, the lost high frequency details can be better preserved at different levels. The adopted multi-level decomposition algorithm utilizes $$l_{1} ,$$ and $$l_{0}$$ sparse regularization with total-variation regularization to keep structure- and edge-preserving smoothing with robustness to noisy degradations. In addition, the proposed classification-based learning algorithm supports the accuracy of discrimination by learning discriminative dictionaries that carry original features about each class and learning common shared dictionaries that represent the shared features between classes. The proposed algorithm has been validated via different experiments under variety of degradations using different datasets from different sensing devices. Results show superiority to the state of the art, especially in case of upsampling noisy low-resolution depth maps.

Learning an Intrinsic Image Decomposer Using Synthesized RGB-D Dataset

Intrinsic image decomposition refers to recover the albedo and shading from images, which is an ill-posed problem in signal processing. As realistic labeled data are severely lacking, it is difficult to apply learning methods in this issue. In this letter, we propose using a synthesized dataset to facilitate the solving of this problem. A physically based renderer is used to generate color images and their underlying ground-truth albedo and shading from three-dimensional models. Additionally, we render a Kinect-like noisy depth map for each instance. We utilize this synthetic dataset to train a deep neural network for intrinsic image decomposition and further fine-tune it for real-world images. Our model supports both RGB and RGB-D as input, and it employs both high-level and low-level features to avoid blurry outputs. Experimental results verify the effectiveness of our model on realistic images.

Curve-Structure Segmentation From Depth Maps: A CNN-Based Approach and Its Application to Exploring Cultural Heritage Objects

Motivated by the important archaeological application of exploring cultural heritage objects, in this paper we study the challenging problem of automatically segmenting curve structures that are very weakly stamped or carved on an object surface in the form of a highly noisy depth map. Different from most classical low-level image segmentation methods that are known to be very sensitive to the noise and occlusions, we propose a new supervised learning algorithm based on Convolutional Neural Network (CNN) to implicitly learn and utilize more curve geometry and pattern information for addressing this challenging problem. More specifically, we first propose a Fully Convolutional Network (FCN) to estimate the skeleton of curve structures and at each skeleton pixel, a scale value is estimated to reflect the local curve width. Then we propose a dense prediction network to refine the estimated curve skeletons. Based on the estimated scale values, we finally develop an adaptive thresholding algorithm to achieve the final segmentation of curve structures. In the experiment, we validate the performance of the proposed method on a dataset of depth images scanned from unearthed pottery shards dating to the Woodland period of Southeastern North America.

Probabilistic RGB-D odometry based on points, lines and planes under depth uncertainty

This work proposes a robust visual odometry method for structured environments that combines point features with line and plane segments, extracted through an RGB-D camera. Noisy depth maps are processed by a probabilistic depth fusion framework based on Mixtures of Gaussians to denoise and derive the depth uncertainty, which is then propagated throughout the visual odometry pipeline. Probabilistic 3D plane and line fitting solutions are used to model the uncertainties of the feature parameters and pose is estimated by combining the three types of primitives based on their uncertainties.Performance evaluation on RGB-D sequences collected in this work and two public RGB-D datasets: TUM and ICL-NUIM show the benefit of using the proposed depth fusion framework and combining the three feature-types, particularly in scenes with low-textured surfaces, dynamic objects and missing depth measurements.

Accurate and fast 3D head pose estimation with noisy RGBD images

Head pose estimation plays an essential role in many high-level face analysis tasks. However, accurate and robust pose estimation with existing approaches remains challenging. In this paper, we propose a novel method for accurate three-dimensional (3D) head pose estimation with noisy depth maps and high-resolution color images that are typically produced by popular RGBD cameras such as the Microsoft Kinect. Our method combines the advantages of the high-resolution RGB image with the 3D information of the depth image. For better accuracy and robustness, features are first detected using only the color image, and then the 3D feature points used for matching are obtained by combining depth information. The outliers are then filtered with depth information using rules proposed for depth consistency, normal consistency, and re-projection consistency, which effectively eliminate the influence of depth noise. The pose parameters are then iteratively optimized using the Extended LM (Levenberg-Marquardt) method. Finally, a Kalman filter is used to smooth the parameters. To evaluate our method, we built a database of more than 10K RGBD images with ground-truth poses recorded using motion capture. Both qualitative and quantitative evaluations show that our method produces notably smaller errors than previous methods.

Depth map enhancement using adaptive moving least squares method with a total variation minimization

Accurate and fast depth map acquisition and enhancement is an important issue in the area of computer vision and image processing. In this study, we present a novel method for enhancing noisy depth maps using adaptive total variation minimization, which facilitates noise smoothing and boundary sharpening for a given depth map image but without previous information. We filter the noise in the depth map with a refined total variation minimization technique. Our experimental results demonstrate that the proposed method outperforms other competitive methods in both objective and subjective comparisons of depth map enhancement and denoising.

High-quality depth map upsampling and completion for RGB-D cameras.

This paper describes an application framework to perform high-quality upsampling and completion on noisy depth maps. Our framework targets a complementary system setup, which consists of a depth camera coupled with an RGB camera. Inspired by a recent work that uses a nonlocal structure regularization, we regularize depth maps in order to maintain fine details and structures. We extend this regularization by combining the additional high-resolution RGB input when upsampling a low-resolution depth map together with a weighting scheme that favors structure details. Our technique is also able to repair large holes in a depth map with consideration of structures and discontinuities utilizing edge information from the RGB input. Quantitative and qualitative results show that our method outperforms existing approaches for depth map upsampling and completion. We describe the complete process for this system, including device calibration, scene warping for input alignment, and even how our framework can be extended for video depth-map completion with the consideration of temporal coherence.

The Robust Sequential Estimator: a general approach and its application to surface organization in range data

Presents an autonomous, statistically robust, sequential function approximation approach to simultaneous parameterization and organization of (possibly partially occluded) surfaces in noisy, outlier-ridden (not Gaussian), functional range data. At the core of this approach is the Robust Sequential Estimator, a robust extension to the method of sequential least squares. Unlike most existing surface characterization techniques, the authors' method generates complete surface hypotheses in parameter space. Given a noisy depth map of an unknown 3-D scene, the algorithm first selects appropriate seed points representing possible surfaces. For each nonredundant seed it chooses the best approximating model from a given set of competing models using a modified Akaike Information Criterion. With this best model, each surface is expanded from its seed over the entire image, and this step is repeated for all seeds. Those points which appear to be outliers with respect to the model in growth are not included in the (possibly disconnected) surface. Point regions are deleted from each newly grown surface in the prune stage. Noise, outliers, or coincidental surface alignment may cause some points to appear to belong to more than one surface. These ambiguities are resolved by a weighted voting scheme within a 5/spl times/5 decision window centered around the ambiguous point. The isolated point regions left after the resolve stage are removed and any missing points in the data are filled by the surface having a majority consensus in an 8-neighborhood. >