Raw Depth Map Research Articles

RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion.

Raw depth images captured in indoor scenarios frequently exhibit extensive missing values due to the inherent limitations of the sensors and environments. For example, transparent materials frequently elude detection by depth sensors; surfaces may introduce measurement inaccuracies due to their polished textures, extended distances, and oblique incidence angles from the sensor. The presence of incomplete depth maps imposes significant challenges for subsequent vision applications, prompting the development of numerous depth completion techniques to mitigate this problem. Numerous methods excel at reconstructing dense depth maps from sparse samples, but they often falter when faced with extensive contiguous regions of missing depth values, a prevalent and critical challenge in indoor environments. To overcome these challenges, we design a novel two-branch end-to-end fusion network named RDFC-GAN, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. The first branch employs an encoder-decoder structure, by adhering to the Manhattan world assumption and utilizing normal maps from RGB-D information as guidance, to regress the local dense depth values from the raw depth map. The other branch applies an RGB-depth fusion CycleGAN, adept at translating RGB imagery into detailed, textured depth maps while ensuring high fidelity through cycle consistency. We fuse the two branches via adaptive fusion modules named W-AdaIN and train the model with the help of pseudo depth maps. Comprehensive evaluations on NYU-Depth V2 and SUN RGB-D datasets show that our method significantly enhances depth completion performance particularly in realistic indoor settings.

Neural-ABC: Neural Parametric Models for Articulated Body With Clothes.

In this paper, we introduce Neural-ABC, a novel parametric model based on neural implicit functions that can represent clothed human bodies with disentangled latent spaces for identity, clothing, shape, and pose. Traditional mesh-based representations struggle to represent articulated bodies with clothes due to the diversity of human body shapes and clothing styles, as well as the complexity of poses. Our proposed model provides a unified framework for parametric modeling, which can represent the identity, clothing, shape and pose of the clothed human body. Our proposed approach utilizes the power of neural implicit functions as the underlying representation and integrates well-designed structures to meet the necessary requirements. Specifically, we represent the underlying body as a signed distance function and clothing as an unsigned distance function, and they can be uniformly represented as unsigned distance fields. Different types of clothing do not require predefined topological structures or classifications, and can follow changes in the underlying body to fit the body. Additionally, we construct poses using a controllable articulated structure. The model is trained on both open and newly constructed datasets, and our decoupling strategy is carefully designed to ensure optimal performance. Our model excels at disentangling clothing and identity in different shape and poses while preserving the style of the clothing. We demonstrate that Neural-ABC fits new observations of different types of clothing. Compared to other state-of-the-art parametric models, Neural-ABC demonstrates powerful advantages in the reconstruction of clothed human bodies, as evidenced by fitting raw scans, depth maps and images. We show that the attributes of the fitted results can be further edited by adjusting their identities, clothing, shape and pose codes. The dataset and trained parametric model will be available at https://ustc3dv.github.io/NeuralABC/.

Surface normal and Gaussian weight constraints for indoor depth structure completion

Raw depth maps captured by depth sensors generally contain missing contents due to glossy, transparent, and sparsity problems. Recent methods well completed flat regions of raw depth maps; however, ignored the accuracy of depth structures. In this paper, an effective depth structure completion method is developed to infer missing depth structures. First, a raw depth map is divided into flat regions and depth structures based on a structure prediction network. Second, two local features including surface normals and Gaussian weights are extracted from a reference RGB image to impose constraints on flat regions and depth structures, separately. Third, a kernel least-square module is adopted to handle the texture-copy artifacts problem. Finally, an iterative optimization model is developed by embedding the two constraints into a Markov random field. The cost function of the model comprises three terms, which limit data fidelity between completed depth map and raw depth map, smoothness of flat regions, and accuracy of depth structures, respectively. The proposed method is evaluated on four indoor datasets including Matterport3D, RealSense, ScanNet, and NYUv2, and compared with eight recent baselines. Quantitative results demonstrate that RMSE and MAE of completed depth maps are considerably reduced by 22.0% and 45.3%, respectively. Visual results show the superiority in completing depth structures and suppressing texture-copy artifacts. Generalization test verify the effectiveness on unseen datasets.

Depth Completion and Super-Resolution with Arbitrary Scale Factors for Indoor Scenes.

Depth sensing has improved rapidly in recent years, which allows for structural information to be utilized in various applications, such as virtual reality, scene and object recognition, view synthesis, and 3D reconstruction. Due to the limitations of the current generation of depth sensors, the resolution of depth maps is often still much lower than the resolution of color images. This hinders applications, such as view synthesis or 3D reconstruction, from providing high-quality results. Therefore, super-resolution, which allows for the upscaling of depth maps while still retaining sharpness, has recently drawn much attention in the deep learning community. However, state-of-the-art deep learning methods are typically designed and trained to handle a fixed set of integer-scale factors. Moreover, the raw depth map collected by the depth sensor usually has many depth data missing or misestimated values along the edges and corners of observed objects. In this work, we propose a novel deep learning network for both depth completion and depth super-resolution with arbitrary scale factors. The experimental results on the Middlebury stereo, NYUv2, and Matterport3D datasets demonstrate that the proposed method can outperform state-of-the-art methods.

CNN-based and DTW features for human activity recognition on depth maps

In this work, we present a new algorithm for human action recognition on raw depth maps. At the beginning, for each class we train a separate one-against-all convolutional neural network (CNN) to extract class-specific features representing person shape. Each class-specific, multivariate time-series is processed by a Siamese multichannel 1D CNN or a multichannel 1D CNN to determine features representing actions. Afterwards, for the nonzero pixels representing the person shape in each depth map we calculate statistical features. On multivariate time-series of such features we determine Dynamic Time Warping (DTW) features. They are determined on the basis of DTW distances between all training time-series. Finally, each class-specific feature vector is concatenated with the DTW feature vector. For each action category we train a multiclass classifier, which predicts probability distribution of class labels. From pool of such classifiers we select a number of classifiers such that an ensemble built on them achieves the best classification accuracy. Action recognition is performed by a soft voting ensemble that averages distributions calculated by such classifiers with the largest discriminative power. We demonstrate experimentally that on MSR-Action3D and UTD-MHAD datasets the proposed algorithm attains promising results and outperforms several state-of-the-art depth-based algorithms.

Blind wayfinding with physically-based liquid sounds

Abstract Translating visual representations of real environments into auditory feedback is one of the key challenges in the design of an electronic travel aid for visually impaired persons. Although the solutions currently available in the literature can lead to effective sensory substitution, high commitment to an extensive training program involving repetitive sonic patterns is typically required, undermining their use in everyday life. The current study explores a novel sensory substitution algorithm that extracts information from raw depth maps and continuously converts it into parameters of a naturally sounding, physically based liquid sound model describing a population of bubbles. This approach is tested in a simplified wayfinding experiment with 14 blindfolded sighted participants and compared against the most popular sensory substitution algorithm available in the literature – the vOICe (Meijer, 1992) – following a short-time training program. The results indicate a superior performance of the proposed sensory substitution algorithm in terms of navigation accuracy, intuitiveness and pleasantness of the delivered sounds compared to the vOICe algorithm. These results should be applied to the visually impaired population with caution.

Probability contour guided depth map inpainting and superresolution using non-local total generalized variation

This paper proposes an image-guided depth super-resolution framework to improve the quality of depth map captured by low-cost depth sensors, like the Microsoft Kinect. First, a contour-guided fast marching method is proposed to preprocess the raw depth map for recovering the missing data. Then, by using the non-local total generalized variation (NL-TGV) regularization, a convex optimization model is constructed to up-sample the preprocessed depth map to a high-resolution one. To preserve the sharpness of depth discontinuities, the color image and its multi-level segmentation information are utilized to assign the weights within the NL-TGV through a novel weight combining scheme. The texture energy from color image and local structure coherence around neighbor pixels in low-resolution depth map are applied to adjust the combination weights for further suppressing texture-transfer. Quantitative and qualitative evaluations of the proposed method on the Middlebury datasets and real-sensor datasets show the promising results in quality.

Unsupervised object region proposals for RGB-D indoor scenes

In this paper, we present a novel unsupervised framework for automatically generating bottom up class independent object candidates for detection and recognition in cluttered indoor environments. Utilizing raw depth map from active sensors such as Kinect, we propose a novel plane segmentation algorithm for dividing an indoor scene into predominant planar regions and non-planar regions. Based on this partition, we are able to effectively predict object locations and their spatial extensions. Our approach automatically generates object proposals considering five different aspects: Non-planar Regions (NPR), Planar Regions (PR), Detected Planes (DP), Merged Detected Planes (MDP) and Hierarchical Clustering (HC) of 3D point clouds. Object region proposals include both bounding boxes and instance segments. Our approach achieves very competitive results and is even able to outperform supervised state-of-the-art algorithms on the challenging NYU-v2 RGB-Depth dataset. In addition, we apply our approach to the most recently released large scale RGB-Depth dataset from Princeton University – “SUN RGBD”, which utilizes four different depth sensors. Its consistent performance demonstrates a general applicability of our approach.

Unified multi-lateral filter for real-time depth map enhancement

This paper proposes a unified multi-lateral filter to efficiently increase the spatial resolution of low-resolution and noisy depth maps in real-time. Time-of-Flight (ToF) cameras have become a very promising alternative to stereo-based range sensing systems as they provide depth measurements at a high frame rate. However, there are actually two main drawbacks that restrict their use in a wide range of applications; namely, their fairly low spatial resolution as well as the amount of noise within the depth estimation. In order to address these drawbacks, we propose a new approach based on sensor fusion. That is, we couple a ToF camera of low-resolution with a 2-D camera of higher resolution to which the low-resolution depth map will be efficiently upsampled. In this paper, we first review the existing depth map enhancement approaches based on sensor fusion and discuss their limitations. We then propose a unified multi-lateral filter that accounts for the inaccuracy of depth edges position due to the low-resolution ToF depth maps. By doing so, unwanted artefacts such as texture copying and edge blurring are almost entirely eliminated. Moreover, the proposed filter is configurable to behave as most of the alternative depth enhancement approaches. Using a convolution-based formulation and data quantization and downsampling, the described filter has been effectively and efficiently implemented for dynamic scenes in real-time applications. The experimental results show a sensitive qualitative as well as quantitative improvement on raw depth maps, outperforming state-of-the-art multi-lateral filters.

Depth map super-resolution using stereo-vision-assisted model

Abstract In this paper, we propose a novel Stereo-Vision-Assisted (SVA) model for depth map super-resolution. Given a low-resolution depth map as input, we recover a high-resolution depth map using the registered high-resolution color stereo image pair. First, based on the mutual benefits between raw depth map and features of high resolution color image, we model the relationship with two constraint terms of local and non-local priors and sufficiently explore their complementary nature. Moreover, by considering reliable disparity pixels calculated from stereo matching algorithm, we formulate a stereo disparity regularization term to further reinforce the preservation of fine depth detail. Hence, our SVA objective function includes a non-local prior constraint, a local prior constraint and a stereo disparity prior constraint. In addition, we employ an efficient algorithm to optimize the objective function. Experimental results demonstrate that our approach can obtain superior performance in comparison with other methods.

Depth map super-resolution via local and nonlocal priors

We propose a method for solving the depth map super-resolution problem. Given a low-resolution depth map as input, we recover a high-resolution depth map using a registered and potentially high-resolution color image. We formulate the super-resolution process as a regularization and optimization framework. Specifically, taking into account that discontinuities in depth and color tend to coalign, we obtain the local and nonlocal regularization terms based on the raw depth map and also the features of high-resolution color images. Different from conventional methods, we model the relationship with two constraint terms of local and nonlocal priors to sufficiently explore the complementary nature. Experimental results demonstrate the effectiveness of our approach, which produces sharper edges and more faithful details compared with other state-of-the-art strategies.

D.E. Anderson, D.M. Rings, Current Veterinary Therapy Food Animal Practice, 5th ed., Saunders Elsevier, 2008, ISBN9781416035916, 736 pp.; £110.00 (hard)

In this paper, we propose a novel Stereo-Vision-Assisted (SVA) model for depth map super-resolution. Given a low-resolution depth map as input, we recover a high-resolution depth map using the registered high-resolution color stereo image pair. First, based on the mutual benefits between raw depth map and features of high resolution color image, we model the relationship with two constraint terms of local and non-local priors and sufficiently explore their complementary nature. Moreover, by considering reliable disparity pixels calculated from stereo matching algorithm, we formulate a stereo disparity regularization term to further reinforce the preservation of fine depth detail. Hence, our SVA objective function includes a non-local prior constraint, a local prior constraint and a stereo disparity prior constraint. In addition, we employ an efficient algorithm to optimize the objective function. Experimental results demonstrate that our approach can obtain superior performance in comparison with other methods.