Depth Inpainting Research Articles

MCD-Net: Toward RGB-D Video Inpainting in Real-World Scenes.

Video inpainting gains an increasing amount of attention ascribed to its wide applications in intelligent video editing. However, despite tremendous progress made in RGB video inpainting, the existing RGB-D video inpainting models are still incompetent to inpaint real-world RGB-D videos, as they simply fuse color and depth via explicit feature concatenation, neglecting the natural modality gap. Moreover, current RGB-D video inpainting datasets are synthesized with homogeneous and delusive RGB-D data, which is far from real-world application and cannot provide comprehensive evaluation. To alleviate these problems and achieve real-world RGB-D video inpainting, on one hand, we propose a Mutually-guided Color and Depth Inpainting Network (MCD-Net), where color and depth are reciprocally leveraged to inpaint each other implicitly, mitigating the modality gap and fully exploiting cross-modal association for inpainting. On the other hand, we build a Video Inpainting with Depth (VID) dataset to supply diverse and authentic RGB-D video data with various object annotation masks to enable comprehensive evaluation for RGB-D video inpainting under real-world scenes. Experimental results on the DynaFill benchmark and our collected VID dataset demonstrate our MCD-Net not only yields the state-of-the-art quantitative performance but successfully achieves high-quality RGB-D video inpainting under real-world scenes. All resources are available at https://github.com/JCATCV/MCD-Net.

SAM-Net: LiDAR Depth Inpainting for 3D Static Map Generation

Various sensors can be attached and added to autonomous vehicles, included visual cameras, radar, LiDAR (Light Detection And Ranging), and GNSS (Global Navigation Satellite System). These sensors have been studied in many research areas, in particular studies on building precise 3D maps. It is essential for autonomous driving to create an accurate 3D map of the surrounding scene. However, creating an accurate static 3D map is difficult due to changes in moving objects or dynamic environments. Spurious objects on the 3D map can be handled by removing or ignoring them for 3D mapping. Following this idea, we propose an object segmentation and inpainting network. The proposed network called SAM-Net, addresses the object duplication issue by segmenting the objects and inpainting them with the segmentation results. Conventional inpainting research has dealt with RGB images. No matter how well such approaches reconstruct holes or corrupted images, they do not establish 3D points’ relationship with the point cloud frame. Therefore, we suggest a depth inpainting method for outdoor object segmentation and inpainting tasks that utilizes a high-precision depth range sensor (Velodyne HDL-64E), which is not suggested before. Unfortunately, no dataset exists for the outdoor depth inpainting task. Thus, to train our model, we generate a new dataset by locating objects on a clean static background. Moreover, our proposed method shows outstanding depth performance compared to the previous visual inpainting method. Our dataset will be available at: “ <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JunhyeopLee/lidar_inpainting</uri> ”.

Joint self-supervised and reference-guided learning for depth inpainting

Depth information can benefit various computer vision tasks on both images and videos. However, depth maps may suffer from invalid values in many pixels, and also large holes. To improve such data, we propose a joint self-supervised and reference-guided learning approach for depth inpainting. For the self-supervised learning strategy, we introduce an improved spatial convolutional sparse coding module in which total variation regularization is employed to enhance the structural information while preserving edge information. This module alternately learns a convolutional dictionary and sparse coding from a corrupted depth map. Then, both the learned convolutional dictionary and sparse coding are convolved to yield an initial depth map, which is effectively smoothed using local contextual information. The reference-guided learning part is inspired by the fact that adjacent pixels with close colors in the RGB image tend to have similar depth values. We thus construct a hierarchical joint bilateral filter module using the corresponding color image to fill in large holes. In summary, our approach integrates a convolutional sparse coding module to preserve local contextual information and a hierarchical joint bilateral filter module for filling using specific adjacent information. Experimental results show that the proposed approach works well for both invalid value restoration and large hole inpainting.

InDepth

Mobile Augmented Reality (AR) demands realistic rendering of virtual content that seamlessly blends into the physical environment. For this reason, AR headsets and recent smartphones are increasingly equipped with Time-of-Flight (ToF) cameras to acquire depth maps of a scene in real-time. ToF cameras are cheap and fast, however, they suffer from several issues that affect the quality of depth data, ultimately hampering their use for mobile AR. Among them, scale errors of virtual objects - appearing much bigger or smaller than what they should be - are particularly noticeable and unpleasant. This article specifically addresses these challenges by proposing InDepth, a real-time depth inpainting system based on edge computing. InDepth employs a novel deep neural network (DNN) architecture to improve the accuracy of depth maps obtained from ToF cameras. The DNN fills holes and corrects artifacts in the depth maps with high accuracy and eight times lower inference time than the state of the art. An extensive performance evaluation in real settings shows that InDepth reduces the mean absolute error by a factor of four with respect to ARCore DepthLab. Finally, a user study reveals that InDepth is effective in rendering correctly-scaled virtual objects, outperforming DepthLab.

Implementation of four kriging models for depth inpainting

Image inpainting is a process of reconstructing the missing pixels by inferencing information from the known part of an image. This paper aims to increase the precision of depth inpainting by proposing four models based on kriging techniques. Our four kriging models are based on two different semivariance models (exponential model and spherical model) and two different color-similarity functions. The inpainting algorithm is designed to extract both color and depth information of RGB-D images. For efficiency assessment, we look at Root Mean Square Error (RMSE), Structural Similarity Index Measure (SSIM), and Peak Signal to Noise Ratio (PSNR) of the reconstructed images. We then make a performance comparison with the previous six conventional methods. Finally, we show that our implemented models outperform the conventional ones. The accuracy of our four kriging models is competitive, and the PSNR values lie between 30.81 to 45.46 dB.

Computer catwalk

A multi-modal deep network for the segmentation of RGB-D images of clothes A group of researchers working in Belgium present a deep-learning based method for the segmentation of clothing worn by RGB-D human models. Over 50,000 samples of characters wearing different clothing styles were used as the dataset for the method, with the characters exhibiting different poses and existing in different environments giving a total of 9 semantic classes. A novel multi-modal encoder-decoder convolutional neural network (CNN) is used to segment and identify the clothing. Computer vision uses clothing segmentation to identify various clothing according to individual pixels being matched with specific garments from a database. This is useful for a variety of reasons, including automatic product tagging and ‘virtual changing rooms’ which can both enhance customer experience by virtue of better product suggestion and visualisation of how those products would look when worn by the customer. Block schematic of the proposed network architecture. Samples from the proposed RGB-D dataset. There have already been several works focusing on the segmentation of clothing, and it is well known that CNNs provide excellent performance when carrying out labelling tasks involving large volumes of information. However, until now, these methods have all relied on RGB images only, made up additively of red, green and blue light. The work presented in Electronics Letters finds its originality in being an image segmentation method using RGB-D, a combination of an RGB image with its corresponding depth image. By incorporating the depth of an image into their methodology, the authors hope to increase the segmentation accuracy in image regions containing difficult textures or shadows. There are a multitude of clothing segmentation datasets available already, but unfortunately these could not be used as they are RGB only, without the corresponding depth image. Thus, the research team had to first develop a new dataset made entirely of RGB-D images, yielding one of the two contributions reported in the Letter. Joukovsky et al. developed a data-generation pipeline enabling low-cost production RGB, depth and ground-truth label maps on a large scale. More than 50,000 3D-rendered samples of characters in various poses, environments and randomised clothing styles have been produced, with 9 categories defined. This data-generation pipeline was then coupled with the team's second contribution in the Letter, the Multi-modal Deeplab v3+, a deep neural network that can perform multi-modal semantic segmentation, behaving as a RGB-D extension of the pre-existing Deeplab v3+ baseline. The network is formed of a 2-encoder-1-decoder set-up with feature fusion modules. The model was trained and quantitatively evaluated using the previously mentioned data-generation pipeline. The network uses multi-modal fusion (MMF) combined with an atrous spatial pyramidal pooling (ASPP) module. This so-called MMF-ASPP efficiently merges the RBG and depth modalities, giving rise to a 3.4% mean intersection-over-union score improvement over MMF alone, showing that inclusion of image depth can lead to improvements in performance. The network was also tested on data gathered from a Kinect v2, where the depth input had been pre-processed using a depth inpainting method. The network demonstrated good generalisation to multi-subject inputs and varying postures, even when trained on single-subject images. This could prove to be interesting for future works. The research group has shown that their neural network can efficiently and more effectively segment clothing from a humanoid figure by using depth image data for an RGB image. They have also produced a data-generation pipeline of RGB-D images which can be used by future research groups to generate RGB-D images more easily. The pipeline could be improved by extending the proposed dataset with images including occlusions and noisy depth data, which could then be used to more extensively train the neural network. With some additions, the data-generation pipeline could also be used to generate pose-estimation datasets for further research.

Exemplar-based depth inpainting with arbitrary-shape patches and cross-modal matching

Commodity RGB-D cameras can provide texture and depth maps in real-time, and thus have facilitated the booming development of various depth-dependent applications. However, depth maps suffer from the loss of valid values, which leads to holes and impairs both research and applications. In this paper, we propose a novel exemplar based method to fill depth holes and thus to improve depth quality. This novel method is based on the fact that a depth map has many similar even identical parts, and the lost depth values can be restored by referring to valid ones. Considering the intrinsic property of depth maps, i.e., the sharpness of object boundaries, we propose to use arbitrary-shape matching patches, instead of fixed squares, to avoid inter-depth-layer distortion and thus improve the boundary. In addition, since depth values do not have distinct features, cross-modal matching, where both depth and texture are involved, is utilized. Moreover, we also investigate the similarity criteria in cross-modal matching, in order to improve the accuracy between the source patch and the target patch. Experimental results demonstrate that the proposed method can accurately recover lost depth information, especially at boundaries, which outperforms state-of-the-art exemplar-based inpainting methods.

Depth map inpainting via sparse distortion model

The depth map captured from a real scene by the Kinect motion sensor is always influenced by noise and other environmental factors. As a result, some depth information is missing from the map. This distortion of the depth map directly deteriorates the quality of the virtual viewpoints rendered in 3D video systems. We propose a depth map inpainting algorithm based on a sparse distortion model. First, we train the sparse distortion model using the distortion and real depth maps to obtain two learning dictionaries: one for distortion and one for real depth maps. Second, the sparse coefficients of the distortion and the real depth maps are calculated by orthogonal matching pursuit. We obtain the approximate features of the distortion from the relationship between the learning dictionary and the sparse coefficients of the distortion map. The noisy images are filtered by the joint space structure filter, and the extraction factor is obtained from the resulting image by the extraction factor judgment method. Finally, we combine the learning dictionary and sparse coefficients from the real depth map with the extraction factor to repair the distortion in the depth map. A quality evaluation method is proposed for the original real depth maps with missing pixels. The proposed method achieves better results than comparable methods in terms of depth inpainting and the subjective quality of the rendered virtual viewpoints.

Learning Joint Intensity-Depth Sparse Representations

This paper presents a method for learning overcomplete dictionaries of atoms composed of two modalities that describe a 3D scene: 1) image intensity and 2) scene depth. We propose a novel joint basis pursuit (JBP) algorithm that finds related sparse features in two modalities using conic programming and we integrate it into a two-step dictionary learning algorithm. The JBP differs from related convex algorithms because it finds joint sparsity models with different atoms and different coefficient values for intensity and depth. This is crucial for recovering generative models where the same sparse underlying causes (3D features) give rise to different signals (intensity and depth). We give a bound for recovery error of sparse coefficients obtained by JBP, and show numerically that JBP is superior to the group lasso algorithm. When applied to the Middlebury depth-intensity database, our learning algorithm converges to a set of related features, such as pairs of depth and intensity edges or image textures and depth slants. Finally, we show that JBP outperforms state of the art methods on depth inpainting for time-of-flight and Microsoft Kinect 3D data.

The rhythm of thought: art, literature, and music after Merleau-Ponty

Between present and past, visible and invisible, and sensation and idea, there is resonance - so philosopher Maurice Merleau-Ponty argued and so Jessica Wiskus explores in The Rhythm of Thought. Holding the poetry of Stephane Mallarme, the paintings of Paul Cezanne, the prose of Marcel Proust, and the music of Claude Debussy under Merleau-Ponty's phenomenological light, she offers innovative interpretations of some of these artists' masterworks, in turn articulating a new perspective on Merleau-Ponty's philosophy. More than merely recovering Merleau-Ponty's thought, Wiskus thinks according to it. First examining these artists in relation to noncoincidence - as silence in poetry, depth in painting, memory in literature, and rhythm in music - she moves through an array of their artworks toward some of Merleau-Ponty's most exciting themes: our bodily relationship to the world and the dynamic process of expression. She closes with an examination of synesthesia as an intertwining of internal and external realms and a call, finally, for philosophical inquiry as a mode of artistic expression. Structured like a piece of music itself, The Rhythm of Thought offers new contexts in which to approach art, philosophy, and the resonance between them.

An improved edge detection algorithm for depth map inpainting

Three-dimensional (3D) measurement technology has been widely used in many scientific and engineering areas. The emergence of Kinect sensor makes 3D measurement much easier. However the depth map captured by Kinect sensor has some invalid regions, especially at object boundaries. These missing regions should be filled firstly. This paper proposes a depth-assisted edge detection algorithm and improves existing depth map inpainting algorithm using extracted edges. In the proposed algorithm, both color image and raw depth data are used to extract initial edges. Then the edges are optimized and are utilized to assist depth map inpainting. Comparative experiments demonstrate that the proposed edge detection algorithm can extract object boundaries and inhibit non-boundary edges caused by textures on object surfaces. The proposed depth inpainting algorithm can predict missing depth values successfully and has better performance than existing algorithm around object boundaries.

Guided depth enhancement via a fast marching method

Range imaging sensors such as Kinect and time-of-flight cameras can produce aligned depth and color images in real time. However, the depth maps captured by such sensors contain numerous invalid regions and suffer from heavy noise. These defects more or less influence the use of depth information in practical applications. In order to enhance the depth maps, this paper proposes a new inpainting approach based on the fast marching method (FMM). We extend the inpainting model and the propagation strategy of FMM to incorporate color information for depth inpainting. An edge-preserving guided filter is further applied for noise reduction. To validate our algorithm, we perform experiments on both Kinect data and Middlebury dataset which, respectively, provide qualitative and quantitative results. Meanwhile, we also compare it to the original FMM and other two state-of-the-art depth enhancement methods. Experimental results show that our method performs better than the local methods in terms of both visual and metric qualities, and it achieves visually comparable results to the time-consuming global method.

Depth inpainting by tensor voting

Depth maps captured by range scanning devices or by using optical cameras often suffer from missing regions due to occlusions, reflectivity, limited scanning area, sensor imperfections, etc. In this paper, we propose a fast and reliable algorithm for depth map inpainting using the tensor voting (TV) framework. For less complex missing regions, local edge and depth information is utilized for synthesizing missing values. The depth variations are modeled by local planes using 3D TV, and missing values are estimated using plane equations. For large and complex missing regions, we collect and evaluate depth estimates from self-similar (training) datasets. We align the depth maps of the training set with the target (defective) depth map and evaluate the goodness of depth estimates among candidate values using 3D TV. We demonstrate the effectiveness of the proposed approaches on real as well as synthetic data.

Gentle ICM energy minimization for Markov random fields with smoothness-based priors

Coordinate descent, also known as iterated conditional mode (ICM) algorithm, is a simple approach for minimizing the energy defined by a Markov random field. Unfortunately, the ICM is very sensitive to the initial values and usually only finds a poor local minimum of the energy. A few modifications of the ICM algorithm are discussed here that ensure a more `gentle' descent during the first iterations of the algorithm and that lead to substantial performance improvements. It is demonstrated that the modified ICM can be competitive to other optimization algorithms on a set of vision problems such as stereo depth estimation, image segmentation, image denoising and inpainting.

Beneath Platonism

It is perhaps appropriate that much of Merleau-Ponty's late thinking comes down to us not in the form of narrative but in the rough outline of notes.1 For it is as if the late Merleau-Ponty deliberately employs language in such a way that his words work not so much to convey an explicit meaning but to articulate an empty space upon the page-as the space beneath thought, as the space that remains to be thought. "It is necessary," Merleau-Ponty writes, "to excavate below ideal identity, Bedeutung, Platonism, essence as given unity of the individual, of the world, and of history" (HP 16). For what the philosopher is called upon to investigate, according to Merleau-Ponty, is not the fixity of the word (nor the fixity of the idea), but rather the viscous link between words. In this way, the thinking that remains for us, through Merleau-Ponty's late sketches, stands as close to the artistic as to the philosophical tradition. Indeed, Merleau-Ponty's persistent attention to the language of oppositional pairs (visible-invisible, sensible-ideal, present-past, activity-passivity) reveals more than a penchant for "ambiguity." Rather, through investigation of the space beneath, between, and behind articulated words, his writing works to disclose the initiation of philosophical thought as truly creative. Merleau-Ponty characterizes his project, in The Visible and the Invisible, as "an elucidation of philosophical expression itself . . . as the expression of what is before expression and sustains it from behind" (VI 167). Merleau-Ponty seeks to elucidate the very depth of expression. Indeed, he writes, in 1959, "philosophy is nothing other than the unconcealment of the depth dimension of all other activities" (HP, 18). And therefore it is of little surprise that Merleau-Ponty devotes a considerable portion of his lectures in 1960-1961 (L'ontologie cartesienne et l'ontologie d'aujourd'hui) to the interrogation of depth within different styles of "non-philosophy"-particularly painting and music. What distinguishes these last notes from those of his course given just two years previously (La philosophie aujourd'hui)-when Merleau-Ponty earlier explored the philosophical significance of painting and music-is precisely the emphasis upon the notion of depth within these two arts. Depth in Painting: Cezanne Because it is seeking depth, Cezanne's style of painting plays a significant role in Merleau-Ponty's later thinking. He writes, "The enigma [of depth] consists in the fact that I see things, each one in its place, precisely because they eclipse one another, and that they are rivals before my sight precisely because each one is in its own place. Their exteriority is known in their envelopment and their mutual dependence in their autonomy."2 This notion of depth demands philosophical attention because of its capacity to unite, in one sole gesture, otherwise contradictory elements: exteriority and envelopment, dependence and autonomy. As a relationship between oppositional (or "incompossible," as he writes) pairs, depth comes to be perceived not in itself but only through the presentation of these elements. That is to say, depth-despite our ability to describe its presence clearly among the situated objects on a painted canvas-does not show itself in the same way as a thing to be seen: it shows itself only through relationship, as (literally) the space between things. Moreover, it shows itself to the extent that the things themselves are obscured from view (as Merleau-Ponty writes, "because they eclipse one another"); depth shows itself as the other side-the unpresentable side-of things. Thus it is through investigation of the notion of depth that Merleau-Ponty can fairly claim that a painting "renders present to us what is absent" (EM 171). Indeed, as a dimension shown through the presence of absence, depth can be conceived neither solely as a positive idea not solely as a sensible thing; it constitutes, rather, a divergence-a "negativity that comes to the world" (VI 250). …

Knowledge of Space Perception and the Portrayal of Depth in Painting

When your Committee did me the honor of inviting me to give the Brodël Lecture it occurred to me that a suitable topic for discussion would be one that lay somehow at the joint boundary between your field and my own. Neither you nor I can claim full expertness at that boundary, but, perhaps together we can discover some logical structure, some intriguing vistas. The slow growth of scientific knowledge regarding space perception has in the past exerted profound influence upon the graphic arts, and artists have from time to time contributed both to the science and to the problems which the science has been required to elucidate. From the viewpoint of visual science the application of its discoveries in the plastic arts represents a field of applied technology. To the artist this technology is one of the several foundations upon which he constructs his visions of harmony. I believe that it is singularly appropriate to associate an inquiry into this technologic fundament of the graphic arts with the name of my old friend, Max Brodël, for he was constantly searching for new modes of portrayal of space, volume, and texture.

Knowledge of Space Perception and the Portrayal of Depth in Painting

When your Committee did me the honor of inviting me to give the Brodel Lecture it occurred to me that a suitable topic for discussion would be one that lay somehow at the joint boundary between your field and my own. Neither you nor I can claim full expertness at that boundary, but, perhaps together we can discover some logical structure, some intriguing vistas. The slow growth of scientific knowledge regarding space perception has in the past exerted profound influence upon the graphic arts, and artists have from time to time contributed both to the science and to the problems which the science has been required to elucidate. From the viewpoint of visual science the application of its discoveries in the plastic arts represents a field of applied technology. To the artist this technology is one of the several foundations upon which he constructs his visions of harmony. I believe that it is singularly appropriate to associate an inquiry into this technologic fundament of the graphic arts with the name of m...