Depth Image Super-resolution Research Articles

TM-GAN: A Transformer-Based Multi-Modal Generative Adversarial Network for Guided Depth Image Super-Resolution

TM-GAN: A Transformer-Based Multi-Modal Generative Adversarial Network for Guided Depth Image Super-Resolution

Method of depth simulation imaging and depth image super-resolution reconstruction for a 2D/3D compatible CMOS image sensor.

This paper presents a depth simulation imaging and depth image super-resolution (SR) method for two-dimensional/three-dimensional compatible CMOS image sensors. A depth perception model is established to analyze the effects of depth imaging parameters and evaluate the real imaging effects. We verify its validity by analyzing the depth error, imaging simulation, and auxiliary physical verification. By means of the depth simulation images, we then propose a depth SR reconstruction algorithm to recover the low-resolution depth maps to the high-resolution depth maps in two types of datasets. With the best situation in depth accuracy kept, the root mean square error (RMSE) of Middlebury dataset images are 0.0156, 0.0179, and 0.0183m. The RMSE of RGB-D dataset images are 0.0223 and 0.0229m. Compared with other listed conventional algorithms, our algorithm reduces the RMSE by more than 16.35%, 17.19%, and 23.90% in the Middlebury dataset images. Besides, our algorithm reduces the RMSE by more than 9.71% and 8.76% in the RGB-D dataset images. The recovery effects achieve optimized results.

Improved Upsampling Based Depth Image Super-Resolution Reconstruction

Constrained by current sensing technology, depth camera only acquires a low-resolution depth image that does not meet actual requirements. To solve this problem, this paper take a divide-and-conquer strategy to synthesize a high-resolution depth image from a low-resolution range image under the guidance of a registered high-resolution color image. Initially, the depth image is divided into planar areas and edge regions. For different zones, we exploit different methods to interpolate the missing depths. At planar area, the linear interpolation method is employed to perform upsampling. At edge region, a segmentation-separation upsampling method is used to interpolate the missing values. Then the upsampling result are refined on the Depth CNN that is built in this paper. We conduct extensive experiments on the benchmark database and real world data with various upsampling rates to illustrate the upsampling ability of our method. The comparison with classical super-resolution algorithms demonstrates that our upsampling algorithm achieves the best quality with fewer artifacts and our depth CNN outperforms the most state-of-the-art methods in terms of qualitative and quantitative evaluations.

Memory-Augmented Deep Unfolding Network for Guided Image Super-resolution

Guided image super-resolution (GISR) aims to obtain a high-resolution (HR) target image by enhancing the spatial resolution of a low-resolution (LR) target image under the guidance of a HR image. However, previous model-based methods mainly take the entire image as a whole, and assume the prior distribution between the HR target image and the HR guidance image, simply ignoring many non-local common characteristics between them. To alleviate this issue, we firstly propose a maximum a posteriori (MAP) estimation model for GISR with two types of priors on the HR target image, i.e., local implicit prior and global implicit prior. The local implicit prior aims to model the complex relationship between the HR target image and the HR guidance image from a local perspective, and the global implicit prior considers the non-local auto-regression property between the two images from a global perspective. Secondly, we design a novel alternating optimization algorithm to solve this model for GISR. The algorithm is in a concise framework that facilitates to be replicated into commonly used deep network structures. Thirdly, to reduce the information loss across iterative stages, the persistent memory mechanism is introduced to augment the information representation by exploiting the Long short-term memory unit (LSTM) in the image and feature spaces. In this way, a deep network with certain interpretation and high representation ability is built. Extensive experimental results validate the superiority of our method on a variety of GISR tasks, including Pan-sharpening, depth image super-resolution, and MR image super-resolution. Code will be released at https://github.com/manman1995/pansharpening.

Depth Image Noise Reduction and Super-Resolution by Pixel-Wise Multi-Frame Fusion

Depth Image Noise Reduction and Super-Resolution by Pixel-Wise Multi-Frame Fusion

Depth map super-resolution based on edge-guided joint trilateral upsampling

Depth image super-resolution (DISR) is a significant yet challenging task. In this paper, we propose a novel edge-guided framework for color-guided DISR aiming at reducing the artifacts caused by the introduced color image. Considering the different view synthesis characteristics of texture and smooth regions in depth image, we propose that the edge and smooth regions of depth map should be reconstructed in different ways. In our framework, a novel joint trilateral filter is built firstly, which has two different modes: one for the pixels on the edges and the other for the pixels in the smooth regions. Secondly, in each filtering iteration during the whole upsampling process, we use the edge map updated by the upsampled depth map as a guidance to decide when to change the filter mode. Benefiting from the strategy, the high-resolution depth map reconstructed has less texture copying and contains sharp and smooth edges. Experimental results demonstrate the effectiveness of our approach over prior depth map upsampling works.

Depth image super-resolution using correlation-controlled color guidance and multi-scale symmetric network

Depth image super-resolution (DISR) is an effective solution to improve the quality of depth images captured by real world low-cost cameras. In this paper, we propose a multi-scale symmetric network with the correlation-controlled color guidance block (CCGB) for DISR. The proposed network consists of two multi-scale sub-networks to respectively provide guidance and estimate depth. A symmetric unit (SU), which is a mini-encoder-decoder structure with residual learning, is designed and used as a basic network atom. The encoder part in SU aims to extract essential features, while the decoder part works to restore edge details. The way the SU processes information matches well with the textureless and sharp-edge characteristics of depth images. The two sub-networks present a high-level symmetric structure connected by dense guidance links in between. Based on the correlation analyses between the two sub-networks, each guidance link will transfer information trough a CCGB designed to implement channel-wise re-weighting mechanism. The accurate color guidance from CCGB helps avoiding artifacts introduced by non-co-occurrence of depth discontinuities and color edges. Experimental results demonstrate the superiority of the proposed method over several state-of-the-art DISR works.

Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion.

In this paper, we propose a novel deep convolutional neural network to solve the general multi-modal image restoration (MIR) and multi-modal image fusion (MIF) problems. Different from other methods based on deep learning, our network architecture is designed by drawing inspirations from a new proposed multi-modal convolutional sparse coding (MCSC) model. The key feature of the proposed network is that it can automatically split the common information shared among different modalities, from the unique information that belongs to each single modality, and is therefore denoted with CU-Net, i.e., common and unique information splitting network. Specifically, the CU-Net is composed of three modules, i.e., the unique feature extraction module (UFEM), common feature preservation module (CFPM), and image reconstruction module (IRM). The architecture of each module is derived from the corresponding part in the MCSC model, which consists of several learned convolutional sparse coding (LCSC) blocks. Extensive numerical results verify the effectiveness of our method on a variety of MIR and MIF tasks, including RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus and multi-exposure image fusion.

Super resolution of single depth image based on multi-dictionary learning with edge feature regularization

Currently, the acquisition and application of depth images are attracting a lot of attention thanks to the rapid development of 3D video. However, the current depth cameras cannot obtain high quality depth images due to the limitations in the imaging system. In this paper, to address this issue, we propose a scheme for single depth image super resolution based on multi-dictionary learning with edge regularization model. In the training stage, we focus on the edge information that represents the structure of the image, and extract the edge part, the low frequency and high frequency parts from the high resolution depth image set respectively. After that, three dictionaries are learned for the three parts with the constraint of the same sparse representation. In the image synthesis stage, we employ an edge-preserving regularization model as a reconstruction constraint to preserve the sharp structure, and reconstruct the depth image via the dictionaries learned from the training stage. Experimental results show that our proposed method can achieve good results in edge preservation, and both PSNR and SSIM values of the reconstructed depth images are superior to the state-of-art methods.

Learned Dynamic Guidance for Depth Image Reconstruction.

The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate a high resolution RGB camera and exploit the statistical correlation of its data and depth. In recent years, both optimization-based and learning-based approaches have been proposed to deal with the guided depth reconstruction problems. In this paper, we introduce a weighted analysis sparse representation (WASR) model for guided depth image enhancement, which can be considered a generalized formulation of a wide range of previous optimization-based models. We unfold the optimization by the WASR model and conduct guided depth reconstruction with dynamically changed stage-wise operations. Such a guidance strategy enables us to dynamically adjust the stage-wise operations that update the depth image, thus improving the reconstruction quality and speed. To learn the stage-wise operations in a task-driven manner, we propose two parameterizations and their corresponding methods: dynamic guidance with Gaussian RBF nonlinearity parameterization (DG-RBF) and dynamic guidance with CNN nonlinearity parameterization (DG-CNN). The network structures of the proposed DG-RBF and DG-CNN methods are designed with the the objective function of our WASR model in mind and the optimal network parameters are learned from paired training data. Such optimization-inspired network architectures enable our models to leverage the previous expertise as well as take benefit from training data. The effectiveness is validated for guided depth image super-resolution and for realistic depth image reconstruction tasks using standard benchmarks. Our DG-RBF and DG-CNN methods achieve the best quantitative results (RMSE) and better visual quality than the state-of-the-art approaches at the time of writing. The code is available at https://github.com/ShuhangGu/GuidedDepthSR.

Depth Image Super Resolution for 3D Reconstruction of Oil Reflnery Buildings

Time-of-Flight (ToF) camera can collect the depth data of dynamic scene surface in real time, which has been applied to 3D reconstruction of refinery buildings. However; due to the limitations of sensor hardware, the resolution of the depth image obtained is very low, so it cannot meet the requirements of dense depth of scene in practical applications such as 3D reconstruction. Therefore, it is necessary to make a breakthrough in software and design a good algorithm to improve the resolution of depth image. We propose of an algorithm of depth image super-resolution by using fusion of multiple progressive convolution neural networks, which uses a context-based network fusion framework to fuse multiple different progressive networks, so as to improve individual network performance and efficiency while maintaining the simplicity of network training. Finally, we have carried out experiments on the public data set, and the experimental results show that the proposed algorithm has reached or even exceeded the most advanced algorithms at present.

RADAR: Robust Algorithm for Depth Image Super Resolution Based on FRI Theory and Multimodal Dictionary Learning

Depth image super-resolution is a challenging problem, since normally high upscaling factors are required (e.g., 16×), and depth images are often noisy. In order to achieve large upscaling factors and resilience to noise, we propose a Robust Algorithm for Depth imAge super Resolution (RADAR) that combines the power of finite rate of innovation (FRI) theory with multimodal dictionary learning. Given a low-resolution (LR) depth image, we first model its rows and columns as piece-wise polynomials and propose an FRI-based depth upscaling (FDU) algorithm to super-resolve the image. Then, the upscaled moderate quality (MQ) depth image is further enhanced with the guidance of a registered high-resolution (HR) intensity image. This is achieved by learning multimodal mappings from the joint MQ depth and HR intensity pairs to the HR depth, through a recently proposed triple dictionary learning (TDL) algorithm. Moreover, to speed up the super-resolution process, we introduce a new projection-based rapid upscaling (PRU) technique that pre-calculates the projections from the joint MQ depth and HR intensity pairs to the HR depth. Compared with the state-of-the-art deep learning-based methods, our approach has two distinct advantages: we need a fraction of training data but can achieve the best performance, and we are resilient to mismatches between training and testing datasets. The extensive numerical results show that the proposed method outperforms other state-of-the-art methods on either noise-free or noisy datasets with large upscaling factors up to 16× and can handle unknown blurring kernels well.

Depth image super-resolution reconstruction based on a modified joint trilateral filter.

Depth image super-resolution (SR) is a technique that uses signal processing technology to enhance the resolution of a low-resolution (LR) depth image. Generally, external database or high-resolution (HR) images are needed to acquire prior information for SR reconstruction. To overcome the limitations, a depth image SR method without reference to any external images is proposed. In this paper, a high-quality edge map is first constructed using a sparse coding method, which uses a dictionary learned from the original images at different scales. Then, the high-quality edge map is used to guide the interpolation for depth images by a modified joint trilateral filter. During the interpolation, some information of gradient and structural similarity (SSIM) are added to preserve the detailed information and suppress the noise. The proposed method can not only preserve the sharpness of image edge, but also avoid the dependence on database. Experimental results show that the proposed method is superior to some state-of-the-art depth image SR methods.

Simultaneous color-depth super-resolution with conditional generative adversarial networks

In this paper, color-depth conditional generative adversarial networks (CDcGAN) are proposed to resolve the problems of simultaneous color image super-resolution and depth image super-resolution in 3D videos. Firstly, a generative network is presented to leverage the mutual information of the low-resolution color image and low-resolution depth image so that they can enhance each other considering their geometric structural similarity in the same scene. Secondly, three auxiliary losses of data loss, total variation loss, and 8-connected gradient difference loss are introduced to train this generative network to ensure that the generated images are close to the real ones in addition to the adversarial loss. Finally, we study the CDcGAN and its variants. Experimental results show that the proposed approach can produce the high-quality color image and depth image from a pair of low-quality images, and it is superior to several other leading methods. Additionally, it has also been used to resolve the problems of concurrent image smoothing and edge detection, as well as the problem of HR-color-image-guided depth super-resolution to show the effectiveness and universality of the proposed method.

Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution.

Depth image super-resolution is a significant yet challenging task. In this paper, we introduce a novel deep color guided coarse-to-fine convolutional neural network (CNN) framework to address this problem. First, we present a datadriven filter method to approximate the ideal filter for depth image super-resolution instead of hand-designed filters. Based on large data samples, the filter learned is more accurate and stable for upsampling depth image. Second, we introduce a coarse-to-fine CNN to learn different sizes of filter kernels. In coarse stage, larger filter kernels are learned by CNN to achieve crude high-resolution depth image. As to fine stage, the crude high-resolution depth image is used as the input so that smaller filter kernels are learned to gain more accurate results. Benefit from this network, we can progressively recover the high frequency details. Third, we construct a color guidance strategy that fuses color difference and spatial distance for depth image upsampling. We revise the interpolated high-resolution depth image according to the corresponding pixels in highresolution color maps. Guided by color information, the depth of high-resolution image obtained can alleviate texture copying artifacts and preserve edge details effectively. Quantitative and qualitative experimental results demonstrate our state-of-the-art performance for depth map super-resolution.

Guided depth image reconstruction from very sparse measurements

Depth images captured from modern depth cameras generally suffer from low spatial resolution, noise, and missing regions. These kinds of images cannot be used directly in applications related to depth images, e.g., robot navigation, 3DTV, and augmented reality, which basically need high-resolution input images with no noise o missing regions to function properly. To address the problem of low spatial resolution, noise degradation, and missing regions in depth images, we propose methods based on a guidance color image for depth reconstruction (DR) from sparse depth inputs and depth image super-resolution (SR). We also suggest a scenario wherein these problems can be integrated and addressed simultaneously. Further, we also demonstrate applications of the proposed approach for depth image denoising and depth image inpainting. In our approach, the guidance color image is used for obtaining the segment cues by applying mean-shift (MS) or simple linear iterative clustering (SLIC) segmentation on it. These strong segment cues help in aiding the DR and SR problems by considering the corresponding segments in the input depth image, and estimate the unknown pixels by either plane fitting or median filling approaches. Furthermore, we explore both direct and pyramidal (hierarchical) approaches for SR and DR-SR for higher upsampling factor. As such, our approaches are relatively simpler than some of the contemporary methods, yet the experimental results of the proposed methods show superior performance as compared with some other state-of-the-art DR and SR methods.

WLDISR: Weighted Local Sparse Representation-Based Depth Image Super-Resolution for 3D Video System.

In this paper, we propose a Weighted Local sparse representation based Depth Image Super-Resolution (WLDISR) schemes aiming at improving the Virtual View Image (VVI) quality of 3D video system. Different from color images, depth images are mainly used to provide geometrical information in synthesizing VVI. Due to the view synthesis characteristics difference between textural structures and smooth regions of depth images, we divide the depth images into edge and smooth patches and learn two local dictionaries, respectively. Meanwhile, the weight term is derived and incorporated explicitly in the cost function to denote different importance of edge structures and smooth regions to the VVI quality. Then, local sparse representation and weighted sparse representation are jointly used in both dictionary learning and reconstruction phases in depth image super-resolution. Based on different optimizations on learning and reconstruction modules, three WLDISR schemes, WLDISR-D, WLDISR-R, and WLDISR-ALL, are proposed. Experimental results on 3D sequences demonstrate that the proposed WLDISR-D, WLDISR-R, and WLDISR-ALL schemes can achieve more than 1.9-, 2.03-, and 2.16-dB gains on average, respectively, in terms of the VVIs' quality, as compared with the state-of-the-art schemes. In addition, the visual quality of VVIs is also improved.

Depth image super-resolution based on joint sparse coding

This paper proposes a new approach to single depth image super-resolution (SR), based upon a novel joint sparse coding model. A low-resolution color is used as a guide in the SR process. Firstly, we introduce synthetic characteristic image patch to learn a joint dictionary from the low-resolution depth map as well as its corresponding low-resolution intensity image. Then, we derive the joint nonlocal center sparse representation model based on sparse coding and theoretical analysis. In reconstruction process, we use Bayesian interpretation approach to estimation the sparse code coefficients for each unknown HR image patch. Meanwhile, we use an iterative algorithm to solve the JSC model. In addition, we exploit image patch redundancy within and across different scales, produce visually pleasing results without extensive training on external database. Experimental results demonstrate that the proposed method outperforms favorably many current state-of-the-art depth map super-resolution approaches on both visual effects and objective image quality and underpin the validity of our proposed model.

Depth Image Super Resolution Based on Edge-Guided Method

Depth image super-resolution (SR) is a technique which can reconstruct a high-resolution (HR) depth image from a low-resolution (LR) depth image. Its purpose is to obtain HR details to meet the needs of various applications in computer vision. In general, conventional depth image SR methods often cause edges in the final HR image to be blurred or ragged. To solve this problem, an edge-guided method for depth image SR is presented in this paper. To get high-quality edge information, a pair of sparse dictionaries was applied to reconstruct edges of depth image. Then, with the guidance of these high-quality edges, a depth image was interpolated by using a modified joint bilateral filter. Edge-guided method can preserve the sharpness of edges and effectively avoid generating blurry and ragged edges when SR is performed. Experiments showed that the proposed method can get better results on both subjective and objective evaluation, and the reconstructed performance was superior to conventional depth image SR methods.

Micro-Lens-Based Matching for Scene Recovery in Lenslet Cameras.

Since a light-field camera is able to capture more information than a traditional camera, a lot of methods, such as depth estimation, image super-resolution, and view synthesis, are explored for recovering scene information. In this paper, we propose a novel framework for scene recovery based on lenslet-based light-field camera images. Instead of using traditional matching terms, we design a new micro-lens-based matching term to calculate structure information and recover several kinds of scene information simultaneously. On the one hand, inherent information in micro-lens images is selected to complement details in sub-aperture images. On the other hand, sub-aperture images are used to expand micro-lens images and synthesize new view images. A new micro-lens-based consistency metric is introduced for the matching term to handle occlusions in depth estimation and image reconstruction. The newly appeared and newly occluded areas in synthesized views are analyzed and recovered based on information from surrounding points. Experimental results show that the proposed depth estimation method outperforms state-of-the-art methods on both synthetic and lenslet-based light-field images, especially in low-texture and occlusion regions. Furthermore, the super-resolution and view synthesis methods are able to acquire view images with more details and less aliasing artifacts.