DSR-Diff: Depth map super-resolution with diffusion model

Abstract

Color-guided depth map super-resolution (CDSR) improves the spatial resolution of a low-quality depth map with the corresponding high-quality color map, benefiting various applications such as 3D reconstruction, virtual reality, and augmented reality. While conventional CDSR methods typically rely on convolutional neural networks or transformers, diffusion models (DMs) have demonstrated notable effectiveness in high-level vision tasks. In this work, we present a novel CDSR paradigm that utilizes a diffusion model within the latent space to generate guidance for depth map super-resolution. The proposed method comprises a guidance generation network (GGN), a depth map super-resolution network (DSRN), and a guidance recovery network (GRN). The GGN is specifically designed to generate the guidance while managing its compactness. Additionally, we integrate a simple but effective feature fusion module and a transformer-style feature extraction module into the DSRN, enabling it to leverage the guiding prior in the extraction, fusion, and reconstruction of multi-model images. Taking into account both accuracy and efficiency, our proposed method has shown superior performance in extensive experiments compared to state-of-the-art methods. Specifically, our proposed DSR-Diff demonstrates remarkable improvements of 18.7% and 15.9% in terms of RMSE at scales 8 and 16 on the NYU v2 dataset while saving nearly 30% of the runtime, compared to existing SOTA method. Our codes will be made available at https://github.com/shiyuan7/DSR-Diff.