3D reconstruction of Borobudur reliefs from 2D monocular photographs based on soft-edge enhanced deep learning

Abstract

We propose a deep learning-based reconstruction method for three-dimensional (3D) cultural heritage objects, which are destroyed or disappear and thus are unavailable for 3D scanning to create digital archives. The proposed reconstruction method uses a single monocular photo and predicts its depth values based on a monocular depth estimation network. While most recent studies have focused on scenes with sharp (high-curvature) edges and large depth differences, such as indoor and outdoor scenes, the proposed method is designed to exert its performance on scenes with small depth differences and soft (low-curvature) edges. The key idea is to involve the soft edges as guidance information in the depth estimation network. We demonstrate the effectiveness of the proposed method by applying it to the hidden reliefs of the Borobudur temple, a UNESCO world heritage site. The hidden reliefs were buried behind stone walls and are no longer visible. The 3D information of the reliefs can only be estimated from their old monocular photos. We achieved 97% reconstruction accuracy for validation data. We also integrate the reconstructed hidden reliefs with 3D scanned data of the visible regions and execute see-through visualization for the integrated data. This visualization can simultaneously show us both visible regions and invisible regions of the temple.