Panorama-to-model registration through integration of image retrieval and semantic reprojection

Abstract

Registering images to a building information model is an effective approach of associating as-built component status to as-designed information. However, registering a single still image to a digital twin is difficult due to pose ambiguities caused by the limited field of view in images. Several recent studies have started focusing on leveraging panoramic images to address problems, such as partial occlusions, repetitive facility components and textureless views, which could cause registration failures in existing image-based registration workflows. Even though having a bigger field of view helps in locating where the image was taken, registering a panoramic image to a building information model is still challenging due to inconsistent visual appearances, such as model vs. image visual differences caused by temporary objects in a scene, lighting condition changes, and different levels of details. In this paper, we present a novel method that registers panoramic images to a digital twin in a hierarchical way: the proposed method first performs rough registration through image retrieval using semantic segmentation, then localizes the image with a fine registration through minimizing semantic reprojection errors. Compared to existing methods, the developed method has the following contributions: 1) semantic-based image retrieval makes the rough registration process robust to lighting condition changes, texture differences, and temporary objects, 2) semantic-based image retrieval allows for bi-directional queries between a model and images, 3) reprojection-based fine registration further reduces the localization error due to the dimension reduction of features during image retrieval. Though the proposed method is developed for panoramic images, it can be generalized to monocular images at the cost of localization accuracy. The developed method was evaluated on a real-world academic building and a synthetic dataset, and the results showed that the proposed method can localize a panorama in less than a second and achieve sub-meter level localization error.