Future landscape visualization using a city digital twin: integration of augmented reality and drones with implementation of 3D model-based occlusion handling

Abstract

Abstract The integration of augmented reality and drones allows past and future landscapes to be visualized from an aerial perspective. However, these visualizations still suffer from the occlusion problem, where the three-dimensional (3D) virtual model displayed in the real world is in front of a real-world object. Currently, city digital twins are essential for the sustainable development of cities and the development of detailed 3D models of cities. By visualizing the city digital twin, augmented reality can facilitate the participation of nonexpert citizens in the decision-making process of urban design, but research examples are limited. Here, using detailed city 3D models, we develop a digital-twin approach to outdoor augmented reality with occlusion handling for both first-person and bird’s-eye views. In a verification experiment, the occlusion handling accuracy of the prototype system was measured to be about 0.8 using intersection over union. The frame rate of the entire prototype system was about 30 fps, and the delay between the controller and the augmented reality device was about 3 s. The internet-based system architecture was developed to integrate augmented reality and drone systems. Our system allows multiple stakeholders involved in building construction projects to observe aerial perspectives of those projects, both on-site and off-site via an internet browser, using augmented reality with occlusion handling.