Semi-supervised adversarial recognition of refined window structures for inverse procedural façade modelling

Abstract

Deep learning methods are typically data-hungry and require many labelled samples. Unfortunately, the amount of effort required to label the data has significantly hindered the application of deep learning methods, especially in 3D modelling tasks requiring heterogeneous samples. This paper proposes a semi-supervised adversarial recognition strategy embedded in the inverse procedural modelling engine to reduce data annotation costs for learning to model 3D façades. Beginning with textured level-of-details models, we use convolutional neural networks to recognise the types and estimate the parameters of windows from image patches. The window types and parameters are then assembled into the procedural grammar. A simple procedural engine is built inside off-the-shelf 3D modelling software, producing fine-grained window geometries. To obtain a useful model from a few labelled samples, we leverage a generative adversarial network to train the feature extractor in a semi-supervised manner. The adversarial training strategy exploits the unlabelled data to stabilise the training phase. Experiments using publicly available façade image datasets reveal that the proposed methods can improve classification accuracy and parameter estimation by approximately 10% and 50%, respectively, under the same network structure. In addition, performance gains are more pronounced when testing against unseen data featuring different façade styles.