MonuMAI: Dataset, deep learning pipeline and citizen science based app for monumental heritage taxonomy and classification

Abstract

An important part of art history can be discovered through the visual information in monument facades. However, the analysis of this visual information, i.e, morphology and architectural elements, requires high expert knowledge. An automatic system for identifying the architectural style or detecting the architectural elements of a monument based on one image will certainly help improving our knowledge in art and history. Building such tool is challenging as some styles share architectural elements, the bad conservation state of some monuments and the noise included in the image itself. The aim of this paper is to introduce MonuMAI (Monument with Mathematics and Artificial Intelligence) framework. In particular, (i) we designed MonuMAI dataset rich with expert knowledge considering the proposed architectural styles taxonomy and key elements relationship, which allows addressing several tasks, e.g., monument style classification and architectural elements detection, (ii) we developed MonuMAI deep learning pipeline based on lightweight MonuNet architecture for monument style classification and MonuMAI Key Elements Detection (MonuMAI-KED) model, and (iii) we built citizen science based MonuMAI mobile app that uses the proposed MonuMAI deep learning pipeline trained on MonuMAI dataset for performing in real life conditions. Our experiments show that both MonuNet architecture and the detection model achieve very good results under real life conditions.