Video2Entities: A computer vision-based entity extraction framework for updating the architecture, engineering and construction industry knowledge graphs

Abstract

Due to the decentralisation and complexity of knowledge in the architecture, engineering and construction (AEC) industry, the research on knowledge graphs (KGs) is still insufficient, and most of the research focuses on text-based KG structuring or updating. Entity extraction, a sub-task of knowledge extraction, is critical in general KG update approaches. While the mainstream approach for this task generally uses textual data, visual data is more readily available, more accurate and has a shorter update cycle than textual data. Therefore, this paper integrates zero-shot learning techniques with general KGs to present a novel framework called “video2entities” that can extract entities from videos to update the AEC KG. The framework combines the perceptual capabilities of computer vision with the cognitive capabilities of KG to improve the accuracy and timeliness of KG updates. Experimental results demonstrate that the framework can extract “new entities” from architectural decoration videos for AEC KG updates.