Multimodal knowledge graph construction of Chinese traditional operas and sentiment and genre recognition

Abstract

The advancement of digital technologies promotes the documentation of traditional operas, leaving a large amount of data but in a state of fragmentation. Constructing a knowledge graph (KG) is an effective way to realize the knowledge integration and reduce fragmentation, which can help the public understand traditional operas. However, constructed KGs in cultural heritage are mainly unimodal, lacking the ability to give the public a comprehensive perception, especially when they do not have related in-depth knowledge. In this paper, we take Chinese nation-level traditional operas as an example and construct a traditional opera ontology OpeOnto including classes with deep semantics (topic, sentiment). Then, we adopt an OpeOnto-driven way to construct multimodal knowledge graph OpeMKG including images and music links from several data resources. Aimed at analysing sentiments in OpeMKG and the automatic genre recognition of works for the preparation of automatic updating of OpeMKG, we develop a novel unified sentiment and genre recognition model (SGRM) for traditional operas with multimodal fusion and multi-task learning (MTL). The proposed model is examined on the built dataset of traditional operas and experimental results demonstrate its superiority compared with several state-of-the-art baselines.