A storytelling framework based on multimedia knowledge graph using linked open data and deep neural networks

Abstract

Listen

Translate article

Automatic storytelling is a broad challenge in research contexts such as Natural Language Processing and Contend Based Image Analysis. Despite the considerable achievements of machine learning techniques in these research fields, combining different approaches to fill the gap between an automatic generated story and human handwriting is hard. This work proposes a novel storytelling framework in the Cultural Heritage domain. We developed our framework based on a Multimedia Knowledge Graph (MKG), a crucial point of our work. Furthermore, we populated our Multimedia Knowledge Graph with a focused crawler that employs deep learning techniques to recognise a multimedia object from web resources. Furthermore, we used a combined approach of deep learning techniques and Linked Open Data (LOD) to retrieve information about images and depicted figures using Instance Segmentation. The system has a dynamic, user-friendly interface that guides the user during the storytelling process. Finally, we evaluated the system from a qualitative and quantitative point of view.