Interactive state-transition diagrams for visualization of multimodal annotation

Abstract

Multimodal Analysis is a branch of Social Semiotics that studies how different resources, such as language, gesture, music, imagery, cinematography, etc. are used in modern media, and by humans in general, to construct and communicate meaning. Though theoretical aspects of multimodal analysis are w ell developed in the literature, and there is progress in the development of annotation software, the interpretation of annotated data still rarely uses algorithmic and data visualization approaches, limiting practical research in multimodal analysis. In the current work we propose how time stamped, tier-based multimodal annotations can be seen as finite state automatons and visualized as state-transition diagrams. We propose an algorithm converting time stamped annotation into a state-transition diagram and discuss how the resulting state-transition diagram can be visualized using freely available graph visualization tools. Then, we discuss common drawbacks of the existing tools, and propose an interactive visualization software tool allowing the analyst to synchronize the annotation state-transition diagram with the original media file. Finally, a case study on how the proposed visualization tools can assist media researchers in detecting patterns in multimodal analyses of news videos is presented.