Understanding the relevance between text and images in disaster‐related social media posts

Abstract

ABSTRACTDisaster‐related social media posts are often attached with images, either relevant or irrelevant to the text. The relevance between text and attached images can be indicative of the quality of posts because relevant images bring enriched information beyond text. A relevance framework is proposed to depict four aspects of relevance between text and images. A dataset about Typhoon Mangkhut was collected from a microblog application in China. A random sample of 3,000 posts along with 6,217 images were independently annotated. In addition, an automated text and image similarity measure is introduced. Annotation results show that more images are relevant to text on the aspects of object and scene relevance than on literal and behavior relevance. Irrelevant images account for roughly 10% of all images. Text and image pairs with different relevance aspects show different similarity scores. The goal of this study is to improve our understanding of the relationship between text and images in disaster‐related social media posts, which have implications for applications based on text and images integration.