SAM-VQA: Supervised Attention-Based Visual Question Answering Model for Post-Disaster Damage Assessment on Remote Sensing Imagery

Abstract

Each natural disaster leaves a trail of destruction and damage that must be effectively managed to reduce its negative impact on human life. Any delay in making proper decisions at the post-disaster managerial level can increase human suffering and waste resources. Proper managerial decisions after any natural disaster rely on an appropriate assessment of damages using data-driven approaches, which are needed to be efficient, fast, and interactive. The goal of this study is to incorporate a deep interactive data-driven framework for proper damage assessment to speed up the response and recovery phases after a natural disaster. Hence, this paper focuses on introducing and implementing the Visual Question Answering (VQA) framework for post-disaster damage assessment based on drone imagery, namely Supervised Attention-Based VQA (SAM-VQA). In visual question answering, query-based answers from images regarding the situation in disaster-affected areas can provide valuable information for decision-making. Unlike other computer vision tasks, visual question answering is more interactive and allows one to get instant and effective scene information by asking questions in natural language from images. In this work, we present a VQA dataset and propose a novel supervised attention-based VQA framework (SAM-VQA) for post-disaster damage assessment on remote sensing images. Our model outperforms state-of-the-art attention-based VQA techniques, including Stacked Attention Networks (SAN) [1] and Multi-modal Factorized Bilinear (MFB) with Co-Attention [2]. Furthermore, our proposed model can derive appropriate visual attention based on questions to predict answers, making our approach trustworthy.