Question Answering for Estimation of Seen Image Contents from Multi-subject fMRI Responses

Abstract

Estimating seen image contents based on the analysis of functional magnetic resonance imaging (fMRI) data has been actively researched. So far, it has been necessary to train a model with individual fMRI data and construct models for each task. In this paper, we propose an estimation method via Visual Question Answering (VQA) from fMRI data from multi-subjects. The task of VQA in the field of computer vision is generating answers when given an image and questions about its contents. The proposed method enables generation accurate answers via the VQA model when given fMRI data and questions about the seen images. Besides, we newly introduce training with multi-subject fMRI data since fMRI datasets are generally small due to the burden for subjects. Then we can realize the estimation of the contents of seen images from fMRI data of a subject whose data are not used in the training phase.