Self-supervised cross-modal visual retrieval from brain activities

Abstract

We study the problem of restoring visual stimuli from visually-evoked electroencephalography (EEG) signals. Using a supervised classification-then-generation framework, the reconstruction-based approaches learn the mapping between distributions of two modalities but fail to reproduce the exact visual stimulus. Instead, we propose a self-supervised cross-modal retrieval paradigm that seeks instance-level alignment by maximizing the mutual information between the EEG encoding and associated visual stimulus. We demonstrate the threefold advantages of the self-supervised retrieval over supervised reconstruction on the largest visual-evoked EEG dataset with two evaluation protocols. First, it restores the exact visual stimulus without accessing the image class information, which was not possible with previous approaches. Second, it produces more recognizable results than generated ones and bypasses the challenge of training an image generator. Finally, it illustrates the benefits of self-supervision over supervised models in handling open-set data.