Deep Learning for Natural Image Reconstruction from Electrocorticography Signals

Abstract

Several recent studies proposed various methods for reconstructing natural images from human functional magnetic resonance imaging (fMRI) data. However, few studies have proposed reconstruction methods for electrophysiolgical brain activities such as electroencephalography (EEG) and electrocorticography (ECoG). To investigate whether natural images can be reconstructed from electrophysiological brain activities, we conducted a large-scale experiment on natural image reconstruction from ECoG signals using deep learning. We first recorded ECoG signals from two macaque monkeys while presenting diverse natural images. Then, we trained several deep learning models for reconstructing presented images from ECoG signals. Comparing reconstruction models, we find that models trained with an adversarial loss produced reconstructions that contain visible features in presented images. Furthermore, our results with downsampled ECoG signals show the importance of rich temporal dynamics in ECoG signals for image reconstruction. Our results indicate the possibility of reconstructing diverse natural images from electrophysiological brain activities using deep learning.