Reconstruction of natural images from evoked brain activity with a dictionary-based invertible encoding procedure

Abstract

Studies on visual encoding and reconstruction based on functional magnetic resonance imaging (fMRI) have inspired each other in recent years. However, as far as we know, there has not been any study that has achieved the reconstruction of natural stimuli by directly reversing an encoding model with strong interpretability; in other words, the interpretability of current decoding methods is weak. To solve this problem, we first design a reversible feature extraction method using Gabor wavelets and build a nonnegative sparse mapping between the features and brain activity, thus achieving visual encoding. Then, based on the mapping, we estimate the features from measured brain activity and reverse the feature extraction method step-by-step. In this process, we use dictionary-learning technology to explore the natural statistical structure of the features from the image database, thereby greatly reducing the negative impact of information loss and fMRI noise. Finally, the stimuli can be reconstructed from the estimated features by back-propagation. Because the encoding procedure is highly transparent, the reconstruction procedure obtained by reversing the encoding model is also highly interpretable. The experiments show that our encoding method can build effective voxel-wise models for early visual areas, and also show that the proposed method is capable of reconstructing the basic outline of the stimuli with low structural complexity.