Reconstructing natural images from human fMRI by alternating encoding and decoding with shared autoencoder regularization

Abstract

Reconstructing the viewed natural images from the corresponding functional magnetic resonance imaging (fMRI) of human visual cortices is extremely difficult. Utilizing deep learning techniques, the quality of reconstructed images can be significantly improved. This however is subject to availability of sufficient number of pair samples, which is not currently the case. In this study, we propose to perform alternating encoding and decoding to realize the mutual promotion of both based on shared semi-supervised learning and accomplish better reconstruction of natural images from the corresponding fMRI voxels. In our proposed method, the encoder and decoder are used to respectively map the images to the fMRI voxels (visual encoding), and the fMRI voxels to the images (visual reconstruction). We argue that combining the encoder and decoder in different sequences can form two converse and shared autoencoders to regularize the supervised learning of both through the unsupervised learning of fMRI voxels and images. More importantly, we alternatingly train the encoder and decoder with shared autoencoder regularization. Here, the premise is that a better encoder can produce a better decoder, and vice versa. The pixel-level identification of the proposed method achieves up to 89.5%, which indicates at least 2% superiority and is the new state-of-the-art compared to the previous works in terms of the image reconstruction performance.