Hyperrealistic neural decoding for reconstructing faces from fMRI activations via the GAN latent space

Thirza Dado,Umut Güçlü,Luca Ambrogioni,Marcel Van Gerven,Gabriëlle Ras,Yağmur Güçlütürk,Sander Bosch

doi:10.1038/s41598-021-03938-w

Thirza Dado, Umut Güçlü + Show 5 more

Open Access

https://doi.org/10.1038/s41598-021-03938-w

Copy DOI

Journal: Scientific Reports	Publication Date: Jan 7, 2022
Citations: 18	License type: open-access

Affiliation: Radboud University Nijmegen

Abstract

Neural decoding can be conceptualized as the problem of mapping brain responses back to sensory stimuli via a feature space. We introduce (i) a novel experimental paradigm that uses well-controlled yet highly naturalistic stimuli with a priori known feature representations and (ii) an implementation thereof for HYPerrealistic reconstruction of PERception (HYPER) of faces from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring fMRI data as participants perceive face images synthesized by the generator network of a GAN. We show that the latent vectors used for generation effectively capture the same defining stimulus properties as the fMRI measurements. As such, these latents (conditioned on the GAN) are used as the in-between feature representations underlying the perceived images that can be predicted in neural decoding for (re-)generation of the originally perceived stimuli, leading to the most accurate reconstructions of perception to date.

Highlights

Neural decoding can be conceptualized as the inverse problem of mapping brain responses back to sensory stimuli via a feature space[1]
We present a model instance of this paradigm for HYperrealistic reconstruction of PERception (HYPER) which elegantly integrates generative adversarial networks (GANs) in neural decoding of faces by combining the following components (Fig. 2):
Neural decoding of fMRI measurements via the GAN latent space has resulted in unprecedented reconstructions of perception

Summary

Introduction

Neural decoding can be conceptualized as the inverse problem of mapping brain responses back to sensory stimuli via a feature space[1] Such a mapping can be modeled as a composite function of linear and nonlinear transformations (Fig. 1). The systematic correspondence between various feature representations of discriminative task-optimized (supervised) deep neural networks and neural representations of sensory cortices are well-established[2,3,4,5,6,7] As such, exploiting this correspondence in neural decoding of visual perception has pushed the state-of-the-art forward[1] such as classification of perceived, imagined and dreamed object c ategories[8,9], and reconstruction of perceived natural images10,11, movies[12] and faces[13,14]. The feature-stimulus transformation entails information loss as the data need to be reconstructed from the predicted feature representations using an approximate inversion network, leading to a severe bottleneck to the maximum possible reconstruction quality (i.e., the noise ceiling)

Objectives

Methods

Results

Discussion

Conclusion