“Lets see what you think! Bayesian reconstruction of perceptual experiences from human brain activity”

Abstract

Summary form only given. Recent interest in brain-computer interfaces has pushed development of decoding models that aim to classify, identify or reconstruct visual stimuli directly from measured brain activity. Most decoding models are based on non-parametric algorithms such as SVM and do not exploit current computational models of visual processing. We have pioneered an alternative approach in which the decoding algorithm is inferred from one or more explicit visual processing (nonlinear filtering) models. In previous work we showed that our approach extracts far more information from functional MRI measurements than was generally believed possible. In this task I will describe a new Bayesian decoding model that can actually reconstruct natural images that were seen by an observer from brain activity measured using fMRI. The decoder combines three elements: (1) a structural encoding model that characterizes signals from early visual areas; (2) a semantic encoding model that characterizes signals from higher visual areas; and (3) appropriate priors that incorporate statistical information about the structure and semantics of natural scenes. By combining all these elements the decoder produces reconstructions that accurately reflect the distribution, structure and semantic category of the objects contained in the original image. These results help clarify how distinct representations in different parts of the brain can be combined to provided a coherent reconstruction of the visual world; they also highlight a potentially important role for prior knowledge in visual perception. Our Bayesian decoding framework can be generalized directly to permit reconstruction of other perceptual dimensions, and might facilitate reconstruction of subjective perceptual processes such as visual imagery and dreaming. In the future Bayesian decoding algorithms might form the basis of powerful new brain-reading technologies and brain-computer interfaces.