Bayesian reconstruction of perceptual experiences from human brain activity

Abstract

Event Abstract Back to Event Bayesian reconstruction of perceptual experiences from human brain activity Sensory systems transform stimulus energy measured at the peripheral transducers into explicit representations of various abstract features. These transformations can be viewed as (nonlinear) mappings between the stimuli and brain activity measurements [1]. Most work in sensory neuroscience has focused on development of quantitative encoding models posed in terms of nonlinear mechanisms that filter the stimuli in order to produce appropriate responses. Recent interest in brain-machine interfaces has pushed development of decoding models that aim to classify, identify or reconstruct the stimulus directly from measured brain activity. These models work in the opposite direction, transforming responses into stimuli features. Most decoding models use non-parametric algorithms such as SVM, and are not based on explicit encoding models. We have pioneered an alternative approach in which the decoding algorithm is inferred from one or more explicit encoding models. In a study published last year [2] we showed that this approach can be used to extract far more information from functional MRI measurements than was generally believed possible. In more recent work along these lines, we have developed a new Bayesian decoding model that can reconstruct natural images seen by an observer from measured brain activity. The decoder combines three elements: a structural encoding model that characterizes signals from early visual areas; a semantic encoding model that characterizes signals from higher visual areas; and appropriate priors that incorporate statistical information about the structure and semantics of natural scenes. By combining all these elements the decoder produces reconstructions that accurately reflect the distribution, structure and semantic category of the objects contained in the original image. These results help clarify how distinct representations in different parts of the brain can be combined to provided a coherent reconstruction of the visual world; they also highlight a potentially important role for prior knowledge in visual perception. Our Bayesian decoding model can be generalized directly to permit reconstruction of other perceptual dimensions, such as color and motion. It might also be possible to use this framework to reconstruct subjective perceptual processes such as visual imagery and dreaming. More generally, these results suggest that Bayesian decoding algorithms could form the basis of powerful new brain-reading technologies and brain-computer interfaces. Acknowledgements. The functional MRI scans were conducted with the support of the staff at the Brain Imaging Center at UC Berkeley and at the Veterans Administration in Martinez, CA. This research was supported by the National Eye Institute and by UC Berkeley intramural funds.