Generic decoding of seen and imagined objects using hierarchical visual features

Tomoyasu Horikawa,Yukiyasu Kamitani

doi:10.1038/ncomms15037

Abstract

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

Highlights

Object recognition is a key function in both human and machine vision
We extracted feature values from object images using a total of 13 visual feature types/ layers (CNN1–8, HMAX1–3, GIST and SIFT þ BoF; B1,000 units for each feature type/layer)
Using the trained decoders, a feature vector was predicted from brain activity measured while seeing or imagining an object that was not used in decoder training

Summary

Introduction

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. Most previous studies have relied on a classification-based approach, where a statistical classifier (decoder) is trained to learn a relationship between fMRI patterns and the target contents to be decoded. Such approaches entail a fundamental constraint on the number of possible outputs. Recent studies have overcome this limitation by designing decoders for retinotopically organized, image-level features[14,15,16] This method enables the decoding of novel visual images not presented during training sessions. Kay et al.[14] built an encoding model consisting of retinotopically organized Gabor wavelet filters They used a visual image database and the predicted brain activities produced by an encoding model. We extended the modular decoding approach originally developed for visual image reconstruction[16] to the decoding of generic object categories

Objectives

Methods

Results

Conclusion