Abstract
Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.
Highlights
Object recognition is a key function in both human and machine vision
We extracted feature values from object images using a total of 13 visual feature types/ layers (CNN1–8, HMAX1–3, GIST and SIFT þ BoF; B1,000 units for each feature type/layer)
Using the trained decoders, a feature vector was predicted from brain activity measured while seeing or imagining an object that was not used in decoder training
Summary
Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. Most previous studies have relied on a classification-based approach, where a statistical classifier (decoder) is trained to learn a relationship between fMRI patterns and the target contents to be decoded. Such approaches entail a fundamental constraint on the number of possible outputs. Recent studies have overcome this limitation by designing decoders for retinotopically organized, image-level features[14,15,16] This method enables the decoding of novel visual images not presented during training sessions. Kay et al.[14] built an encoding model consisting of retinotopically organized Gabor wavelet filters They used a visual image database and the predicted brain activities produced by an encoding model. We extended the modular decoding approach originally developed for visual image reconstruction[16] to the decoding of generic object categories
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.