Visual representation decoding from human brain activity using machine learning: A baseline study

Abstract

Visual representation decoding refers to the task of deciphering what a subject is seeing or visualizing by observing the brain state via neuroimaging. In recent years, there is an increasing interest towards tackling the aforementioned task through the use of machine learning approaches. This study provides an extensive evaluation that will serve as a baseline for visual representation decoding, by exploring a wide range of model configurations, feature representations and evaluation setups. In this way, this work lays the groundwork for developing more sophisticated and accurate decoding pipelines. The evaluation results suggest that neural networks provide, on average, the best performance, while choosing the most appropriate similarity metric for the class decoding process depends mostly on the task at hand. Finally, this work may also assist domain experts to gain high-level insights about the brain’s function, through several interesting observations, e.g., our findings hint brain regions that are dominant for specific tasks and back up related claims about potential correspondence of the cortical hierarchy with deep visual representations.