Computational discrimination between natural images based on gaze during mental imagery

Xi Wang,James Hays,Sebastian Koch,Marc Alexa,Kenneth Holmqvist,Andreas Ley

doi:10.1038/s41598-020-69807-0

Abstract

When retrieving image from memory, humans usually move their eyes spontaneously as if the image were in front of them. Such eye movements correlate strongly with the spatial layout of the recalled image content and function as memory cues facilitating the retrieval procedure. However, how close the correlation is between imagery eye movements and the eye movements while looking at the original image is unclear so far. In this work we first quantify the similarity of eye movements between recalling an image and encoding the same image, followed by the investigation on whether comparing such pairs of eye movements can be used for computational image retrieval. Our results show that computational image retrieval based on eye movements during spontaneous imagery is feasible. Furthermore, we show that such a retrieval approach can be generalized to unseen images.

Highlights

When retrieving image from memory, humans usually move their eyes spontaneously as if the image were in front of them
Results suggest that eye movements while looking at nothing play a functional role during memory retrieval, and act as “spatial indices”[4,5,11,12,16] that may provide assistance in memorizing the spacial layout of a s cene[4,5,6,11]
In the end we looked at the results in relation to individual behavior and image content to investigate how they influence the accuracy of the retrieval task

Summary

Introduction

When retrieving image from memory, humans usually move their eyes spontaneously as if the image were in front of them. Results suggest that eye movements while looking at nothing play a functional role during memory retrieval, and act as “spatial indices”[4,5,11,12,16] that may provide assistance in memorizing the spacial layout of a s cene[4,5,6,11]. It has been argued whether such eye movements can facilitate memory retrieval as additional cues[17,18]. Eye movements during imagery are the only muscular activities that largely replicate the activities during actual perception

Objectives

Methods

Results

Discussion

Conclusion