An image reconstruction framework for characterizing initial visual encoding.

Ling-Qi Zhang,David H Brainard,Nicolas P Cottaris

doi:10.7554/elife.71132

Abstract

We developed an image-computable observer model of the initial visual encoding that operates on natural image input, based on the framework of Bayesian image reconstruction from the excitations of the retinal cone mosaic. Our model extends previous work on ideal observer analysis and evaluation of performance beyond psychophysical discrimination, takes into account the statistical regularities of the visual environment, and provides a unifying framework for answering a wide range of questions regarding the visual front end. Using the error in the reconstructions as a metric, we analyzed variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method is widely applicable to experiments and applications in which the initial visual encoding plays an important role.

Highlights

28 29 Visual perception begins at the retina, which takes sensory measurements of the light incident at the eyes
We show analyses that: 1) use image reconstruction error as an information metric to understand the retinal mosaic “design” problem, with one example examining the implications of different allocations of retinal cone types; 2) allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling and how this varies with eccentricity, as well as with different types of color deficiency; 3)
We provide an extended discussion of key findings, as well as of some interesting open questions and future directions. 594 First, we cast retinal mosaic design as a “likelihood design” problem

Summary

Introduction

28 29 Visual perception begins at the retina, which takes sensory measurements of the light incident at the eyes. 36 One approach to understanding the implications of such information loss is ideal observer analysis, which evaluates the optimal performance on psychophysical discrimination tasks This allows for quantification of the limits imposed by features of the initial visual encoding, as well as predictions of the effect of variation in these features (Geisler 1989, 2011). 53 It is generally accepted that the visual system has internalized the statistical regularities of natural scenes, so as to take advantage of these regularities for making perceptual inferences (Attneave 1954; Field 1987; Shepard 1987; Knill, Kersten, and Yuille 1996) This motivates interest in extending ideal observer analysis to apply to fully naturalistic input, while incorporating the statistical regularities of natural scenes (Burge 2020). Combine the image reconstruction approach with analysis of psychophysical discrimination, providing an way to incorporate into such analyses the assumption that our visual system takes into account the statistical regularities of natural images

75 Results

Discussion

Materials and Methods