Visual Neuroscience Research Articles

Leveraging textured flickers: a leap toward practical, visually comfortable, and high-performance dry EEG code-VEP BCI.

Reactive Brain-Computer Interfaces (rBCIs) typically rely on repetitive visual stimuli, which can strain the eyes and cause attentional distraction. To address these challenges, we propose a novel approach rooted in visual neuroscience to design visual Stimuli for Augmented Response (StAR). The StAR stimuli consist of small, randomly-oriented Gabor or Ricker patches that optimize foveal neural response while reducing peripheral distraction.

Methods: In a factorial design study, 24 participants equipped with an 8-dry electrode EEG system focused on series of target flickers presented under three formats: traditional Plain flickers, Gabor-based, or Ricker-based flickers. These flickers were part of a five-class Code Visually Evoked Potentials (c-VEP) paradigm featuring low-frequency, short, and aperiodic visual flashes.

Results: Subjective ratings revealed that Gabor and Ricker stimuli were visually comfortable and nearly invisible in peripheral vision compared to plain flickers. Moreover, Gabor and Ricker-based textures achieved higher accuracy (93.6% and 96.3%, respectively) with only 88 seconds of calibration data, compared to plain flickers (65.6%). A follow-up online implementation of this experiment was conducted to validate our findings in naturalistic operations. During this trial, remarkable accuracies of 97.5% in a cued task and 94.3% in an asynchronous digicode task were achieved, with a mean decoding time as low as 1.68 seconds.

Conclusion: This work demonstrates the potential to expand BCI applications beyond the lab by integrating visually unobtrusive systems with gel-free, low-density EEG technology, thereby making BCIs more accessible and efficient. The datasets, algorithms, and BCI implementations are shared through open-access repositories.

Neuroscientific insights about computer vision models: a concise review.

The development of biologically-inspired computational models has been the focus of study ever since the artificial neuron was introduced by McCulloch and Pitts in 1943. However, a scrutiny of literature reveals that most attempts to replicate the highly efficient and complex biological visual system have been futile or have met with limited success. The recent state-of the-art computer vision models, such as pre-trained deep neural networks and vision transformers, may not be biologically inspired per se. Nevertheless, certain aspects of biological vision are still found embedded, knowingly or unknowingly, in the architecture and functioning of these models. This paper explores several principles related to visual neuroscience and the biological visual pathway that resonate, in some manner, in the architectural design and functioning of contemporary computer vision models. The findings of this survey can provide useful insights for building futuristic bio-inspired computer vision models. The survey is conducted from a historical perspective, tracing the biological connections of computer vision models starting with the basic artificial neuron to modern technologies such as deep convolutional neural network (CNN) and spiking neural networks (SNN). One spotlight of the survey is a discussion on biologically plausible neural networks and bio-inspired unsupervised learning mechanisms adapted for computer vision tasks in recent times.

Feature-specific divisive normalization improves natural image encoding for depth perception.

Vision science and visual neuroscience seek to understand how stimulus and sensor properties limit the precision with which behaviorally-relevant latent variables are encoded and decoded. In the primate visual system, binocular disparity-the canonical cue for stereo-depth perception-is initially encoded by a set of binocular receptive fields with a range of spatial frequency preferences. Here, with a stereo-image database having ground-truth disparity information at each pixel, we examine how response normalization and receptive field properties determine the fidelity with which binocular disparity is encoded in natural scenes. We quantify encoding fidelity by computing the Fisher information carried by the normalized receptive field responses. Several findings emerge from an analysis of the response statistics. First, broadband (or feature-unspecific) normalization yields Laplace-distributed receptive field responses, and narrowband (or feature-specific) normalization yields Gaussian-distributed receptive field responses. Second, the Fisher information in narrowband-normalized responses is larger than in broadband-normalized responses by a scale factor that grows with population size. Third, the most useful spatial frequency decreases with stimulus size and the range of spatial frequencies that is useful for encoding a given disparity decreases with disparity magnitude, consistent with neurophysiological findings. Fourth, the predicted patterns of psychophysical performance, and absolute detection threshold, match human performance with natural and artificial stimuli. The current computational efforts establish a new functional role for response normalization, and bring us closer to understanding the principles that should govern the design of neural systems that support perception in natural scenes.

From Cats to the Cortex: Unravelling the Hierarchical Processing System of Vision and Brain Plasticity.

The groundbreaking research conducted by neurophysiologists David Hubel and Torsten Wiesel during the late 1950s and 1960s revolutionized the field of visual neuroscience. Through single-unit recordings in the visual cortex of cats, they made several key discoveries that fundamentally changed our understanding of visual processing. Their work introduced the concept of orientation selectivity, revealing that neurons in the visual cortex are specifically tuned to line orientations, thereby illustrating how the brain constructs visual representations through edge detection. Additionally, they discovered ocular dominance columns, the specialized cortical regions that respond preferentially to input from one eye, providing crucial insights into the organization of visual processing and the importance of binocular vision. Hubel and Wiesel's research also established the concept of a critical period in visual development, demonstrating that early visual experiences are essential for the proper maturation of the visual system. This discovery has had significant implications for understanding neural plasticity and the role of sensory input in neural development. The impact of their work goes beyond theoretical knowledge, contributing to the development of therapeutic strategies for some visual disorders and guiding current research into brain plasticity and visual processing. This review synthesizes the monumental contributions of Hubel and Wiesel, evaluating how their key discoveries have shaped subsequent research in visual neuroscience. It traces the evolution of knowledge related to visual pathways, feature detection, and brain plasticity, highlighting the enduring influence of their foundational work on contemporary studies. By exploring the progression from their pioneering findings to modern advancements, this review emphasizes the legacy of Hubel and Wiesel's contributions to our understanding of vision and neural function.

Saccade size predicts onset time of object processing during visual search of an open world virtual environment

ObjectiveTo date the vast majority of research in the visual neurosciences have been forced to adopt a highly constrained perspective of the vision system in which stimuli are processed in an open-loop reactive fashion (i.e., abrupt stimulus presentation followed by an evoked neural response). While such constraints enable high construct validity for neuroscientific investigation, the primary outcomes have been a reductionistic approach to isolate the component processes of visual perception. In electrophysiology, of the many neural processes studied under this rubric, the most well-known is, arguably, the P300 evoked response. There is, however, relatively little known about the real-world corollary of this component in free-viewing paradigms where visual stimuli are connected to neural function in a closed-loop. While growing evidence suggests that neural activity analogous to the P300 does occur in such paradigms, it is an open question when this response occurs and what behavioral or environmental factors could be used to isolate this component. ApproachThe current work uses convolutional networks to decode neural signals during a free-viewing visual search task in a closed-loop paradigm within an open-world virtual environment. From the decoded activity we construct fixation-locked response profiles that enable estimations of the variable latency of any P300 analogue around the moment of fixation. We then use these estimates to investigate which factors best reduce variable latency and, thus, predict the onset time of the response. We consider measurable, search-related factors encompassing top-down (i.e., goal driven) and bottom-up (i.e., stimulus driven) processes, such as fixation duration and salience. We also consider saccade size as an intermediate factor reflecting the integration of these two systems. Main resultsThe results show that of these factors only saccade size reliably determines the onset time of P300 analogous activity for this task. Specifically, we find that for large saccades the variability in response onset is small enough to enable analysis using traditional ensemble averaging methods. SignificanceThe results show that P300 analogous activity does occur during closed-loop, free-viewing visual search while highlighting distinct differences between the open-loop version of this response and its real-world analogue. The results also further establish saccades, and saccade size, as a key factor in real-world visual processing.

What can cartographers learn from artistic paintings when stylizing maps? A preliminary synthesis from the perspective of visual neuroscience

ABSTRACT Creating a map style from scratch is challenging. Cartographers may find inspirations from artistic paintings and transfer their style to maps. However, the reference painting and the extracted style elements may not fit well with the content and function of maps, resulting in undesirable or even frustrating map stylization. The question of how to choose appropriate paintings as inspirations and utilize their essential visual elements to stylize maps remains unanswered. As neuroscience has developed, an increasing number of neural mechanisms underlying visual perception have been discovered, deepening our understanding of artistic styles and their adaptation to maps. In this paper, we draw connections between visual neurons and style elements of both artistic paintings and maps by examining seven neural mechanisms (i.e. central-peripheral vision, color-to-contour adhesion, double-opponent effects, visual filling-in, orientation selectivity, latency variation, and what-where separation) that are typical of artistic paintings. We also demonstrate that these mechanisms are transferable to maps. We then create a cartographic style syntactics to generally indicate how and to what degree these mechanisms can be reflected in four cartographic stylistic criteria (i.e. identification, expressiveness, aesthetic appeal and emotional impact). Our work contributes to the integration of principles from neuroscience with cartographic design to deepen our understanding of map perception and to facilitate map style transfer.

Inner limiting Membrane Peel Extends In vivo Calcium Imaging of Retinal Ganglion Cell Activity Beyond the Fovea in Non-Human Primate.

High resolution retinal imaging paired with intravitreal injection of a viral vector coding for the calcium indicator GCaMP has enabled visualization of activity dependent calcium changes in retinal ganglion cells (RGCs) at single cell resolution in the living eye. The inner limiting membrane (ILM) is a barrier for viral vectors, restricting transduction to a ring of RGCs serving the fovea in both humans and non-human primates (NHP). We evaluate peeling the ILM prior to intravitreal injection as a strategy to expand calcium imaging beyond the fovea in the NHP eye in vivo. Five Macaca fascicularis eyes (age 3-10y; n=3 individuals; 2M, 1F) underwent vitrectomy and 5 to 6-disc diameter ILM peel centered on the fovea prior to intravitreal delivery of 7m8:SNCG:GCaMP8s. Calcium responses from RGCs were recorded using a fluorescence adaptive optics scanning laser ophthalmoscope. In all eyes GCaMP was expressed throughout the peeled area, representing a mean 8-fold enlargement in area of expression relative to a control eye. Calcium recordings were obtained up to 11 degrees from the foveal center. RGC responses were comparable to the fellow control eye and showed no significant decrease over the 6 months post ILM peel, suggesting that RGC function was not compromised by the surgical procedure. In addition, we demonstrate that activity can be recorded directly from the retinal nerve fiber layer. This approach will be valuable for a range of applications in visual neuroscience including pre-clinical evaluation of retinal function, detecting vision loss, and assessing the impact of therapeutic interventions.

What Is a Visual Stream?

The dual stream model of the human and non-human primate visual systems remains Leslie Ungerleider's (1946-2020) most indelible contribution to visual neuroscience. In this model, a dorsal "where" stream specialized for visuospatial representation extends through occipitoparietal cortex, whereas a ventral "what" stream specialized for representing object qualities extends through occipito-temporal cortex. Over time, this model underwent a number of revisions and expansions. In one of her last scientific contributions, Leslie proposed a third visual stream specialized for representing dynamic signals related to social perception. This alteration invites the question: What is a visual stream, and how are different visual streams individuated? In this article, we first consider and reject a simple answer to this question based on a common idealizing visualization of the model, which conflicts with the complexities of the visual system that the model was intended to capture. Next, we propose a taxonomic answer that takes inspiration from the philosophy of science and Leslie's body of work, which distinguishes between neural mechanisms, pathways, and streams. In this taxonomy, visual streams are superordinate to pathways and mechanisms and provide individuation conditions for determining whether collections of cortical connections delineate different visual streams. Given this characterization, we suggest that the proposed third visual stream does not yet meet these conditions, although the tripartite model still suggests important revisions to how we think about the organization of the human and non-human primate visual systems.

Denis Aristide Baylor. 30 January 1940—16 March 2022

Denis Baylor was one of the leading sensory neuroscientists of the latter part of the twentieth century. His research on the rod and cone photoreceptors of the retina provided a deep understanding of the electrical responses of these cells to light and also of the molecular mechanisms of photo­transduction, whereby light absorption is linked via biochemical reactions within the cell to ion channels in its membrane and thence to cellular membrane voltage changes. He also made major contributions to knowledge of the transmission of these signals to the retina's ganglion cells, which forward information to the brain. Over his career, he invented and/or exploited three important methods for electrophysiological recording: ultra-high-resistance sharp microelectrodes; the suction-pipette technique for photoreceptors; and multielectrode arrays for retina. His research findings have provided important advances in our understanding of visual neuroscience, from the detection of individual photon absorptions by rod photoreceptors, through the underlying molecular mechanisms, to the processing and transmission of these visual signals through the retina. He instilled a love for experimental neuroscience and rigorous scientific investigation into a generation of postdoctoral trainees, students and colleagues. He died of a heart attack on 16 March 2022.

(Don't) look where you are going: Evidence for a travel direction signal in humans that is independent of head direction.

We often assume that travel direction is redundant with head direction, but from first principles, these two factors provide differing spatial information. Although head direction has been found to be a fundamental component of human navigation, it is unclear how self-motion signals for travel direction contribute to forming a travel trajectory. Employing a novel motion adaptation paradigm from visual neuroscience designed to preclude a contribution of head direction, we found high-level aftereffects of perceived travel direction, indicating that travel direction is a fundamental component of human navigation. Interestingly, we discovered a higher frequency of reporting perceived travel toward the adapted direction compared to a no-adapt control-an aftereffect that runs contrary to low-level motion aftereffects. This travel aftereffect was maintained after controlling for possible response biases and approaching effects, and it scaled with adaptation duration. These findings demonstrate the first evidence of how a pure travel direction signal might be represented in humans, independent of head direction. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Visual neuroscience: A brain area tuned for processing social interactions

Socialising with others is part of everyday life. A new study demonstrates that a brain area specialised for visual body perception is attuned to processing social interactions between two people. Intriguingly, this area is lateralised in the left hemisphere.

Adaptive smoothing of retinotopic maps based on Teichmüller parametrization

Retinotopic mapping, the mapping between visual inputs on the retina and neural responses on the cortical surface, is one of the fundamental topics in visual neuroscience. In human studies, retinotopic maps are conventionally constructed and processed by decoding blood oxygenation-level dependent (BOLD) functional magnetic resonance imaging (fMRI) responses to designed visual stimuli on the cortical surface. However, these methods frequently generate retinotopic maps that do not preserve topology, contradicting a fundamental property of retinotopic maps observed in neurophysiology. To address this problem, we propose an integrated approach to simultaneously refine the flattening from the 3D cortical surface to the 2D parametric space and adaptively smooth retinotopic perception centers in the visual space to make the retinotopic maps topological. One key element of the approach is the enhanced error tolerant Teichmüller mapping, which refines the parametrization by minimizing angle distortions and maximizing alignment to noisy landmarks. We validated our overall approach with synthetic and real retinotopic mapping datasets and applied it to compute cortical magnification factor (CMF). The results showed that the proposed approach was superior to other conventional retinotopic mapping methods in predicting BOLD fMRI time series and preserving topology.

Carl Bergmann (1814–1865) and the discovery of the anatomical site in the retina where vision is initiated

ABSTRACT A preeminent quest of nineteenth-century visual neuroscience was to identify the anatomical elements of the retina that respond to light. A major breakthrough came in 1854, when Carl Bergmann discovered through direct observation that the human fovea contains only rods and cones. On this basis, he argued that these must necessarily be the light-sensitive elements (i.e., photoreceptors) that initiate vision. Bergmann also argued that Henle’s fibers form part of the necessary anatomical link between these distal elements and the proximal ganglion cells, which transmit visual signals to the brain via the optic nerve. However, despite his achievement, Heinrich Müller, not Bergmann, is remembered as the discoverer of human photoreceptors in the literature. This article seeks to correct the record. It situates Bergmann’s work alongside that of his contemporaries, sets out his arguments and the critique he received using archival documents, and makes this history more accessible for current readers by comparing what was said to what we know now. We argue that Bergmann’s arguments are at least as compelling as those of Müller, and that he should be recognized as a codiscoverer of the anatomic site in the retina where vision is initiated.

Poster Session I: Digital dual-Purkinje-image eye tracking enables precise determination of visual receptive fields in fixating macaques.

Understanding the relationship between visual stimuli and neural activity is a fundamental goal in visual neuroscience. However, the study of visual neurophysiology in awake primates is complicated by the constant occurrence of eye movements, even during periods of nominal fixation. To address this challenge, we adapted a recently developed high-resolution digital dual-Purkinje-image (dDPI) eye tracker (Wu et al., 2023) for use with macaque monkeys. In addition to tracking the Purkinje images, we simultaneously estimate the pupil center and size: a first for video eye tracking. We then sought to evaluate the efficacy of dDPI eye tracking for studying visual processing in fixating macaques by recording single neurons from the lateral geniculate nucleus while a spatially-correlated noise stimulus was displayed. Our analyses show that, as a result of properly accounting for eye movements post-hoc, the predictive performance of generalized linear models improves and the estimates center radii contract to values equal to or smaller than those reported in the literature. Notably, correcting for eye movements using the locations of the pupil center and corneal reflection-the standard method in video eye tracking-yielded worse model fits and larger receptive field sizes. This finding implies that the pupil center is an inaccurate reporter of small eye movements, while the Purkinje images may be veridical during fixations.

Visual feature tuning properties of stimulus-driven saccadic inhibition in macaque monkeys.

Saccadic inhibition refers to a short-latency transient cessation of saccade generation after visual sensory transients. This oculomotor phenomenon occurs with a latency that is consistent with a rapid influence of sensory responses, such as stimulus-induced visual bursts, on oculomotor control circuitry. However, the neural mechanisms underlying saccadic inhibition are not well understood. Here, we exploited the fact that macaque monkeys experience robust saccadic inhibition to test the hypothesis that inhibition time and strength exhibit systematic visual feature tuning properties to a multitude of visual feature dimensions commonly used in vision science. We measured saccades in three monkeys actively controlling their gaze on a target, and we presented visual onset events at random times. Across seven experiments, the visual onsets tested size, spatial frequency, contrast, orientation, motion direction, and motion speed dependencies of saccadic inhibition. We also investigated how inhibition might depend on the behavioral relevance of the appearing stimuli. We found that saccadic inhibition starts earlier, and is stronger, for large stimuli of low spatial frequencies and high contrasts. Moreover, saccadic inhibition timing depends on motion direction and orientation, with earlier inhibition systematically occurring for horizontally drifting vertical gratings. On the other hand, saccadic inhibition is stronger for faster motions and when the appearing stimuli are subsequently foveated. Besides documenting a range of feature tuning dimensions of saccadic inhibition to the properties of exogenous visual stimuli, our results establish macaque monkeys as an ideal model system for unraveling the neural mechanisms underlying a ubiquitous oculomotor phenomenon in visual neuroscience.NEW & NOTEWORTHY Visual onsets dramatically reduce saccade generation likelihood with very short latencies. Such latencies suggest that stimulus-induced visual responses, normally jump-starting perceptual and scene analysis processes, can also directly impact the decision of whether to generate saccades or not, causing saccadic inhibition. Consistent with this, we found that changing the appearance of the visual onsets systematically alters the properties of saccadic inhibition. These results constrain neurally inspired models of coordination between saccade generation and exogenous sensory stimulation.

Stakes of neuromorphic foveation: a promising future for embedded event cameras.

Foveation can be defined as the organic action of directing the gaze towards a visual region of interest to acquire relevant information selectively. With the recent advent of event cameras, we believe that taking advantage of this visual neuroscience mechanism would greatly improve the efficiency of event data processing. Indeed, applying foveation to event data would allow to comprehend the visual scene while significantly reducing the amount of raw data to handle. In this respect, we demonstrate the stakes of neuromorphic foveation theoretically and empirically across several computer vision tasks, namely semantic segmentation and classification. We show that foveated event data have a significantly better trade-off between quantity and quality of the information conveyed than high- or low-resolution event data. Furthermore, this compromise extends even over fragmented datasets. Our code is publicly available online at: https://github.com/amygruel/FoveationStakes_DVS .

A large-scale fMRI dataset for the visual processing of naturalistic scenes

One ultimate goal of visual neuroscience is to understand how the brain processes visual stimuli encountered in the natural environment. Achieving this goal requires records of brain responses under massive amounts of naturalistic stimuli. Although the scientific community has put a lot of effort into collecting large-scale functional magnetic resonance imaging (fMRI) data under naturalistic stimuli, more naturalistic fMRI datasets are still urgently needed. We present here the Natural Object Dataset (NOD), a large-scale fMRI dataset containing responses to 57,120 naturalistic images from 30 participants. NOD strives for a balance between sampling variation between individuals and sampling variation between stimuli. This enables NOD to be utilized not only for determining whether an observation is generalizable across many individuals, but also for testing whether a response pattern is generalized to a variety of naturalistic stimuli. We anticipate that the NOD together with existing naturalistic neuroimaging datasets will serve as a new impetus for our understanding of the visual processing of naturalistic stimuli.

Spatiotemporal visual statistics of aquatic environments in the natural habitats of zebrafish

Animal sensory systems are tightly adapted to the demands of their environment. In the visual domain, research has shown that many species have circuits and systems that exploit statistical regularities in natural visual signals. The zebrafish is a popular model animal in visual neuroscience, but relatively little quantitative data is available about the visual properties of the aquatic habitats where zebrafish reside, as compared to terrestrial environments. Improving our understanding of the visual demands of the aquatic habitats of zebrafish can enhance the insights about sensory neuroscience yielded by this model system. We analyzed a video dataset of zebrafish habitats captured by a stationary camera and compared this dataset to videos of terrestrial scenes in the same geographic area. Our analysis of the spatiotemporal structure in these videos suggests that zebrafish habitats are characterized by low visual contrast and strong motion when compared to terrestrial environments. Similar to terrestrial environments, zebrafish habitats tended to be dominated by dark contrasts, particularly in the lower visual field. We discuss how these properties of the visual environment can inform the study of zebrafish visual behavior and neural processing and, by extension, can inform our understanding of the vertebrate brain.

Fovea-UNet: detection and segmentation of lymph node metastases in colorectal cancer with deep learning

BackgroundColorectal cancer is one of the most serious malignant tumors, and lymph node metastasis (LNM) from colorectal cancer is a major factor for patient management and prognosis. Accurate image detection of LNM is an important task to help clinicians diagnose cancer. Recently, the U-Net architecture based on convolutional neural networks (CNNs) has been widely used to segment image to accomplish more precise cancer diagnosis. However, the accurate segmentation of important regions with high diagnostic value is still a great challenge due to the insufficient capability of CNN and codec structure in aggregating the detailed and non-local contextual information. In this work, we propose a high performance and low computation solution.MethodsInspired by the working principle of Fovea in visual neuroscience, a novel network framework based on U-Net for cancer segmentation named Fovea-UNet is proposed to adaptively adjust the resolution according to the importance-aware of information and selectively focuses on the region most relevant to colorectal LNM. Specifically, we design an effective adaptively optimized pooling operation called Fovea Pooling (FP), which dynamically aggregate the detailed and non-local contextual information according to the pixel-level feature importance. In addition, the improved lightweight backbone network based on GhostNet is adopted to reduce the computational cost caused by FP.ResultsExperimental results show that our proposed framework can achieve higher performance than other state-of-the-art segmentation networks with 79.38% IoU, 88.51% DSC, 92.82% sensitivity and 84.57% precision on the LNM dataset, and the parameter amount is reduced to 23.23 MB.ConclusionsThe proposed framework can provide a valid tool for cancer diagnosis, especially for LNM of colorectal cancer.

Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues.

Convolutional Neural Networks (CNN) are a class of machine learning models predominately used in computer vision tasks and can achieve human-like performance through learning from experience. Their striking similarities to the structural and functional principles of the primate visual system allow for comparisons between these artificial networks and their biological counterparts, enabling exploration of how visual functions and neural representations may emerge in the real brain from a limited set of computational principles. After considering the basic features of CNNs, we discuss the opportunities and challenges of endorsing CNNs as in silico models of the primate visual system. Specifically, we highlight several emerging notions about the anatomical and physiological properties of the visual system that still need to be systematically integrated into current CNN models. These tenets include the implementation of parallel processing pathways from the early stages of retinal input and the reconsideration of several assumptions concerning the serial progression of information flow. We suggest design choices and architectural constraints that could facilitate a closer alignment with biology provide causal evidence of the predictive link between the artificial and biological visual systems. Adopting this principled perspective could potentially lead to new research questions and applications of CNNs beyond modeling object recognition.