Brain Processes Visual Information Research Articles

From sight to insight: A multi-task approach with the visual language decoding model

Visual neural decoding aims to unlock the mysteries of how the human brain interprets the visual world through predicting perceived visual information from visual neural activity. While early studies made some progress in decoding visual activity for singular type of information, they failed to concurrently reveal the multi-level interweaving linguistic information in the brain. Here, we developed a novel Visual Language Decoding Model (VLDM) that can simultaneously decode the main categories, semantic labels, and textual descriptions of visual stimuli from visual activities. The large-scale NSD dataset was utilized to ensure the efficiency of the decoding model in joint training and evaluation across multiple tasks. For category decoding, we achieved the effective classification of 12 categories with an accuracy of nearly 70 %, significantly surpassing the chance level. For label decoding, we attained the precise prediction of 80 specific semantic labels with a 16-fold improvement over the chance level. For text decoding, the scores of the decoded text surpassed the corresponding baseline levels by remarkable margins on six evaluation metrics. These results highlight the complexity of how the brain processes visual information and the close connection between visual perception and language cognition. This study contributes significantly to extensive applications in multi-layered brain-computer interfaces, potentially leading to more natural and efficient human-computer interaction experiences.

A neurocomputational model of decision and confidence in object recognition task

How does the brain process natural visual stimuli to make a decision? Imagine driving through fog. An object looms ahead. What do you do? This decision requires not only identifying the object but also choosing an action based on your decision confidence. In this circumstance, confidence is making a bridge between seeing and believing. Our study unveils how the brain processes visual information to make such decisions with an assessment of confidence, using a model inspired by the visual cortex. To computationally model the process, this study uses a spiking neural network inspired by the hierarchy of the visual cortex in mammals to investigate the dynamics of feedforward object recognition and decision-making in the brain. The model consists of two modules: a temporal dynamic object representation module and an attractor neural network-based decision-making module. Unlike traditional models, ours captures the evolution of evidence within the visual cortex, mimicking how confidence forms in the brain. This offers a more biologically plausible approach to decision-making when encountering real-world stimuli. We conducted experiments using natural stimuli and measured accuracy, reaction time, and confidence. The model's estimated confidence aligns remarkably well with human-reported confidence. Furthermore, the model can simulate the human change-of-mind phenomenon, reflecting the ongoing evaluation of evidence in the brain. Also, this finding offers decision-making and confidence encoding share the same neural circuit.

Analysis and Classification of Event-Related Potentials During Image Observation.

In the field of cognitive neuroscience, researchers have conducted extensive studies on object categorization using Event-Related Potential (ERP) analysis, specifically by analyzing electroencephalographic (EEG) response signals triggered by visual stimuli. The most common approach for visual ERP analysis is to use a low presentation rate of images and an active task where participants actively discriminate between target and non-target images. However, researchers are also interested in understanding how the human brain processes visual information in real-world scenarios. To simulate real-life object recognition, this study proposes an analysis pipeline of visual ERPs evoked by images presented in a Rapid Serial Visual Presentation (RSVP) paradigm. Such an approach allows for the investigation of recurrent patterns of visual ERP signals across specific categories and subjects. The pipeline includes segmentation of the EEGs in epochs, and the use of the resulting features as inputs for Support Vector Machine (SVM) classification. Results demonstrate common ERP patterns across the selected categories and the ability to obtain discriminant information from single visual stimuli presented in the RSVP paradigm.

STNet: A novel spiking neural network combining its own time signal with the spatial signal of an artificial neural network.

This article proposes a novel hybrid network that combines the temporal signal of a spiking neural network (SNN) with the spatial signal of an artificial neural network (ANN), namely the Spatio-Temporal Combined Network (STNet). Inspired by the way the visual cortex in the human brain processes visual information, two versions of STNet are designed: a concatenated one (C-STNet) and a parallel one (P-STNet). In the C-STNet, the ANN, simulating the primary visual cortex, extracts the simple spatial information of objects first, and then the obtained spatial information is encoded as spiking time signals for transmission to the rear SNN which simulates the extrastriate visual cortex to process and classify the spikes. With the view that information from the primary visual cortex reaches the extrastriate visual cortex via ventral and dorsal streams, in P-STNet, the parallel combination of the ANN and the SNN is employed to extract the original spatio-temporal information from samples, and the extracted information is transferred to a posterior SNN for classification. The experimental results of the two STNets obtained on six small and two large benchmark datasets were compared with eight commonly used approaches, demonstrating that the two STNets can achieve improved performance in terms of accuracy, generalization, stability, and convergence. These prove that the idea of combining ANN and SNN is feasible and can greatly improve the performance of SNN.

From perception to behavior: The neural circuits underlying prey hunting in larval zebrafish.

A key challenge for neural systems is to extract relevant information from the environment and make appropriate behavioral responses. The larval zebrafish offers an exciting opportunity for studying these sensing processes and sensory-motor transformations. Prey hunting is an instinctual behavior of zebrafish that requires the brain to extract and combine different attributes of the sensory input and form appropriate motor outputs. Due to its small size and transparency the larval zebrafish brain allows optical recording of whole-brain activity to reveal the neural mechanisms involved in prey hunting and capture. In this review we discuss how the larval zebrafish brain processes visual information to identify and locate prey, the neural circuits governing the generation of motor commands in response to prey, how hunting behavior can be modulated by internal states and experience, and some outstanding questions for the field.

A New Paradigm for Safety Data Signal Detection and Evaluation Using Open-Source Software Created by an Interdisciplinary Working Group

Techniques to evaluate large amounts of safety data continue to evolve based on a greater understanding of how the brain processes visual information and the advancement of programing tools. The Interactive Safety Graphics Task Force of the American Statistical Association Biopharmaceutical Safety Working Group has assembled a multidisciplinary team of experts in a variety of domains to develop the next generation of open-source visual analytical tools for safety data based on these advances. The multidisciplinary approach resulted in the rapid development of the first tool, a novel interactive version of the familiar Evaluation of Drug-Induced Serious Hepatotoxicity (eDISH) graphic along with a unique clinical workflow to guide the reviewer through the data analysis. This now serves as the model for the team to expand the open-source platform into a suite of other interactive safety analysis tools.

Frontal cortical regions associated with attention connect more strongly to central than peripheral V1

The functionality of central vision is different from peripheral vision. Central vision is used for fixation and has higher acuity, making it useful for everyday activities such as reading and object identification. The central and peripheral representations in primary visual cortex (V1) also differ in how higher-order processing areas modulate their responses. For example, attention and expectation are top-down processes (i.e., high-order cognitive functions) that influence visual information processing during behavioral tasks. This top-down control is different for central vs. peripheral vision. Since functional networks can influence visual information processing in different ways, networks (such as the Fronto-Parietal (FPN), Default Mode (DMN), and Cingulo-Opercular (CON)) likely differ in how they connect to representations of the visual field across V1. Prior work indicated the central representing portion of V1 was more functionally connected to regions belonging to the FPN, and the far-peripheral representing portion of V1 was more functionally connected to regions belonging to the DMN.Our goals were (1) Assess the reproducibility and generalizability of retinotopic effects on functional connections between V1 and functional networks. (2) Extend this work to understand structural connections of central vs. peripheral representations in V1. (3) Examine the overlapping eccentricity differences in functional and structural connections of V1. (4) Examine the major white matter tracks connecting central V1 to frontal regions.We used resting-state BOLD fMRI and DWI to examine whether portions of V1 that represent different visual eccentricities differ in their functional and structural connectivity to functional networks. All data were acquired and minimally preprocessed by the Human Connectome Project. We identified central and far-peripheral representing regions from a retinotopic template. Functional connectivity was measured by correlated activity between V1 and functional networks, and structural connectivity was measured by probabilistic tractography and converted to track probability. In both modalities, differences between V1 eccentricity segment connections were compared by paired, two-tailed t-test. A spatial permutation approach was used to determine the statistical significance of the spatial overlap between modalities. The identified spatial overlap was then used in a deterministic tractography approach to identify the white matter pathways connecting the overlap to central V1.We found (1) Centrally representing portions of V1 are more strongly functionally connected to frontal regions than are peripherally representing portions of V1, (2) Structural connections also show stronger connections between central V1 and frontal regions, (3) Patterns of structural and functional connections overlaps in the lateral frontal cortex, (4) This lateral frontal overlap is connected to central V1 via the IFOF.In summary, the work's main contribution is a greater understanding of higher-order functional networks' connectivity to V1. There are stronger structural connections to central representations in V1, particularly for lateral frontal regions, implying that the functional relationship between central V1 and frontal regions is built upon direct, long-distance connections via the IFOF. Overlapping structural and functional connections reflect differences in V1 eccentricities, with central V1 preferentially connected to attention-associated regions. Understanding how V1 is functionally and structurally connected to higher-order brain areas contributes to our understanding of how the human brain processes visual information and forms a baseline for understanding any modifications in processing that might occur with training or experience.

Neural activity in human visual cortex is transformed by learning real world size

The way that our brain processes visual information is directly affected by our experience. Repeated exposure to a visual stimulus triggers experience-dependent plasticity in the visual cortex of many species. Humans also have the unique ability to acquire visual knowledge through instruction. We introduced human participants to the real-world size of previously unfamiliar species, and to the functional motion of novel tools, during a functional magnetic resonance imaging scan. Using machine learning, we compared activity patterns evoked by images of the new items, before and after participants learned the animals' real-world size or tools' motion. We found that, after acquiring size information, participants' visual activity patterns for the new animals became more confusable with activity patterns evoked by similar-sized known animals in early visual cortex, but not in ventral temporal cortex, reflecting an influence of new size knowledge on posterior, but not anterior, components of the ventral stream. In contrast, learning the functional motion of new tools did not lead to an equivalent change in recorded activity. Finally, the time-points marked by evidence of new size information in early visual cortex were more likely to show size information and greater activation in the right angular gyrus, a key hub of semantic knowledge and spatial cognition. Overall, these findings suggest that learning an item's real-world size by instruction influences subsequent activity in visual cortex and in a region that is central to semantic and spatial brain systems.

Global/local integration and corpus callosum: Anatomical and behavioural study of case of Allgrove syndrome (triple-A syndrome)

Visuospatial perception is a complex ability, with various aspects. We are continuously integrating different information, like when our brain processes visual information from left and right hemisphere, in order to form a single and unitary image. The processing of visuospatial information, and more specifically its global/local aspect, has led to many studies, indicating the active involvement of the corpus callosum. Objective To study the influence of an alteration of the corpus callosum on the integration of global/local information. Methods C.M. presents an Allgrove syndrome [1] , likely a pathology of white matter fibres. This patient, with a singular neuropsychological profile, is partially impaired in global and local information integration and shows a corpus callosum alteration. This research, a case-study with a group of 6 matched control subjects, relied on a two-level approach, anatomical (MRI-DTI) and behavioural, using a hierarchical letter paradigme [2] . Results Despite all the precautions needed regarding the interpretation of our results, some specificities are emphasized: at an anatomical level, we note, for C.M., a microscopic alteration of her corpus callosum; the results from our behavioural protocol show, for C.M., a higher precedence effect, a greater sensibility to interference and a facilitation effect more important. Discussion These different results, never reported before in this pathology, indicates, for C.M., an impairment in global/local integration associated with a corpus callosum alteration. This alteration, probably part of a more global degenerative process linked to her pathology, shows the relevance of an extended neuropsychological assessment coupled to a DTI study, in order to better understand impairments generally appearing in neuropsychological rehabilitation.

Real-time mutual gaze perception enhances collaborative learning and collaboration quality

In this paper we present the results of an eye-tracking study on collaborative problem-solving dyads. Dyads remotely collaborated to learn from contrasting cases involving basic concepts about how the human brain processes visual information. In one condition, dyads saw the eye gazes of their partner on the screen; in a control group, they did not have access to this information. Results indicated that this real-time mutual gaze perception intervention helped students achieve a higher quality of collaboration and a higher learning gain. Implications for supporting group collaboration are discussed.

Visual processing for action resists similarity of relevant and irrelevant object features

It has been suggested that the human brain processes visual information in different manners, depending on whether the information is used for perception or for action control. This distinction has been criticized for the lack of behavioral dissociations that unambiguously support the proposed two-visual-pathways model. Here we present a new and simple dissociation between vision for perception and vision for action: Perceptual judgments are affected by the similarity of relevant and irrelevant stimulus features, while object-oriented actions are not. This dissociation overcomes the methodological problems of previously proposed differences in terms of vulnerability to visual illusions or to variability in irrelevant object features, and it can also serve as an easily applicable behavioral indicator of underlying processing modes.