Visual Sensory Information Research Articles

An experimental demonstration of neuromorphic sensing of chemical species using electro-optical reservoir computing.

A chemical discrimination system based on photonic reservoir computing is demonstrated experimentally for the first time. The system is inspired by the way humans perceive and process visual sensory information. The electro-optical reservoir computing system is a photonic analogue of the human nervous system with the read-out layer acting as the 'brain', and the sensor that of the human eye. A task-specific optimisation of the system is implemented, and the performance of the system for the discrimination between three chemicals is presented. The results are compared to the previously published numerical simulation (Anufriev et al. in Opt Mater Express 12:1767-1783, 2022, 10.1364/OME.449036). This publication provides a feasibility assessment and a demonstration of a practical realisation of photonic reservoir computing for a new neuromorphic sensing system - the next generation sensor with a built-in 'intelligence' which can be trained to 'understand' and to make a real time sensing decision based on the training data.

Genetically defined neuron types underlying visuomotor transformation in the superior colliculus.

The superior colliculus (SC) is a conserved midbrain structure that is important for transforming visual and other sensory information into motor actions. Decades of investigations in numerous species have made the SC and its nonmammalian homologue, the optic tectum, one of the best studied structures in the brain, with rich information now available regarding its anatomical organization, its extensive inputs and outputs and its important functions in many reflexive and cognitive behaviours. Excitingly, recent studies using modern genomic and physiological approaches have begun to reveal the diverse neuronal subtypes in the SC, as well as their unique functions in visuomotor transformation. Studies have also started to uncover how subtypes of SC neurons form intricate circuits to mediate visual processing and visually guided behaviours. Here, we review these recent discoveries on the cell types and neuronal circuits underlying visuomotor transformations mediated by the SC. We also highlight the important future directions made possible by these new developments.

Widespread, perception-related information in the human brain scales with levels of consciousness

Abstract How does the human brain generate coherent, subjective perceptions—transforming yellow and oblong visual sensory information into the perception of an edible banana? This is a hard problem. According to the standard viewpoint, processing in groups of dedicated regions—identified as active “blobs” when using functional magnetic resonance imaging (fMRI)—gives rise to perception. Here, our research reveals a new organizational concept by discovering stimulus-specific information distributed throughout the whole brain. Using fMRI, we found stimulus-specific information across the neocortex, even in voxels previously considered “noise,” challenging traditional analytical approaches. Surprisingly, these stimulus-specific signals were also present in the subcortex and cerebellum and could be detected from across-subject variances. Additionally, we observed that the widespread signal in brain regions beyond the primary and secondary sensory cortices is influenced by sedation levels, suggesting a connection to perception rather than sensory encoding. We hypothesize that these widespread, stimulus-specific, and consciousness level-dependent signals may underlie coherent and subjective perceptions.

Towards knowledge-infused automated disease diagnosis assistant

With the advancement of internet communication and telemedicine, people are increasingly turning to the web for various healthcare activities. With an ever-increasing number of diseases and symptoms, diagnosing patients becomes challenging. In this work, we build a diagnosis assistant to assist doctors, which identifies diseases based on patient–doctor interaction. During diagnosis, doctors utilize both symptomatology knowledge and diagnostic experience to identify diseases accurately and efficiently. Inspired by this, we investigate the role of medical knowledge in disease diagnosis through doctor–patient interaction. We propose a two-channel, knowledge-infused, discourse-aware disease diagnosis model (KI-DDI), where the first channel encodes patient–doctor communication using a transformer-based encoder, while the other creates an embedding of symptom-disease using a graph attention network (GAT). In the next stage, the conversation and knowledge graph embeddings are infused together and fed to a deep neural network for disease identification. Furthermore, we first develop an empathetic conversational medical corpus comprising conversations between patients and doctors, annotated with intent and symptoms information. The proposed model demonstrates a significant improvement over the existing state-of-the-art models, establishing the crucial roles of (a) a doctor’s effort for additional symptom extraction (in addition to patient self-report) and (b) infusing medical knowledge in identifying diseases effectively. Many times, patients also show their medical conditions, which acts as crucial evidence in diagnosis. Therefore, integrating visual sensory information would represent an effective avenue for enhancing the capabilities of diagnostic assistants.

Separation of bimodal fMRI responses in mouse somatosensory areas into V1 and non-V1 contributions

Multisensory integration is necessary for the animal to survive in the real world. While conventional methods have been extensively used to investigate the multisensory integration process in various brain areas, its long-range interactions remain less explored. In this study, our goal was to investigate interactions between visual and somatosensory networks on a whole-brain scale using 15.2-T BOLD fMRI. We compared unimodal to bimodal BOLD fMRI responses and dissected potential cross-modal pathways with silencing of primary visual cortex (V1) by optogenetic stimulation of local GABAergic neurons. Our data showed that the influence of visual stimulus on whisker activity is higher than the influence of whisker stimulus on visual activity. Optogenetic silencing of V1 revealed that visual information is conveyed to whisker processing via both V1 and non-V1 pathways. The first-order ventral posteromedial thalamic nucleus (VPM) was functionally affected by non-V1 sources, while the higher-order posterior medial thalamic nucleus (POm) was predominantly modulated by V1 but not non-V1 inputs. The primary somatosensory barrel field (S1BF) was influenced by both V1 and non-V1 inputs. These observations provide valuable insights for into the integration of whisker and visual sensory information.

Evidence for tactile 3D shape discrimination by octopus.

Octopuses integrate visual, chemical and tactile sensory information while foraging and feeding in complex marine habitats. The respective roles of these modes are of interest ecologically, neurobiologically, and for development of engineered soft robotic arms. While vision guides their foraging path, benthic octopuses primarily search "blindly" with their arms to find visually hidden prey amidst rocks, crevices and coral heads. Each octopus arm is lined with hundreds of suckers that possess a combination of chemo- and mechanoreceptors to distinguish prey. Contact chemoreception has been demonstrated in lab tests, but mechanotactile sensing is less well characterized. We designed a non-invasive live animal behavioral assay that isolated mechanosensory capabilities of Octopus bimaculoides arms and suckers to discriminate among five resin 3D-printed prey and non-prey shapes (all with identical chemical signatures). Each shape was introduced inside a rock dome and was only accessible to the octopus' arms. Octopuses' responses were variable. Young octopuses discriminated the crab prey shape from the control, whereas older octopuses did not. These experiments suggest that mechanotactile sensing of 3D shapes may aid in prey discrimination; however, (i) chemo-tactile information may be prioritized over mechanotactile information in prey discrimination, and (ii) mechanosensory capability may decline with age.

Robust Sensory Information Reconstruction and Classification With Augmented Spikes.

Sensory information recognition is primarily processed through the ventral and dorsal visual pathways in the primate brain visual system, which exhibits layered feature representations bearing a strong resemblance to convolutional neural networks (CNNs), encompassing reconstruction and classification. However, existing studies often treat these pathways as distinct entities, focusing individually on pattern reconstruction or classification tasks, overlooking a key feature of biological neurons, the fundamental units for neural computation of visual sensory information. Addressing these limitations, we introduce a unified framework for sensory information recognition with augmented spikes. By integrating pattern reconstruction and classification within a single framework, our approach not only accurately reconstructs multimodal sensory information but also provides precise classification through definitive labeling. Experimental evaluations conducted on various datasets including video scenes, static images, dynamic auditory scenes, and functional magnetic resonance imaging (fMRI) brain activities demonstrate that our framework delivers state-of-the-art pattern reconstruction quality and classification accuracy. The proposed framework enhances the biological realism of multimodal pattern recognition models, offering insights into how the primate brain visual system effectively accomplishes the reconstruction and classification tasks through the integration of ventral and dorsal pathways.

Exploring neurophysiological correlates of visually induced motion sickness using electroencephalography (EEG).

Visually induced motion sickness (VIMS) is a common phenomenon when using visual devices such as smartphones and virtual reality applications, with symptoms including nausea, fatigue, and headache. To date, the neuro-cognitive processes underlying VIMS are not fully understood. Previous studies using electroencephalography (EEG) delivered mixed findings, with some reporting an increase in delta and theta power, and others reporting increases in alpha and beta frequencies. The goal of the study was to gain further insight into EEG correlates for VIMS. Participants viewed a VIMS-inducing visual stimulus, composed of moving black-and-white vertical bars presented on an array of three adjacent monitors. The EEG was recorded during visual stimulation and VIMS ratings were recorded after each trial using the Fast Motion Sickness Scale. Time-frequency analyses were conducted comparing neural activity of participants reporting minimal VIMS (n = 21) and mild-moderate VIMS (n = 12). Results suggested a potential increase in delta power in the centro-parietal regions (CP2) and a decrease in alpha power in the central regions (Cz) for participants experiencing mild-moderate VIMS compared to those with minimal VIMS. Event-related spectral perturbations (ERSPs) suggested that group differences in EEG activity developed with increasing duration of a trial. These results support the hypothesis that the EEG might be sensitive to differences in information processing in VIMS and minimal VIMS contexts, and indicate that it may be possible to identify neurophysiological correlate of VIMS. Differences in EEG activity related to VIMS may reflect differential processing of conflicting visual and vestibular sensory information.

General tau theory as a model to evaluate audiovisual interplay in interceptive actions

When interacting with the environment, sensory information is essential to guide movements. Picking up the appropriate sensory information (both visual and auditory) about the progression of an event is required to reach the right place at the right time. In this study, we aimed to see if general tau theory could explain the audiovisual guidance of movement in interceptive action (an interception task). The specific contributions of auditory and visual sensory information were tested by timing synchronous and asynchronous audiovisual interplays in successful interceptive trials. The performance was computed by using the tau-coupling model for information-movement guidance. Our findings revealed that while the auditory contribution to movement guidance did change across conditions, the visual contribution remained constant. In addition, when comparing the auditory and visual contributions, the results revealed a significant decrease in the auditory compared to the visual contribution in just one of the asynchronous conditions where the visual target was presented after the sound. This may be because more attention was drawn to the visual information, resulting in a decrease in the auditory guidance of movement. To summarize, our findings reveal how tau-coupling can be used to disentangle the relative contributions of the visual and auditory sensory modalities in movement planning.

The use of multimedia technologies in robotics lessons as the development of cognitive activity

In the environment of educational robotics, the introduction of new pedagogical technologies is relevant. Robotics is at the junction of promising fields of knowledge: mechanics, electronics, automation, assembly, programming, circuit engineering and technical design. The purpose of the article is to describe multimedia software in lessons on the basics of robotics and teaching methods using multimedia technologies at each stage of training. The research methods were analysis, synthesis, and conversation. The authors of the article have developed Digital educational resources for the study of robotics. These resources are an auxiliary tool for the diversity of the lesson. With the help of this software product, robotics training becomes more accessible and convenient. Multimedia allows you to combine verbal and visual-sensory information, which helps to increase the motivation of students, create certain learning parameters and develop cognitive activity. When organizing educational material, the teacher should focus on the activation of cognitive activity of students. In the classroom, not all students turn on active cognitive activity. Active teaching methods are used to purposefully influence this activity. The very name contains the essence of the active method - it is not simple memorization, but the active intellectual activity of students. The result of this study is a digital learning center with the help of which it is possible to conduct lessons using multimedia technologies. Keywords: educational robotics, multimedia technologies, cognitive activity, digital educational resources.

The influence of visual deprivation on the development of the thalamocortical network: Evidence from congenitally blind children and adults

The thalamus is heavily involved in relaying sensory signals to the cerebral cortex. A relevant issue is how the deprivation of congenital visual sensory information modulates the development of the thalamocortical network. The answer is unclear because previous studies on this topic did not investigate network development, structure-function combinations, and cognition-related behaviors in the same study. To overcome these limitations, we recruited 30 congenitally blind subjects (8 children, 22 adults) and 31 sighted subjects (10 children, 21 adults), and conducted multiple analyses [i.e., gray matter volume (GMV) analysis using the voxel-based morphometry (VBM) method, resting-state functional connectivity (FC), and brain-behavior correlation]. We found that congenital blindness elicited significant changes in the development of GMV in visual and somatosensory thalamic regions. Blindness also resulted in significant changes in the development of FC between somatosensory thalamic regions and visual cortical regions as well as advanced information processing regions. Moreover, the somatosensory thalamic regions and their FCs with visual cortical regions were reorganized to process high-level tactile language information in blind individuals. These findings provide a refined understanding of the neuroanatomical and functional plasticity of the thalamocortical network.

Synaptic alterations in visual cortex reshape contrast-dependent gamma oscillations and inhibition-excitation ratio in a genetic mouse model of migraine

BackgroundMigraine affects a significant fraction of the world population, yet its etiology is not completely understood. In vitro results highlighted thalamocortical and intra-cortical glutamatergic synaptic gain-of-function associated with a monogenic form of migraine (familial-hemiplegic-migraine-type-1: FHM1). However, how these alterations reverberate on cortical activity remains unclear. As altered responsivity to visual stimuli and abnormal processing of visual sensory information are common hallmarks of migraine, herein we investigated the effects of FHM1-driven synaptic alterations in the visual cortex of awake mice.MethodsWe recorded extracellular field potentials from the primary visual cortex (V1) of head-fixed awake FHM1 knock-in (n = 12) and wild type (n = 12) mice in response to square-wave gratings with different visual contrasts. Additionally, we reproduced in silico the obtained experimental results with a novel spiking neurons network model of mouse V1, by implementing in the model both the synaptic alterations characterizing the FHM1 genetic mouse model adopted.ResultsFHM1 mice displayed similar amplitude but slower temporal evolution of visual evoked potentials. Visual contrast stimuli induced a lower increase of multi-unit activity in FHM1 mice, while the amount of information content about contrast level remained, however, similar to WT.Spectral analysis of the local field potentials revealed an increase in the β/low γ range of WT mice following the abrupt reversal of contrast gratings. Such frequency range transitioned to the high γ range in FHM1 mice. Despite this change in the encoding channel, these oscillations preserved the amount of information conveyed about visual contrast. The computational model showed how these network effects may arise from a combination of changes in thalamocortical and intra-cortical synaptic transmission, with the former inducing a lower cortical activity and the latter inducing the higher frequencies ɣ oscillations.ConclusionsContrast-driven ɣ modulation in V1 activity occurs at a much higher frequency in FHM1. This is likely to play a role in the altered processing of visual information. Computational studies suggest that this shift is specifically due to enhanced cortical excitatory transmission. Our network model can help to shed light on the relationship between cellular and network levels of migraine neural alterations.Graphical

Augmented Reality VS Virtual Reality

Augmented reality (AR) which is defined as that “the real time use of information in the form of graphics, text, audio, or other virtual enhancements integrated with real world objects”. Augmented reality involves auditory, overlaying visual or other sensory information onto the world in order to enhance one’s experiment.AR works on to by adding the digital contents onto a live camera feed, making that digital content looks as if it is part of the physical world around as. The applications which are used in AR is Healthcare, Education, Retail ,Military, Industrial and manufacturing, Engineering, Marketing and advertising etc. Virtual reality (VR) which is defined as that a computer generation simulation in which a person can interact with in an artificial 3-D environment using electronic devices such as special goggles with a screen or gloves fitted with sensors. The Virtual Reality Modelling Language (VRML) allows the creator to specify images and the rules for their display and interactions using textual language statements. The applications which are used in VR is Military, Sports, Fashion, Medical training, Education etc.

Thermal pain and detection threshold modulation in augmented reality

Augmented Reality (AR) overlays computer-generated visual, auditory or other sensory information onto the real world. Due to recent technological advancement in the field, it can become increasingly difficult for the user to differentiate between sensory information coming from real and virtual objects, leading to interesting perceptual phenomena. For example, an AR experience in which users can experience their own hands in flames has been shown to elicit heat illusions on the affected hands. In this study, we investigate the potential that AR has for top-down modulation of pain and thermal perception. We assessed thermal pain and detection thresholds on the participant’s right hand while covering it with realistic virtual flames. We compared this experience to a baseline condition with no additional stimuli. We also report on a condition in which the hand is covered by a blue fluid not instantly associated with fire. We found that experiencing a virtual burning hand induces analgesic as well hyperalgesic effects as participants begin to feel heat related pain at lower temperatures and cold related pain at higher temperatures. The experience also impacts significantly on the lowest temperature at which participants starts perceiving warmth. The blue fluid do not affect the thresholds corresponding to the baseline condition. Our research thus confirms previous experiments showing that pain and thermal perception can be manipulated by by AR, while providing quantitative results on the magnitude of this effect.

Deep Movement Primitives: Toward Breast Cancer Examination Robot

Breast cancer is the most common type of cancer worldwide. A robotic system performing autonomous breast palpation can make a significant impact on the related health sector worldwide. However, robot programming for breast palpating with different geometries is very complex and unsolved. Robot learning from demonstrations (LfD) reduces the programming time and cost. However, the available LfD are lacking the modelling of the manipulation path/trajectory as an explicit function of the visual sensory information. This paper presents a novel approach to manipulation path/trajectory planning called deep Movement Primitives that successfully generates the movements of a manipulator to reach a breast phantom and perform the palpation. We show the effectiveness of our approach by a series of real-robot experiments of reaching and palpating a breast phantom. The experimental results indicate our approach outperforms the state-of-the-art method.

Effects of Light Isoflurane Anesthesia on Organization of Direction and Orientation Selectivity in the Superficial Layer of the Mouse Superior Colliculus.

The superior colliculus (SC) is the midbrain center for integrating visual and multimodal sensory information. Neurons in the SC exhibit direction and orientation selectivity. Recent studies reported that neurons with similar preferences formed clusters in the mouse SC (Ahmadlou and Heimel, 2015; Feinberg and Meister, 2015; de Malmazet et al., 2018; Li et al., 2020). However, it remains controversial as to how these clusters are organized within the SC (Inayat et al., 2015; Chen et al., 2021). Here, we found that different brain states (i.e., awake or anesthetized with isoflurane) changed the selectivity of individual SC neurons and organizations of the neuronal population in both male and female mice. Using two-photon Ca2+ imaging, we examined both individual neuronal responses and the spatial patterns of their population responses. Under isoflurane anesthesia, orientation selectivity increased and a larger number of orientation-selective cells were observed when compared with the awake condition, whereas the proportions of direction-selective cells were similar in both conditions. Furthermore, direction- and orientation-selective cells located at closer positions showed more similar preferences, and cluster-like spatial patterns were enhanced. Inhibitory responses of direction-selective neurons were also reduced under isoflurane anesthesia. Thus, the changes in the spatial organization of response patterns were considered to be because of changes in the balance of excitation and inhibition, with excitation dominance, in the local circuits. These results provide new insights into the possibility that the functional organization of feature selectivity in the brain is affected by brain state.SIGNIFICANCE STATEMENT Recent large-scale recording studies are changing our view of visual maps in the superior colliculus (SC), including findings of cluster-like localizations of direction- and orientation-selective neurons. However, results from several laboratories are conflicting regarding the presence of cluster-like organization. Here, we demonstrated that light isoflurane anesthesia affected the direction- and orientation-tuning properties in the mouse superficial SC and that their cluster-like localization pattern was enhanced by the anesthesia. Furthermore, the effect of anesthesia on direction selectivity appeared to be different in the excitatory and inhibitory populations in the SC. Our results suggest that the functional organization of direction and orientation selectivity might be regulated by the excitation-inhibition balance that depends on the brain state.

Dynamics of microglia and dendritic spines in early adolescent cortex after developmental alcohol exposure.

Fetal alcohol spectrum disorder patients suffer from many cognitive disabilities. These include impaired auditory, visual, and tactile sensory information processing, making it more difficult for these patients to learn to navigate social scenarios. Rodent studies have shown that alcohol exposure during the brain growth spurt (BGS) can lead to acute neuronal apoptosis and an immunological response by microglia in the somatosensory cortex. Since microglia have critical physiological functions, including the support of excitatory synapse remodeling via interactions with dendritic spines, we sought to understand whether BGS alcohol exposure has long-term effects on microglial or dendritic spine dynamics. Using in vivo two-photon microscopy in 4-5 week old mice, we evaluated microglial functions such as process motility, the response to tissue injury, and the dynamics of physical interactions between microglial processes and dendritic spines. We also investigated potential differences in the morphology, density, or dynamics of dendritic spines in layer I/II primary sensory cortex of control and BGS alcohol exposed mice. We found that microglial process motility and contact with dendritic spines were not altered after BGS alcohol exposure. While the response of microglial processes toward tissue injury was not significantly altered by prior alcohol exposure, there was a trend suggesting that alcohol early in life may prime microglia to respond more quickly to secondary injury. Spine density, morphology, stability, and remodeling over time were not perturbed after BGS alcohol exposure. We demonstrate that after BGS alcohol exposure, the physiological functions of microglia and excitatory neurons remain intact in early adolescence.

Deep-LfD: Deep robot learning from demonstrations

<h2>Abstract</h2> Like other robot learning from demonstration (LfD) approaches, deep-LfD builds a task model from sample demonstrations. However, unlike conventional LfD, the deep-LfD model learns the relation between high dimensional visual sensory information and robot trajectory/path. This paper presents a dataset of successful needle insertion by da Vinci Research Kit into deformable objects based on which several deep-LfD models are built as a benchmark of models learning robot controller for the needle insertion task.

The US Department of Justice stumbles on visual perception

A large and highly valuable category of forensic evidence consists of patterned impressions created during the perpetration of a crime. These crime scene artifacts, such as fingerprints or tire tracks, offer visual sensory information that is assessed by trained human observers and compared to sensory experiences elicited by model patterns that would have been produced under a hypothesized set of conditions. By means of this “forensic feature comparison,” the observer makes a judgment about whether the evidence and the model are sufficiently similar to support common origin. In light of documented failures of this approach, significant concerns have been raised about its scientific validity. In response to these concerns, the US Department of Justice has made assertions about how forensic examiners perform feature comparison tasks that are not consistent with modern scientific understanding of the processes of sensation and perception. Clarification of these processes highlights new ways of thinking about and improving the accuracy of forensic feature comparison and underscores the vital role of science in achieving justice.

The effect of substrate on prey capture does not match natural substrate use in a wolf spider

Because habitats present different costs and benefits in contexts ranging from foraging to mating to predator avoidance, decisions about habitat use likely involve significant trade-offs across behavioural contexts. We examine this possibility in Schizocosa floridana , a cursorial wolf spider whose habitat consists of multiple substrates, including oak and pine litter, and sand. However, S. floridana restricts its use of habitat to oak litter. Substrate environments vary significantly in the degree to which they transmit visual and vibratory information. Previous work found that oak litter best transmits the substrate-borne vibrations that are critical to mating communication. However, oak litter may reduce the availability of visual information, which is known to be important for prey capture in other wolf spider species. Here, we test the relationship between substrate and prey capture efficiency in S. floridana . We examine substrate use in a foraging context via two laboratory-based experiments to (1) assess differences in prey capture efficiency among the three most common substrates in the field and to (2) determine which sensory modalities are involved in prey capture. We found that prey capture rates were highest on sand despite the fact that sand is the least used substrate by S. floridana in the wild. We also found no conclusive evidence that either the visual or the vibratory sensory modalities are critical for prey capture. Our results thus suggest that strict habitat specialization such as that exhibited by S. floridana is beneficial in some ecological contexts but costly in others. • Schizocosa floridana wolf spiders restrict their habitat use to oak litter. • Spiders catch prey more efficiently on sand than on oak litter. • Sand transmits substrate-borne vibrations poorly. • Loss of vibratory or visual sensory information does not impede prey capture.