Multimodal Sensory Information Research Articles

Disruption of network hierarchy pattern in bulimia nervosa reveals brain information integration disorder

The human brain works as a hierarchical organization that is a continuous axis spanning sensorimotor cortex to transmodal cortex (referring to cortex that integrates multimodal sensory information and participates in complex cognitive functions). Previous studies have demonstrated abnormalities in several specific networks that may account for their multiple behavioral deficits in patients with bulimia nervosa (BN), but whether and how the network hierarchical organization changes in BN remain unknown. This study aimed to investigate alterations of the hierarchy organization in BN network and their clinical relevance. Connectome gradient analyses were applied to depict the network hierarchy patterns of fifty-nine patients with BN and thirty-nine healthy controls (HCs). Then, we evaluated the network- and voxel-level gradient alterations of BN by comparing gradient values in each network and each voxel between patients with BN and HCs. Finally, the association between altered gradient values and clinical variables was explored. In the principal gradient, patients with BN exhibited reduced gradient values in dorsal attention network and increased gradient values in subcortical regions compared to HCs. In the secondary gradient, patients with BN showed decreased gradient values in ventral attention network and increased gradient values in limbic network. Regionally, the areas with altered principal or secondary gradient values in BN group were mainly located in transmodal networks, i.e., the default-mode and frontoparietal network. In BN group, the principal gradient values of right inferior frontal gyrus were negatively associated with external eating behavior. This study revealed the disordered network hierarchy patterns in patients with BN, which suggested a disturbance of brain information integration from attention network and subcortical regions to transmodal networks in these patients. These findings may provide insight into the neurobiological underpinnings of BN.

Modeling neurodegenerative and neurodevelopmental disorders in the Drosophila mushroom body.

The common fruit fly Drosophila melanogaster provides a powerful platform to investigate the genetic, molecular, cellular, and neural circuit mechanisms of behavior. Research in this model system has shed light on multiple aspects of brain physiology and behavior, from fundamental neuronal function to complex behaviors. A major anatomical region that modulates complex behaviors is the mushroom body (MB). The MB integrates multimodal sensory information and is involved in behaviors ranging from sensory processing/responses to learning and memory. Many genes that underlie brain disorders are conserved, from flies to humans, and studies in Drosophila have contributed significantly to our understanding of the mechanisms of brain disorders. Genetic mutations that mimic human diseases-such as Fragile X syndrome, neurofibromatosis type 1, Parkinson's disease, and Alzheimer's disease-affect MB structure and function, altering behavior. Studies dissecting the effects of disease-causing mutations in the MB have identified key pathological mechanisms, and the development of a complete connectome promises to add a comprehensive anatomical framework for disease modeling. Here, we review Drosophila models of human neurodevelopmental and neurodegenerative disorders via the effects of their underlying mutations on MB structure, function, and the resulting behavioral alterations.

To attract a moth: Wind tunnel and field testing of plant odor and light stimuli and their combination for Ostrinia nubilalis

AbstractThe European corn borer, Ostrinia nubilalis (Hubner), relies on multimodal sensory information to find food, mates, mating and ovipositional grounds. Successful phytosanitary monitoring demands for the bait for the field traps to obtain the most reliable representation of pest abundance. Attraction to light and blend of key components of host plant odor, was tested both in the laboratory and field conditions. Ultraviolet light, which was the most effective in the wind tunnel experiments, was further tested in the field alone and in combination with bisexual lure. Bisexual lure, being attractive in the lab, as well as in the field, did not improve responses to ultraviolet in both experimental designs. All three baits attracted significantly more females than males in the field. Wind tunnel experiments revealed that ultraviolet elicited the shortest response latencies either alone or paired with the odor bait. The lack of synergistic effect between attractive light and odor stimuli is an important issue for pest monitoring. The possible reasons for the observed lack of synergy are the hierarchy of behavioral responses to different stimuli or the intensities of both stimuli are critically important for attractivity of combined stimulus and differ from separately presented ones.

Contribution of oxytocin and dopamine to the formation of neural clusters in the neocortex representing multimodal sensory stimuli

We have previously proposed a unified mechanism for the formation of contrasted representations of multimodal sensory stimuli in the activity of neocortical neurons. Contrasting is based on the opposite sign of modification of the efficacy of strong and weak excitatory inputs to the spiny cells of the striatum (the input structure of the basal ganglia) and the subsequent dopamine-dependent activity reorganizations in parallel cortico – basal ganglia – thalamocortical loops. Oxytocin and dopamine (through D1 receptors) can improve the contrast of these representations, contributing to the induction of LTP of the efficacy of excitation of cortical, thalamic, and hippocampal neurons innervating spiny cells. In addition, oxytocin and dopamine can improve contrasting enhancement by increasing the signal-to-noise ratio in the neocortex, hippocampus, and striatum. A proposed mechanism for increasing the signal-to-noise ratio is based on the opposite sign of a long-term modification of the efficacy of monosynaptic excitatory and disynaptic inhibitory inputs, simultaneously affecting the postsynaptic neuron. The proposed mechanisms may underlie the contribution of oxytocin and dopamine to improving the formation and long-term maintenance of activity in neuronal groups with similar receptive fields that form columns in the primary visual cortex, a tonotopic map in the primary auditory cortex, a somatotopic map in the sensorimotor cortex, and distributed clusters in the olfactory piriform cortex. These mechanisms differ from the commonly accepted mechanisms of the formation of neuronal clusters in the neocortex with similar RPs, that are based on afferent and lateral excitation and inhibition, which does not allow providing the specificity and duration of effects. Understanding the mechanisms of involvement of oxytocin and dopamine in the processing of multimodal sensory information may be useful for developing treatments for some disorders of social behavior.

Neural substrates of marriage on self-parents processing and the association with a parents-oriented perspective shift in a collectivistic culture

Relationship with parents is a special bond that shapes self-other representations and have an impact on adult-child’s marriage, especially in the early stages of marriage. This study sought to investigate the neural mechanisms underlying self-parents processing as well as their relationship with marriage. Seventy-eight premarital Korean participants were scanned in functional MRI while evaluating traits of the self and parents. Then, 21 of them returned after being married to engage in the identical task three years later. The precuneus and temporoparietal junction were identified to activate stronger for parents than self at both marital statuses. The dorsal anterior cingulate cortex, posterior cingulate cortex, parietal operculum, and caudate activated more for self than parents before marriage, but their activities changed during marriage. The activation increase of the parietal operculum between marital statuses in the parents condition was negatively correlated with the level of marital dissatisfaction, and this association only appeared among participants with a child. Self-parents processing may recruit brain regions involved in autobiographical memory and self-other distinction, and marriage has an impact on the way individuals process rewards and multimodal sensory information during this processing. Marriage may lead to changes in brain function that affect the processing of emotions toward parents and a more parents-oriented perspective shift in collectivistic societies.

The anterior retrosplenial cortex is required for short-term object in place recognition memory retrieval: Role of ionotropic glutamate receptors in male and female Long-Evans rats.

The anterior retrosplenial cortex (aRSC) integrates multimodal sensory information into cohesive associative recognition memories. Little is known about how information is integrated during different learning phases (i.e., encoding and retrieval). Additionally, sex differences are observed in performance of some visuospatial memory tasks; however, inconsistent findings warrant more research. We conducted three experiments using the 1-h delay object-in-place (1-h OiP) test to assess recognition memory retrieval in male and female Long-Evans rats. (i) We found both sexes performed equally in three repeated 1-h OiP test sessions. (ii) We showed infusions of a mixture of muscimol/baclofen (GABAA/B receptor agonists) into the aRSC ~15-min prior to the test phase disrupted 1-h OiP in both sexes. (iii) We assessed the role of aRSC ionotropic glutamate receptors in 1-h OiP retrieval using another squad of cannulated rats and confirmed that infusions of either the competitive AMPA/Kainate receptor antagonist CNQX (3 mM) or competitive NMDA receptor antagonist AP-5 (30 mM) (volumes = 0.50 uL/side) significantly impaired 1-h OiP retrieval in both sexes compared to controls. Taken together, findings challenge reported sex differences and clearly establish a role for aRSC ionotropic glutamate receptors in short-term visuospatial recognition memory retrieval. Thus, modulating neural activity in the aRSC may alleviate some memory processing impairments in related disorders.

Multimodal Sensors Enabled Autonomous Soft Robotic System with Self-Adaptive Manipulation.

Human hands are amazingly skilled at recognizing and handling objects of different sizes and shapes. To date, soft robots rarely demonstrate autonomy equivalent to that of humans for fine perception and dexterous operation. Here, an intelligent soft robotic system with autonomous operation and multimodal perception ability is developed by integrating capacitive sensors with triboelectric sensor. With distributed multiple sensors, our robot system can not only sense and memorize multimodal information but also enable an adaptive grasping method for robotic positioning and grasp control, during which the multimodal sensory information can be captured sensitively and fused at feature level for crossmodally recognizing objects, leading to a highly enhanced recognition capability. The proposed system, combining the performance and physical intelligence of biological systems (i.e., self-adaptive behavior and multimodal perception), will greatly advance the integration of soft actuators and robotics in many fields.

Quantitative functional imaging of the pigeon brain: implications for the evolution of avian powered flight.

The evolution of flight is a rare event in vertebrate history, and one that demands functional integration across multiple anatomical/physiological systems. The neuroanatomical basis for such integration and the role that brain evolution assumes in behavioural transformations remain poorly understood. We make progress by (i) generating a positron emission tomography (PET)-based map of brain activity for pigeons during rest and flight, (ii) using these maps in a functional analysis of the brain during flight, and (iii) interpreting these data within a macroevolutionary context shaped by non-avian dinosaurs. Although neural activity is generally conserved from rest to flight, we found significant increases in the cerebellum as a whole and optic flow pathways. Conserved activity suggests processing of self-movement and image stabilization are critical when a bird takes to the air, while increased visual and cerebellar activity reflects the importance of integrating multimodal sensory information for flight-related movements. A derived cerebellar capability likely arose at the base of maniraptoran dinosaurs, where volumetric expansion and possible folding directly preceded paravian flight. These data represent an important step toward establishing how the brain of modern birds supports their unique behavioural repertoire and provide novel insights into the neurobiology of the bird-like dinosaurs that first achieved powered flight.

Robust Sensory Information Reconstruction and Classification With Augmented Spikes.

Sensory information recognition is primarily processed through the ventral and dorsal visual pathways in the primate brain visual system, which exhibits layered feature representations bearing a strong resemblance to convolutional neural networks (CNNs), encompassing reconstruction and classification. However, existing studies often treat these pathways as distinct entities, focusing individually on pattern reconstruction or classification tasks, overlooking a key feature of biological neurons, the fundamental units for neural computation of visual sensory information. Addressing these limitations, we introduce a unified framework for sensory information recognition with augmented spikes. By integrating pattern reconstruction and classification within a single framework, our approach not only accurately reconstructs multimodal sensory information but also provides precise classification through definitive labeling. Experimental evaluations conducted on various datasets including video scenes, static images, dynamic auditory scenes, and functional magnetic resonance imaging (fMRI) brain activities demonstrate that our framework delivers state-of-the-art pattern reconstruction quality and classification accuracy. The proposed framework enhances the biological realism of multimodal pattern recognition models, offering insights into how the primate brain visual system effectively accomplishes the reconstruction and classification tasks through the integration of ventral and dorsal pathways.

Multisensory input modulates memory-guided spatial navigation in humans

Efficient navigation is supported by a cognitive map of space. The hippocampus plays a key role for this map by linking multimodal sensory information with spatial memory representations. However, in human navigation studies, the full range of sensory information is often unavailable due to the stationarity of experimental setups. We investigated the contribution of multisensory information to memory-guided spatial navigation by presenting a virtual version of the Morris water maze on a screen and in an immersive mobile virtual reality setup. Patients with hippocampal lesions and matched controls navigated to memorized object locations in relation to surrounding landmarks. Our results show that availability of multisensory input improves memory-guided spatial navigation in both groups. It has distinct effects on navigational behaviour, with greater improvement in spatial memory performance in patients. We conclude that congruent multisensory information shifts computations to extrahippocampal areas that support spatial navigation and compensates for spatial navigation deficits.

A multimodal cortical network of sensory expectation violation revealed by fMRI.

The brain is subjected to multi-modal sensory information in an environment governed by statistical dependencies. Mismatch responses (MMRs), classically recorded with EEG, have provided valuable insights into the brain's processing of regularities and the generation of corresponding sensory predictions. Only few studies allow for comparisons of MMRs across multiple modalities in a simultaneous sensory stream and their corresponding cross-modal context sensitivity remains unknown. Here, we used a tri-modal version of the roving stimulus paradigm in fMRI to elicit MMRs in the auditory, somatosensory and visual modality. Participants (N = 29) were simultaneously presented with sequences of low and high intensity stimuli in each of the three senses while actively observing the tri-modal input stream and occasionally reporting the intensity of the previous stimulus in a prompted modality. The sequences were based on a probabilistic model, defining transition probabilities such that, for each modality, stimuli were more likely to repeat (p = .825) than change (p = .175) and stimulus intensities were equiprobable (p = .5). Moreover, each transition was conditional on the configuration of the other two modalities comprising global (cross-modal) predictive properties of the sequences. We identified a shared mismatch network of modality general inferior frontal and temporo-parietal areas as well as sensory areas, where the connectivity (psychophysiological interaction) between these regions was modulated during mismatch processing. Further, we found deviant responses within the network to be modulated by local stimulus repetition, which suggests highly comparable processing of expectation violation across modalities. Moreover, hierarchically higher regions of the mismatch network in the temporo-parietal area around the intraparietal sulcus were identified to signal cross-modal expectation violation. With the consistency of MMRs across audition, somatosensation and vision, our study provides insights into a shared cortical network of uni- and multi-modal expectation violation in response to sequence regularities.

Possible rodent equivalent of the posterior cingulate cortex (area 23) interconnects with multimodal cortical and subcortical regions.

Posterior cingulate cortex (area 23, A23) in human and monkeys is a critical component of the default mode network and is involved in many diseases such as Alzheimer's disease, autism, depression, attention deficit hyperactivity disorder and schizophrenia. However, A23 has not yet identified in rodents, and this makes modeling related circuits and diseases in rodents very difficult. Using a comparative approach, molecular markers and unique connectional patterns this study has uncovered the location and extent of possible rodent equivalent (A23~) of the primate A23. A23 ~ but not adjoining areas in the rodents displays strong reciprocal connections with anteromedial thalamic nucleus. Rodent A23 ~ reciprocally connects with the medial pulvinar and claustrum as well as with anterior cingulate, granular retrosplenial, medial orbitofrontal, postrhinal, and visual and auditory association cortices. Rodent A23 ~ projects to dorsal striatum, ventral lateral geniculate nucleus, zona incerta, pretectal nucleus, superior colliculus, periaqueductal gray, and brainstem. All these findings support the versatility of A23 in the integration and modulation of multimodal sensory information underlying spatial processing, episodic memory, self-reflection, attention, value assessment and many adaptive behaviors. Additionally, this study also suggests that the rodents could be used to model monkey and human A23 in future structural, functional, pathological, and neuromodulation studies.

Soft Robotic Perception System with Ultrasonic Auto-Positioning and Multimodal Sensory Intelligence

Flexible electronics such as tactile cognitive sensors have been broadly adopted in soft robotic manipulators to enable human-skin-mimetic perception. To achieve appropriate positioning for randomly distributed objects, an integrated guiding system is inevitable. Yet the conventional guiding system based on cameras or optical sensors exhibits limited environment adaptability, high data complexity, and low cost effectiveness. Herein, a soft robotic perception system with remote object positioning and a multimodal cognition capability is developed by integrating an ultrasonic sensor with flexible triboelectric sensors. The ultrasonic sensor is able to detect the object shape and distance by reflected ultrasound. Thereby the robotic manipulator can be positioned to an appropriate position to perform object grasping, during which the ultrasonic and triboelectric sensors can capture multimodal sensory information such as object top profile, size, shape, hardness, material, etc. These multimodal data are then fused for deep-learning analytics, leading to a highly enhanced accuracy in object identification (∼100%). The proposed perception system presents a facile, low-cost, and effective methodology to integrate positioning capability with multimodal cognitive intelligence in soft robotics, significantly expanding the functionalities and adaptabilities of current soft robotic systems in industrial, commercial, and consumer applications.

Multimodal Neuromorphic Sensory-Processing System With Memristor Circuits for Smart Home Applications

To enable smart home applications, embedding home monitoring systems with different sensors can be a feasible remedy to capture the multimodal sensory information from daily life. However, the previously developed monitoring systems have high implementation cost, large power consumption, and complicated device configuration, which are not conducive to achieving carbon neutrality and carbon peak goals. Here, we demonstrate a multimodal neuromorphic sensory-processing system with memristor circuits for smart homes, offering a more environmentally friendly approach with low cost and easily deployable hardware. We used three components to facilitate the multimodal neuromorphic sensory-processing system design. First, a memristor crossbar array is fabricated using low-cost, reliable, and eco-friendly 2-D materials; enabling a parallel connection to reduce the system complexity and the deep-learning computational cost. Second, a neuromorphic multimodal sensory-processing module is designed for collecting and processing multiple sensory cues efficiently; to establish accurate depiction of the environment and realize timely response. Third, the biomimetic hierarchical learning module is realized for smart home monitoring and relative applications (e.g., indoor human behavior recognition). The developed multimodal neuromorphic sensory-processing system effectively combines the nanotechnology, intelligent sensing, and deep learning; indicating an advancement in smart home applications.

Fusion of tactile and visual information in deep learning models for object recognition

Humans use multimodal sensory information to understand the physical properties of their environment. Intelligent decision-making systems such as the ones used in robotic applications could also utilize the fusion of multimodal information to improve their performance and reliability. In recent years, machine learning and deep learning methods are used at the heart of such intelligent systems. Developing visuo-tactile models is a challenging task due to various problems such as performance, limited datasets, reliability, and computational efficiency. In this research, we propose four efficient models based on dynamic neural network architectures for unimodal and multimodal object recognition. For unimodal object recognition, TactileNet and VisionNet are proposed. For multimodal object recognition, the FusionNet-A and the FusionNet-B are designed to implement early and late fusion strategies, respectively. The proposed models have a flexible structure and are able to change at the train or test phase to accommodate the amount of available information. Model confidence calibration is employed to enhance the reliability and generalization of the models. The proposed models are evaluated on MIT CSAIL large-scale multimodal dataset. Our results demonstrate accurate performance in both unimodal and multimodal scenarios. It has been illustrated that by using different fusion strategies and augmenting the tactile-based models with visual information, the top-1 error rate of the single-frame tactile model was reduced by 78% and the mean average precision was increased by 2.19 times. Although the focus has been on the fusion of tactile and visual modalities, the proposed design methodology can be generalized to include more modalities.

Passive somatosensory training enhances piano skill in adolescent and adult pianists: A preliminary study.

Sensory afferent information, such as auditory and somatosensory feedback while moving, plays a crucial role in both control and learning of motor performance across the lifespan. Music performance requires skillful integration of multimodal sensory information for the production of dexterous movements. However, it has not been understood what roles somatosensory afferent information plays in the acquisition and sophistication of specialized motor skills of musicians across different stages of development. In the present preliminary study, we addressed this issue by using a novel technique with a hand exoskeleton robot that can externally move the fingers of pianists. Short-term exposure to fast and complex finger movements generated by the exoskeleton (i.e., passive movements) increased the maximum rate of repetitive piano keystrokes by the pianists. This indicates that somatosensory inputs derived from the externally generated motions enhanced the quickness of the sequential finger movements in piano performance, even though the pianists did not voluntarily move the fingers. The enhancement of motor skill through passive somatosensory training using the exoskeleton was more pronounced in adolescent pianists than adult pianists. These preliminary results implicate a sensitive period of neuroplasticity of the somatosensory-motor system of trained pianists, which emphasizes the importance of somatosensory-motor training in professional music education during adolescence.

Processing of sensory, painful and vestibular stimuli in the thalamus

ObjectivesThe thalamus plays an important role in the mediation and integration of various stimuli (e.g., somatosensory, pain, and vestibular). Whether a stimulus-specific and topographic organization of the thalamic nuclei exists is still unknown. The aim of our study was to define a functional, in vivo map of multimodal sensory processing within the human thalamus.MethodsTwenty healthy individuals (10 women, 21–34 years old) participated. Defined sensory stimuli were applied to both hands (innocuous touch, mechanical pain, and heat pain) and the vestibular organ (galvanic stimulation) during 3 T functional MRI.ResultsBilateral thalamic activations could be detected for touch, mechanical pain, and vestibular stimulation within the left medio-dorsal and right anterior thalamus. Heat pain did not lead to thalamic activation at all. Stimuli applied to the left body side resulted in stronger activation patterns. Comparing an early with a late stimulation interval, the mentioned activation patterns were far more pronounced within the early stimulation interval.ConclusionsThe right anterior and ventral-anterior nucleus and the left medio-dorsal nucleus appear to be important for the processing of multimodal sensory information. In addition, galvanic stimulation is processed more laterally compared to mechanical pain. The observed changes in activity within the thalamic nuclei depending on the stimulation interval suggest that the stimuli are processed in a thalamic network rather than a distinct nucleus. In particular, the vestibular network within the thalamus recruits bilateral nuclei, rendering the thalamus an important integrative structure for vestibular function.

A Bayesian model for chronic pain.

The perceiving mind constructs our coherent and embodied experience of the world from noisy, ambiguous and multi-modal sensory information. In this paper, we adopt the perspective that the experience of pain may similarly be the result of a probabilistic, inferential process. Prior beliefs about pain, learned from past experiences, are combined with incoming sensory information in a Bayesian manner to give rise to pain perception. Chronic pain emerges when prior beliefs and likelihoods are biased towards inferring pain from a wide range of sensory data that would otherwise be perceived as harmless. We present a computational model of interoceptive inference and pain experience. It is based on a Bayesian graphical network which comprises a hidden layer, representing the inferred pain state; and an observable layer, representing current sensory information. Within the hidden layer, pain states are inferred from a combination of priors , transition probabilities between hidden states and likelihoods of certain observations . Using variational inference and free-energy minimization, the model is able to learn from observations over time. By systematically manipulating parameter settings, we demonstrate that the model is capable of reproducing key features of both healthy- and chronic pain experience. Drawing on mathematical concepts, we finally simulate treatment resistant chronic pain and discuss mathematically informed treatment options.

Robust Transcoding Sensory Information With Neural Spikes.

Neural coding, including encoding and decoding, is one of the key problems in neuroscience for understanding how the brain uses neural signals to relate sensory perception and motor behaviors with neural systems. However, most of the existed studies only aim at dealing with the continuous signal of neural systems, while lacking a unique feature of biological neurons, termed spike, which is the fundamental information unit for neural computation as well as a building block for brain-machine interface. Aiming at these limitations, we propose a transcoding framework to encode multi-modal sensory information into neural spikes and then reconstruct stimuli from spikes. Sensory information can be compressed into 10% in terms of neural spikes, yet re-extract 100% of information by reconstruction. Our framework can not only feasibly and accurately reconstruct dynamical visual and auditory scenes, but also rebuild the stimulus patterns from functional magnetic resonance imaging (fMRI) brain activities. More importantly, it has a superb ability of noise immunity for various types of artificial noises and background signals. The proposed framework provides efficient ways to perform multimodal feature representation and reconstruction in a high-throughput fashion, with potential usage for efficient neuromorphic computing in a noisy environment.

Lateral Habenula Beyond Avoidance: Roles in Stress, Memory, and Decision-Making With Implications for Psychiatric Disorders.

In this Perspective review, we highlight some of the less explored aspects of lateral habenula (LHb) function in contextual memory, sleep, and behavioral flexibility. We provide evidence that LHb is well-situated to integrate different internal state and multimodal sensory information from memory-, stress-, motivational-, and reward-related circuits essential for both survival and decision making. We further discuss the impact of early life stress (ELS) on LHb function as an example of stress-induced hyperactivity and dysregulation of neuromodulatory systems within the LHb that promote anhedonia and motivational deficits following ELS. We acknowledge that recent technological advancements in manipulation and recording of neural circuits in simplified and well-controlled behavioral paradigms have been invaluable in our understanding of the critical role of LHb in motivation and emotional regulation as well as the involvement of LHb dysfunction in stress-induced psychopathology. However, we also argue that the use of ethologically-relevant behaviors with consideration of complex aspects of decision-making is warranted for future studies of LHb contributions in a wide range of psychiatric illnesses. We conclude this Perspective with some of the outstanding issues for the field to consider where a multi-systems approach is needed to investigate the complex nature of LHb circuitry interactions with environmental stimuli that predisposes psychiatric disorders.