Multimodal Sensory Information Research Articles

To attract a moth: Wind tunnel and field testing of plant odor and light stimuli and their combination for Ostrinia nubilalis

AbstractThe European corn borer, Ostrinia nubilalis (Hubner), relies on multimodal sensory information to find food, mates, mating and ovipositional grounds. Successful phytosanitary monitoring demands for the bait for the field traps to obtain the most reliable representation of pest abundance. Attraction to light and blend of key components of host plant odor, was tested both in the laboratory and field conditions. Ultraviolet light, which was the most effective in the wind tunnel experiments, was further tested in the field alone and in combination with bisexual lure. Bisexual lure, being attractive in the lab, as well as in the field, did not improve responses to ultraviolet in both experimental designs. All three baits attracted significantly more females than males in the field. Wind tunnel experiments revealed that ultraviolet elicited the shortest response latencies either alone or paired with the odor bait. The lack of synergistic effect between attractive light and odor stimuli is an important issue for pest monitoring. The possible reasons for the observed lack of synergy are the hierarchy of behavioral responses to different stimuli or the intensities of both stimuli are critically important for attractivity of combined stimulus and differ from separately presented ones.

Neural substrates of marriage on self-parents processing and the association with a parents-oriented perspective shift in a collectivistic culture

Relationship with parents is a special bond that shapes self-other representations and have an impact on adult-child’s marriage, especially in the early stages of marriage. This study sought to investigate the neural mechanisms underlying self-parents processing as well as their relationship with marriage. Seventy-eight premarital Korean participants were scanned in functional MRI while evaluating traits of the self and parents. Then, 21 of them returned after being married to engage in the identical task three years later. The precuneus and temporoparietal junction were identified to activate stronger for parents than self at both marital statuses. The dorsal anterior cingulate cortex, posterior cingulate cortex, parietal operculum, and caudate activated more for self than parents before marriage, but their activities changed during marriage. The activation increase of the parietal operculum between marital statuses in the parents condition was negatively correlated with the level of marital dissatisfaction, and this association only appeared among participants with a child. Self-parents processing may recruit brain regions involved in autobiographical memory and self-other distinction, and marriage has an impact on the way individuals process rewards and multimodal sensory information during this processing. Marriage may lead to changes in brain function that affect the processing of emotions toward parents and a more parents-oriented perspective shift in collectivistic societies.

The anterior retrosplenial cortex is required for short-term object in place recognition memory retrieval: Role of ionotropic glutamate receptors in male and female Long-Evans rats.

The anterior retrosplenial cortex (aRSC) integrates multimodal sensory information into cohesive associative recognition memories. Little is known about how information is integrated during different learning phases (i.e., encoding and retrieval). Additionally, sex differences are observed in performance of some visuospatial memory tasks; however, inconsistent findings warrant more research. We conducted three experiments using the 1-h delay object-in-place (1-h OiP) test to assess recognition memory retrieval in male and female Long-Evans rats. (i) We found both sexes performed equally in three repeated 1-h OiP test sessions. (ii) We showed infusions of a mixture of muscimol/baclofen (GABAA/B receptor agonists) into the aRSC ~15-min prior to the test phase disrupted 1-h OiP in both sexes. (iii) We assessed the role of aRSC ionotropic glutamate receptors in 1-h OiP retrieval using another squad of cannulated rats and confirmed that infusions of either the competitive AMPA/Kainate receptor antagonist CNQX (3 mM) or competitive NMDA receptor antagonist AP-5 (30 mM) (volumes = 0.50 uL/side) significantly impaired 1-h OiP retrieval in both sexes compared to controls. Taken together, findings challenge reported sex differences and clearly establish a role for aRSC ionotropic glutamate receptors in short-term visuospatial recognition memory retrieval. Thus, modulating neural activity in the aRSC may alleviate some memory processing impairments in related disorders.

Multimodal Sensors Enabled Autonomous Soft Robotic System with Self-Adaptive Manipulation.

Human hands are amazingly skilled at recognizing and handling objects of different sizes and shapes. To date, soft robots rarely demonstrate autonomy equivalent to that of humans for fine perception and dexterous operation. Here, an intelligent soft robotic system with autonomous operation and multimodal perception ability is developed by integrating capacitive sensors with triboelectric sensor. With distributed multiple sensors, our robot system can not only sense and memorize multimodal information but also enable an adaptive grasping method for robotic positioning and grasp control, during which the multimodal sensory information can be captured sensitively and fused at feature level for crossmodally recognizing objects, leading to a highly enhanced recognition capability. The proposed system, combining the performance and physical intelligence of biological systems (i.e., self-adaptive behavior and multimodal perception), will greatly advance the integration of soft actuators and robotics in many fields.

Quantitative functional imaging of the pigeon brain: implications for the evolution of avian powered flight.

The evolution of flight is a rare event in vertebrate history, and one that demands functional integration across multiple anatomical/physiological systems. The neuroanatomical basis for such integration and the role that brain evolution assumes in behavioural transformations remain poorly understood. We make progress by (i) generating a positron emission tomography (PET)-based map of brain activity for pigeons during rest and flight, (ii) using these maps in a functional analysis of the brain during flight, and (iii) interpreting these data within a macroevolutionary context shaped by non-avian dinosaurs. Although neural activity is generally conserved from rest to flight, we found significant increases in the cerebellum as a whole and optic flow pathways. Conserved activity suggests processing of self-movement and image stabilization are critical when a bird takes to the air, while increased visual and cerebellar activity reflects the importance of integrating multimodal sensory information for flight-related movements. A derived cerebellar capability likely arose at the base of maniraptoran dinosaurs, where volumetric expansion and possible folding directly preceded paravian flight. These data represent an important step toward establishing how the brain of modern birds supports their unique behavioural repertoire and provide novel insights into the neurobiology of the bird-like dinosaurs that first achieved powered flight.

Robust Sensory Information Reconstruction and Classification With Augmented Spikes.

Sensory information recognition is primarily processed through the ventral and dorsal visual pathways in the primate brain visual system, which exhibits layered feature representations bearing a strong resemblance to convolutional neural networks (CNNs), encompassing reconstruction and classification. However, existing studies often treat these pathways as distinct entities, focusing individually on pattern reconstruction or classification tasks, overlooking a key feature of biological neurons, the fundamental units for neural computation of visual sensory information. Addressing these limitations, we introduce a unified framework for sensory information recognition with augmented spikes. By integrating pattern reconstruction and classification within a single framework, our approach not only accurately reconstructs multimodal sensory information but also provides precise classification through definitive labeling. Experimental evaluations conducted on various datasets including video scenes, static images, dynamic auditory scenes, and functional magnetic resonance imaging (fMRI) brain activities demonstrate that our framework delivers state-of-the-art pattern reconstruction quality and classification accuracy. The proposed framework enhances the biological realism of multimodal pattern recognition models, offering insights into how the primate brain visual system effectively accomplishes the reconstruction and classification tasks through the integration of ventral and dorsal pathways.

Multisensory input modulates memory-guided spatial navigation in humans

Efficient navigation is supported by a cognitive map of space. The hippocampus plays a key role for this map by linking multimodal sensory information with spatial memory representations. However, in human navigation studies, the full range of sensory information is often unavailable due to the stationarity of experimental setups. We investigated the contribution of multisensory information to memory-guided spatial navigation by presenting a virtual version of the Morris water maze on a screen and in an immersive mobile virtual reality setup. Patients with hippocampal lesions and matched controls navigated to memorized object locations in relation to surrounding landmarks. Our results show that availability of multisensory input improves memory-guided spatial navigation in both groups. It has distinct effects on navigational behaviour, with greater improvement in spatial memory performance in patients. We conclude that congruent multisensory information shifts computations to extrahippocampal areas that support spatial navigation and compensates for spatial navigation deficits.

A multimodal cortical network of sensory expectation violation revealed by fMRI.

The brain is subjected to multi-modal sensory information in an environment governed by statistical dependencies. Mismatch responses (MMRs), classically recorded with EEG, have provided valuable insights into the brain's processing of regularities and the generation of corresponding sensory predictions. Only few studies allow for comparisons of MMRs across multiple modalities in a simultaneous sensory stream and their corresponding cross-modal context sensitivity remains unknown. Here, we used a tri-modal version of the roving stimulus paradigm in fMRI to elicit MMRs in the auditory, somatosensory and visual modality. Participants (N = 29) were simultaneously presented with sequences of low and high intensity stimuli in each of the three senses while actively observing the tri-modal input stream and occasionally reporting the intensity of the previous stimulus in a prompted modality. The sequences were based on a probabilistic model, defining transition probabilities such that, for each modality, stimuli were more likely to repeat (p = .825) than change (p = .175) and stimulus intensities were equiprobable (p = .5). Moreover, each transition was conditional on the configuration of the other two modalities comprising global (cross-modal) predictive properties of the sequences. We identified a shared mismatch network of modality general inferior frontal and temporo-parietal areas as well as sensory areas, where the connectivity (psychophysiological interaction) between these regions was modulated during mismatch processing. Further, we found deviant responses within the network to be modulated by local stimulus repetition, which suggests highly comparable processing of expectation violation across modalities. Moreover, hierarchically higher regions of the mismatch network in the temporo-parietal area around the intraparietal sulcus were identified to signal cross-modal expectation violation. With the consistency of MMRs across audition, somatosensation and vision, our study provides insights into a shared cortical network of uni- and multi-modal expectation violation in response to sequence regularities.

Possible rodent equivalent of the posterior cingulate cortex (area 23) interconnects with multimodal cortical and subcortical regions.

Posterior cingulate cortex (area 23, A23) in human and monkeys is a critical component of the default mode network and is involved in many diseases such as Alzheimer's disease, autism, depression, attention deficit hyperactivity disorder and schizophrenia. However, A23 has not yet identified in rodents, and this makes modeling related circuits and diseases in rodents very difficult. Using a comparative approach, molecular markers and unique connectional patterns this study has uncovered the location and extent of possible rodent equivalent (A23~) of the primate A23. A23 ~ but not adjoining areas in the rodents displays strong reciprocal connections with anteromedial thalamic nucleus. Rodent A23 ~ reciprocally connects with the medial pulvinar and claustrum as well as with anterior cingulate, granular retrosplenial, medial orbitofrontal, postrhinal, and visual and auditory association cortices. Rodent A23 ~ projects to dorsal striatum, ventral lateral geniculate nucleus, zona incerta, pretectal nucleus, superior colliculus, periaqueductal gray, and brainstem. All these findings support the versatility of A23 in the integration and modulation of multimodal sensory information underlying spatial processing, episodic memory, self-reflection, attention, value assessment and many adaptive behaviors. Additionally, this study also suggests that the rodents could be used to model monkey and human A23 in future structural, functional, pathological, and neuromodulation studies.

Soft Robotic Perception System with Ultrasonic Auto-Positioning and Multimodal Sensory Intelligence

Flexible electronics such as tactile cognitive sensors have been broadly adopted in soft robotic manipulators to enable human-skin-mimetic perception. To achieve appropriate positioning for randomly distributed objects, an integrated guiding system is inevitable. Yet the conventional guiding system based on cameras or optical sensors exhibits limited environment adaptability, high data complexity, and low cost effectiveness. Herein, a soft robotic perception system with remote object positioning and a multimodal cognition capability is developed by integrating an ultrasonic sensor with flexible triboelectric sensors. The ultrasonic sensor is able to detect the object shape and distance by reflected ultrasound. Thereby the robotic manipulator can be positioned to an appropriate position to perform object grasping, during which the ultrasonic and triboelectric sensors can capture multimodal sensory information such as object top profile, size, shape, hardness, material, etc. These multimodal data are then fused for deep-learning analytics, leading to a highly enhanced accuracy in object identification (∼100%). The proposed perception system presents a facile, low-cost, and effective methodology to integrate positioning capability with multimodal cognitive intelligence in soft robotics, significantly expanding the functionalities and adaptabilities of current soft robotic systems in industrial, commercial, and consumer applications.

Multimodal Neuromorphic Sensory-Processing System With Memristor Circuits for Smart Home Applications

To enable smart home applications, embedding home monitoring systems with different sensors can be a feasible remedy to capture the multimodal sensory information from daily life. However, the previously developed monitoring systems have high implementation cost, large power consumption, and complicated device configuration, which are not conducive to achieving carbon neutrality and carbon peak goals. Here, we demonstrate a multimodal neuromorphic sensory-processing system with memristor circuits for smart homes, offering a more environmentally friendly approach with low cost and easily deployable hardware. We used three components to facilitate the multimodal neuromorphic sensory-processing system design. First, a memristor crossbar array is fabricated using low-cost, reliable, and eco-friendly 2-D materials; enabling a parallel connection to reduce the system complexity and the deep-learning computational cost. Second, a neuromorphic multimodal sensory-processing module is designed for collecting and processing multiple sensory cues efficiently; to establish accurate depiction of the environment and realize timely response. Third, the biomimetic hierarchical learning module is realized for smart home monitoring and relative applications (e.g., indoor human behavior recognition). The developed multimodal neuromorphic sensory-processing system effectively combines the nanotechnology, intelligent sensing, and deep learning; indicating an advancement in smart home applications.

Fusion of tactile and visual information in deep learning models for object recognition

Humans use multimodal sensory information to understand the physical properties of their environment. Intelligent decision-making systems such as the ones used in robotic applications could also utilize the fusion of multimodal information to improve their performance and reliability. In recent years, machine learning and deep learning methods are used at the heart of such intelligent systems. Developing visuo-tactile models is a challenging task due to various problems such as performance, limited datasets, reliability, and computational efficiency. In this research, we propose four efficient models based on dynamic neural network architectures for unimodal and multimodal object recognition. For unimodal object recognition, TactileNet and VisionNet are proposed. For multimodal object recognition, the FusionNet-A and the FusionNet-B are designed to implement early and late fusion strategies, respectively. The proposed models have a flexible structure and are able to change at the train or test phase to accommodate the amount of available information. Model confidence calibration is employed to enhance the reliability and generalization of the models. The proposed models are evaluated on MIT CSAIL large-scale multimodal dataset. Our results demonstrate accurate performance in both unimodal and multimodal scenarios. It has been illustrated that by using different fusion strategies and augmenting the tactile-based models with visual information, the top-1 error rate of the single-frame tactile model was reduced by 78% and the mean average precision was increased by 2.19 times. Although the focus has been on the fusion of tactile and visual modalities, the proposed design methodology can be generalized to include more modalities.

Passive somatosensory training enhances piano skill in adolescent and adult pianists: A preliminary study.

Sensory afferent information, such as auditory and somatosensory feedback while moving, plays a crucial role in both control and learning of motor performance across the lifespan. Music performance requires skillful integration of multimodal sensory information for the production of dexterous movements. However, it has not been understood what roles somatosensory afferent information plays in the acquisition and sophistication of specialized motor skills of musicians across different stages of development. In the present preliminary study, we addressed this issue by using a novel technique with a hand exoskeleton robot that can externally move the fingers of pianists. Short-term exposure to fast and complex finger movements generated by the exoskeleton (i.e., passive movements) increased the maximum rate of repetitive piano keystrokes by the pianists. This indicates that somatosensory inputs derived from the externally generated motions enhanced the quickness of the sequential finger movements in piano performance, even though the pianists did not voluntarily move the fingers. The enhancement of motor skill through passive somatosensory training using the exoskeleton was more pronounced in adolescent pianists than adult pianists. These preliminary results implicate a sensitive period of neuroplasticity of the somatosensory-motor system of trained pianists, which emphasizes the importance of somatosensory-motor training in professional music education during adolescence.

Processing of sensory, painful and vestibular stimuli in the thalamus

ObjectivesThe thalamus plays an important role in the mediation and integration of various stimuli (e.g., somatosensory, pain, and vestibular). Whether a stimulus-specific and topographic organization of the thalamic nuclei exists is still unknown. The aim of our study was to define a functional, in vivo map of multimodal sensory processing within the human thalamus.MethodsTwenty healthy individuals (10 women, 21–34 years old) participated. Defined sensory stimuli were applied to both hands (innocuous touch, mechanical pain, and heat pain) and the vestibular organ (galvanic stimulation) during 3 T functional MRI.ResultsBilateral thalamic activations could be detected for touch, mechanical pain, and vestibular stimulation within the left medio-dorsal and right anterior thalamus. Heat pain did not lead to thalamic activation at all. Stimuli applied to the left body side resulted in stronger activation patterns. Comparing an early with a late stimulation interval, the mentioned activation patterns were far more pronounced within the early stimulation interval.ConclusionsThe right anterior and ventral-anterior nucleus and the left medio-dorsal nucleus appear to be important for the processing of multimodal sensory information. In addition, galvanic stimulation is processed more laterally compared to mechanical pain. The observed changes in activity within the thalamic nuclei depending on the stimulation interval suggest that the stimuli are processed in a thalamic network rather than a distinct nucleus. In particular, the vestibular network within the thalamus recruits bilateral nuclei, rendering the thalamus an important integrative structure for vestibular function.

A Bayesian model for chronic pain.

The perceiving mind constructs our coherent and embodied experience of the world from noisy, ambiguous and multi-modal sensory information. In this paper, we adopt the perspective that the experience of pain may similarly be the result of a probabilistic, inferential process. Prior beliefs about pain, learned from past experiences, are combined with incoming sensory information in a Bayesian manner to give rise to pain perception. Chronic pain emerges when prior beliefs and likelihoods are biased towards inferring pain from a wide range of sensory data that would otherwise be perceived as harmless. We present a computational model of interoceptive inference and pain experience. It is based on a Bayesian graphical network which comprises a hidden layer, representing the inferred pain state; and an observable layer, representing current sensory information. Within the hidden layer, pain states are inferred from a combination of priors , transition probabilities between hidden states and likelihoods of certain observations . Using variational inference and free-energy minimization, the model is able to learn from observations over time. By systematically manipulating parameter settings, we demonstrate that the model is capable of reproducing key features of both healthy- and chronic pain experience. Drawing on mathematical concepts, we finally simulate treatment resistant chronic pain and discuss mathematically informed treatment options.

Robust Transcoding Sensory Information With Neural Spikes.

Neural coding, including encoding and decoding, is one of the key problems in neuroscience for understanding how the brain uses neural signals to relate sensory perception and motor behaviors with neural systems. However, most of the existed studies only aim at dealing with the continuous signal of neural systems, while lacking a unique feature of biological neurons, termed spike, which is the fundamental information unit for neural computation as well as a building block for brain-machine interface. Aiming at these limitations, we propose a transcoding framework to encode multi-modal sensory information into neural spikes and then reconstruct stimuli from spikes. Sensory information can be compressed into 10% in terms of neural spikes, yet re-extract 100% of information by reconstruction. Our framework can not only feasibly and accurately reconstruct dynamical visual and auditory scenes, but also rebuild the stimulus patterns from functional magnetic resonance imaging (fMRI) brain activities. More importantly, it has a superb ability of noise immunity for various types of artificial noises and background signals. The proposed framework provides efficient ways to perform multimodal feature representation and reconstruction in a high-throughput fashion, with potential usage for efficient neuromorphic computing in a noisy environment.

Lateral Habenula Beyond Avoidance: Roles in Stress, Memory, and Decision-Making With Implications for Psychiatric Disorders.

In this Perspective review, we highlight some of the less explored aspects of lateral habenula (LHb) function in contextual memory, sleep, and behavioral flexibility. We provide evidence that LHb is well-situated to integrate different internal state and multimodal sensory information from memory-, stress-, motivational-, and reward-related circuits essential for both survival and decision making. We further discuss the impact of early life stress (ELS) on LHb function as an example of stress-induced hyperactivity and dysregulation of neuromodulatory systems within the LHb that promote anhedonia and motivational deficits following ELS. We acknowledge that recent technological advancements in manipulation and recording of neural circuits in simplified and well-controlled behavioral paradigms have been invaluable in our understanding of the critical role of LHb in motivation and emotional regulation as well as the involvement of LHb dysfunction in stress-induced psychopathology. However, we also argue that the use of ethologically-relevant behaviors with consideration of complex aspects of decision-making is warranted for future studies of LHb contributions in a wide range of psychiatric illnesses. We conclude this Perspective with some of the outstanding issues for the field to consider where a multi-systems approach is needed to investigate the complex nature of LHb circuitry interactions with environmental stimuli that predisposes psychiatric disorders.

Multimodality in VR: A Survey

Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive to create a unified perception of the virtual world. In this survey, we review the body of work addressing multimodality in VR and its role and benefits in user experience, together with different applications that leverage multimodality in many disciplines. These works thus encompass several fields of research and demonstrate that multimodality plays a fundamental role in VR, enhancing the experience, improving overall performance, and yielding unprecedented abilities in skill and knowledge transfer.

Idiosyncratic multisensory reweighting as the common cause for motion sickness susceptibility and adaptation to postural perturbation.

Numerous empirical and modeling studies have been done to find a relationship between postural stability and the susceptibility to motion sickness (MS). However, while the demonstration of a causal relationship between postural stability and the susceptibility to MS is still lacking, recent studies suggest that motion sick individuals have genuine deficits in selecting and reweighting multimodal sensory information. Here we investigate how the adaptation to changing postural situations develops and how the dynamics in multisensory integration is modulated on an individual basis along with MS susceptibility. We used a postural task in which participants stood on a posturographic platform with either eyes open (EO) or eyes closed (EC) during three minutes. The platform was static during the first minute (baseline phase), oscillated harmonically during the second minute (perturbation phase) and returned to its steady state for the third minute (return phase). Principal component (PC) analysis was applied to the sequence of short-term power density spectra of the antero-posterior position of the center of pressure. Results showed that the less motion-sick a participant is, the more similar is his balance between high and low frequencies for EO and EC conditions (as calculated from the eigenvector of the first PC). By fitting exponential decay models to the first PC score in the return phase, we estimated, for each participant in each condition, the sluggishness to return to the baseline spectrum. We showed that the de-adaptation following platform oscillation depends on the susceptibility to MS. These results suggest that non motion-sick participants finely adjust their spectrum in the perturbation phase (i.e. reweighting) and therefore take longer to return to their initial postural control particularly with eyes closed. Thus, people have idiosyncratic ways of doing sensory reweighting for postural control, these processes being tied to MS susceptibility.

Effects of floor pattern on flight behaviour of the smaller tea tortrix, Adoxophyes honmai, during orientation flight in a sex pheromone plume

AbstractInsects control their flight toward a distant odour source by integrating multimodal sensory information, such as olfactory and visual information. Control of flight speed in orientation flight toward an odour source using visual patterning below has been observed in many species of moths. However, there are few reports about moth flight behaviour over solid‐coloured floors. To examine, which visual stimuli affect flight behaviour of moths, male smaller tea tortrix (Adoxophyes honmai) were released in a pheromone plume over two striped floors (longitudinal and transverse stripes) and two unpatterned floors (solid white and solid black). Flight behaviours were videotaped, and upwind flight speed, height (altitude) and the number of backward movements moths made as they approached the pheromone source were determined. Male moths significantly increased mean and maximum upwind flight speed over longitudinal stripes relative to their speed over transverse stripes. Their speed over a solid black floor was the fastest of all floor patterns tested. As males approached the pheromone source, their mean upwind speed decreased significantly over all floor patterns, except for the black floor. Fewer moths made backward movements over the black floor compared with the other three floors. In addition, male moths flew significantly higher over the white floor than over the other three floors, while 60–40 cm from the pheromone source. This is the first time that differences in moth flight behaviour over two patternless floors have been reported. These results suggest that orientation flight behaviour of male moths is affected not only by visual patterns, but also by the reflectance or reflected wavelength spectrum of the floor.