Cross-modal Sensory Integration Research Articles

A rodent obstacle course procedure controls delivery of enrichment and enhances complex cognitive functions

Enrichment in rodents affects brain structure, improves behavioral performance, and is neuroprotective. Similarly, in humans, according to the cognitive reserve concept, enriched experience is functionally protective against neuropathology. Despite this parallel, the ability to translate rodent studies to human clinical situations is limited. This limitation is likely due to the simple cognitive processes probed in rodent studies and the inability to control, with existing methods, the degree of rodent engagement with enrichment material. We overcome these two difficulties with behavioral tasks that probe, in a fine-grained manner, aspects of higher-order cognition associated with deterioration with aging and dementia, and a new enrichment protocol, the ‘Obstacle Course’ (OC), which enables controlled enrichment delivery, respectively. Together, these two advancements will enable better specification (and comparisons) of the nature of impairments in animal models of complex mental disorders and the potential for remediation from various types of intervention (e.g., enrichment, drugs). We found that two months of OC enrichment produced substantial and sustained enhancements in categorization memory, perceptual object invariance, and cross-modal sensory integration in mice. We also tested mice on behavioral tasks previously shown to benefit from traditional enrichment: spontaneous object recognition, object location memory, and pairwise visual discrimination. OC enrichment improved performance relative to standard housing on all six tasks and was in most cases superior to conventional home-cage enrichment and exercise track groups.

The Bouba-Kiki Effect in Children With Childhood Apraxia of Speech.

The bouba-kiki (BK) effect refers to associations between visual shapes and auditory pseudonames. Thus, when tested, people tend to associate the pseudowords bouba and kiki with round or spiky shapes, respectively. This association requires cross-modal sensory integration. The ability to integrate information from different sensory modalities is crucial for speech development. A clinical population that may be impaired in cross-modal sensory integration is children with childhood apraxia of speech (CAS). The purpose of this study was to examine the involvement of cross-modal sensory integration in children with (CAS). The BK effect was assessed in participants with CAS (n = 18) and two control groups: One control group was composed of children with developmental language disorder (DLD), also termed specific language impairment (n = 15), and a second group included typically developing (TD) children (n = 22). The children were presented with 14 pairs of novel visual displays and nonwords. All the children were asked to state which shape and nonword correspond to one another. In addition, background cognitive (Leiter-3) and language measures (Hebrew PLS-4) were determined for all children. Children in the CAS group were less successful in associating between visual shapes and corresponding auditory pseudonames (e.g., associating the spoken word "bouba" with a round shape; the spoken word "kiki" with a spiky shape). Thus, children with CAS demonstrated a statistically significant reduced BK effect compared with participants with TD and participants with DLD. No significant difference was found between the TD group and the DLD group. The reduced BK effect in children with CAS supports the notion that cross-modal sensory integration may be altered in these children. Cross-modal sensory integration is the basis for speech production. Thus, difficulties in sensory integration may contribute to speech difficulties in CAS.

An assistive computer vision tool to automatically detect changes in fish behavior in response to ambient odor

The analysis of fish behavior in response to odor stimulation is a crucial component of the general study of cross-modal sensory integration in vertebrates. In zebrafish, the centrifugal pathway runs between the olfactory bulb and the neural retina, originating at the terminalis neuron in the olfactory bulb. Any changes in the ambient odor of a fish’s environment warrant a change in visual sensitivity and can trigger mating-like behavior in males due to increased GnRH signaling in the terminalis neuron. Behavioral experiments to study this phenomenon are commonly conducted in a controlled environment where a video of the fish is recorded over time before and after the application of chemicals to the water. Given the subtleties of behavioral change, trained biologists are currently required to annotate such videos as part of a study. This process of manually analyzing the videos is time-consuming, requires multiple experts to avoid human error/bias and cannot be easily crowdsourced on the Internet. Machine learning algorithms from computer vision, on the other hand, have proven to be effective for video annotation tasks because they are fast, accurate, and, if designed properly, can be less biased than humans. In this work, we propose to automate the entire process of analyzing videos of behavior changes in zebrafish by using tools from computer vision, relying on minimal expert supervision. The overall objective of this work is to create a generalized tool to predict animal behaviors from videos using state-of-the-art deep learning models, with the dual goal of advancing understanding in biology and engineering a more robust and powerful artificial information processing system for biologists.

Sensory Integration: Cross-Modal Communication Between the Olfactory and Visual Systems in Zebrafish.

Cross-modal sensory communication is an innate biological process that refers to the combination and/or interpretation of different types of sensory input in the brain. Often, this process conjugates with neural modulation, by which the neural signals that convey sensory information are adjusted, such as intensity, frequency, complexity, and/or novelty. Although the anatomic pathways involved in cross-modal sensory integration have been previously described, the course of development and the physiological roles of multisensory signaling integration in brain functions remain to be elucidated. In this article, I review some of the recent findings in sensory integration from research using the zebrafish models. In zebrafish, cross-modal sensory integration occurs between the olfactory and visual systems. It is mediated by the olfacto-retinal centrifugal (ORC) pathway, which originates from the terminalis nerve (TN) in the olfactory bulb and terminates in the neural retina. In the retina, the TNs synapse with the inner nuclear layer dopaminergic interplexiform cells (DA-IPCs). Through the ORC pathway, stimulation of the olfactory neurons alters the cellular activity of TNs and DA-IPCs, which in turn modulates retinal neural function and increases behavioral visual sensitivity.

Modulation of sensory perception of cheese attributes intensity and texture liking via ortho- and retro-nasal odors

Cross-modal sensory integration plays a key role in food flavor perception and acceptance during consumption. The current study investigated the effect of a butter odor, delivered at various stages of the oral processing cycle, on modulating the sensory properties of cheese. Twenty healthy volunteers (aged between 25 and 29 years, 12 women) were measured for their detection thresholds for the butter odor. In the sensory evaluation sessions, participants chewed and swallowed three types of cheese (low-fat, 20% fat content, LF; a medium-fat, 30% fat content, MF; high-fat, 40% fat content, HF, served in 16 × 16 × 12 mm3 cubes) while the butter odor was presented ortho- and retronasally in two concentrations at various points of the oral processing cycle. After swallowing, participants rated on a visual analogue scale for the intensities of cheese creaminess, butter note, overall flavor, and the pleasantness for cheese texture. Enhancement of added butter odor on perceived sensory attributes differed as a function of the delivery routes and timings. Creaminess intensity increased significantly when butter odor presented retro-nasally at the start of chewing. Butter note was enhanced when the retro-nasal odor was added during chewing. The texture pleasantness was increased with ortho-nasal odor presentation. In addition, for the creaminess intensity and texture liking enhancement, the observed effects were more pronounced with butter odor presentation at the lower concentration. Taken together, these findings suggested the importance of temporal congruency for cross-modal sensory enhancement in food flavor perception. The findings help to better understand flavor perception during oral processing of solid food and add value for future development of foods with nutritional benefits.

SU32. Impact of Catechol-O-Methyl Transferase Polymorrphism on Cognition and Information Processing in Schizophrenia Using Novel MBIAL Computerized Battery—A Case-Control Study

Abstract Background: Frontal executive dysfunction have been robustly demonstrated in schizophrenia. Several fundamental information processing deficits were reported in schizophrenia underlying the various symptom dimensions. Results of previous studies of neuropsychological functioning in schizophrenia were inconsistent. Association between COMT allele type and cognition was not confirmed in all studies. This study attempted to assess the neuropsychological performance on a hypothesis-driven MBIAL-computerized neuropsychological battery in a sample of schizophrenia subjects and matched healthy comparison subjects and its relationship with the gene polymorphism. Methods: Total 34 patients and 30 matched healthy controls were recruited after obtaining informed consent. An HMSE was applied to exclude those with scores <23. The matrix reasoning module of WAIS-III was applied to obtain an estimate of their perceptual organization index as a reflection of their intellectual abilities, independent of verbal abilities. A novel MBIAL Computerized neuropsychological battery consisting of 6 tests was applied on all the subjects. Later, both the groups were assessed for symptom severity using PANSS. All Neuropsychological tests and PANSS Interviews using SCI-PANSS were Video and Audio recorded. Genotyping for COMTval158met polymorphism was accomplished by PCR-RFLP method. Genotype frequencies across groups were noted using chi-square test and later were compared using ANCOVA as we had WAIS III and HMSE as covariates. Results: Schizophrenia patients performed significantly poorly on most of the neuropsychological tests as compared with healthy matched controls. Negative symptom severity was associated with significant impairment in neuropsychological deficits as compared to positive symptoms. Separate analysis of association of gene polymorphism on these cognitive tests done on patients and controls did not reveal much results. But on pooled data, we found Val-Carriers performed poorly on most of the Neuropsychological and Information processing tasks and have high threat perception compared to Non-Val Carriers. Conclusion: Hypothesis-driven neuropsychological battery was used in schizophrenia that incorporated tests for frontal executive functioning; rostral prefrontal functioning; threat perception (aberration of which is postulated to underlie delusion formation); and cross-modal sensory integration (aberration of which is postulated to underlie development of hallucinations). The MBIAL battery is sensitive in detecting differences between patients and controls in the adult population. We suggest here that Val158Met polymorphism rather exerts a modifying influence on brain activation in general than impacts directly on performance of the particular neuropsychological task.

Pharmacologic attenuation of cross-modal sensory augmentation within the chronic pain insula.

Pain can be elicited through all mammalian sensory pathways yet cross-modal sensory integration, and its relationship to clinical pain, is largely unexplored. Centralized chronic pain conditions such as fibromyalgia are often associated with symptoms of multisensory hypersensitivity. In this study, female patients with fibromyalgia demonstrated cross-modal hypersensitivity to visual and pressure stimuli compared with age- and sex-matched healthy controls. Functional magnetic resonance imaging revealed that insular activity evoked by an aversive level of visual stimulation was associated with the intensity of fibromyalgia pain. Moreover, attenuation of this insular activity by the analgesic pregabalin was accompanied by concomitant reductions in clinical pain. A multivariate classification method using support vector machines (SVM) applied to visual-evoked brain activity distinguished patients with fibromyalgia from healthy controls with 82% accuracy. A separate SVM classification of treatment effects on visual-evoked activity reliably identified when patients were administered pregabalin as compared with placebo. Both SVM analyses identified significant weights within the insular cortex during aversive visual stimulation. These data suggest that abnormal integration of multisensory and pain pathways within the insula may represent a pathophysiological mechanism in some chronic pain conditions and that insular response to aversive visual stimulation may have utility as a marker for analgesic drug development.

Speed on the dance floor: Auditory and visual cues for musical tempo

Musical tempo is most strongly associated with the rate of the beat or “tactus,” which may be defined as the most prominent rhythmic periodicity present in the music, typically in a range of 1.67–2Hz. However, other factors such as rhythmic density, mean rhythmic inter-onset interval, metrical (accentual) structure, and rhythmic complexity can affect perceived tempo (Drake, Gros, & Penel, 1999; London, 2011Drake, Gros, & Penel, 1999; London, 2011). Visual information can also give rise to a perceived beat/tempo (Iversen, et al., 2015), and auditory and visual temporal cues can interact and mutually influence each other (Soto-Faraco & Kingstone, 2004; Spence, 2015). A five-part experiment was performed to assess the integration of auditory and visual information in judgments of musical tempo. Participants rated the speed of six classic R&B songs on a seven point scale while observing an animated figure dancing to them. Participants were presented with original and time-stretched (±5%) versions of each song in audio-only, audio+video (A+V), and video-only conditions. In some videos the animations were of spontaneous movements to the different time-stretched versions of each song, and in other videos the animations were of “vigorous” versus “relaxed” interpretations of the same auditory stimulus. Two main results were observed. First, in all conditions with audio, even though participants were able to correctly rank the original vs. time-stretched versions of each song, a song-specific tempo-anchoring effect was observed, such that sped-up versions of slower songs were judged to be faster than slowed-down versions of faster songs, even when their objective beat rates were the same. Second, when viewing a vigorous dancing figure in the A+V condition, participants gave faster tempo ratings than from the audio alone or when viewing the same audio with a relaxed dancing figure. The implications of this illusory tempo percept for cross-modal sensory integration and working memory are discussed, and an “energistic” account of tempo perception is proposed.

Motor skill learning in children with Developmental Coordination Disorder

Children with Developmental Coordination Disorder (DCD) are characterized as having motor difficulties and learning impairment that may last well into adolescence and adulthood. Although behavioral deficits have been identified in many domains such as visuo-spatial processing, kinesthetic perception, and cross-modal sensory integration, recent studies suggested that the functional impairment of certain brain areas, such as cerebellum and basal ganglia, are the underlying causes of DCD. This review focuses on the "motor learning deficits" in DCD and their possible neural correlates. It presents recent evidence from both behavioral and neuroimaging studies and discusses dominant neural hypotheses in DCD. Given the heterogeneity of this disorder, a successful intervention program should target the specific deficits on an individual basis. Future neuroimaging studies are critical steps in enhancing our understanding of learning deficits in DCD.

The Autism Psychodynamic Evaluation of Changes (APEC) scale: A reliability and validity study on a newly developed standardized psychodynamic assessment for youth with Pervasive Developmental Disorders

The present study was designed to examine the reliability and validity of the Autism Psychodynamic Evaluation of Changes (APEC) scale, developed to assess the evolution in individuals with autism under treatment. The APEC scale focuses on the key role of impairment in body image construction, which requires cross-modal sensory integration through emotional communication with motor representations. Thus, the body image construction is associated simultaneously with spatial and temporal organization and allows the emergence of self- and others-representations. The use of the APEC scale, with its seven domains (expression of emotion in relationships, eye contact, body image, graphic productions, exploration of space and objects, time perception, and verbal language), underlines the importance in autistic disorder of anxieties related to body and spatial representations, and of impairment in the body ego construction which is closely linked to the emergence of individuation/separation processes. This study was conducted on 73 children and adolescents with autistic disorder. They were recruited in day care facilities where two caregivers independently gave their ratings based on their clinical observation on a daily basis during the same month. Analyses included assessing construct validity through correspondence analyses and inter-rater reliability using kappa coefficients. The APEC scale offers a reliable and validated psychodynamic assessment of interest for professionals (such as child psychiatrists, caregivers, therapists or teachers) and researchers working with children, adolescents and adults with autistic disorder, especially in the follow-up of their evolution. The APEC scale provides an approach at the interface of psychoanalysis and neuroscience, and is also of interest for clinical and developmental psychology. Using the APEC scale in a range of different practical and research settings will foster links between psychoanalytic perspectives and educational training for children with autistic disorder, and will contribute to the dialogue between psychoanalysis, neuroscience and psychology.

Early Cortical Connective Network Relating to Audiovisual Stimulation by Partial Directed Coherence Analysis

Cross-modal sensory integration between the auditory and visual systems has long been a topic of interest in cognitive neuroscience, but spatiotemporal cortical interdependence under audiovisual stimulation is still poorly understood. In this letter, parametric spectral analysis, specifically partial directed coherence (PDC), is used to study the cortical connective causal-interdependence network under audiovisual cognitive processes within the time window of the N1 component, i.e., 96--145 ms poststimulus in the event-related potential (ERP). A total of 13 subjects (age: 22.9+/-0.8 years; male/female = 9/4) performed an audiovisual recognition task. Multichannel EEG signals were analyzed using an adaptive multivariate autoregressive model to identify causal influences between different cortical areas. Three different cortical connective networks were detected, and by comparing the cortical PDC network in the early cognitive process of audiovisual (VA) perception with that in unimodal situations (V or A), it was found that the left temporal cortex (TC) might be responsible for audiovisual integration. Such electrophysiological findings were highly consistent with previous functional neuroimaging results. Moreover, the results presented here show the dynamics of lateral causal influences during audiovisual integration.

Morphology of the Insula in Relation to Hearing Status and Sign Language Experience

We investigated whether auditory deprivation and/or sign language exposure during development alters the macroscopic neuroanatomy of the human insula. Volumetric analyses were based on MRI data from 25 congenitally deaf subjects who were native users of American Sign Language (ASL), 25 hearing subjects with no knowledge of ASL, and 16 hearing subjects who grew up in deaf families and were native ASL signers. Significant variation in insula volume was associated with both hearing status and sign language experience. Compared with both hearing groups, deaf subjects exhibited a significant increase in the amount of gray matter in the left posterior insular lobule, which we hypothesize may be related to the dependence on lip-reading and articulatory-based (rather than auditory-based) representations of speech for deaf individuals. Both deaf and hearing signers exhibited an increased volume of white matter in the right insula compared with hearing nonsigners. We hypothesize that the distinct morphology of the right insula for ASL signers may arise from enhanced connectivity resulting from an increased reliance on cross-modal sensory integration in sign language compared with spoken language.

Spatial coding of visual and somatic sensory information in body-centred coordinates.

Because sensory systems use different spatial coordinate frames, cross-modal sensory integration and sensory-motor coordinate transformations must occur to build integrated spatial representations. Multimodal neurons using non-retinal body-centred reference frames are found in the posterior parietal and frontal cortices of monkeys. We used functional magnetic resonance imaging to reveal regions of the human brain using body-centred coordinates to code the spatial position of both visual and somatic sensory stimuli. Participants determined whether a visible vertical bar (visual modality) or a location touched by the right index finger (somatic sensory modality) lay to the left or to the right of their body mid-sagittal plane. This task was compared to a spatial control task having the same stimuli and motor responses and comparable difficulty, but not requiring body-centred coding of stimulus position. In both sensory modalities, the body-centred coding task activated a bilateral fronto-parietal network, though more extensively in the right hemisphere, to include posterior parietal regions around the intraparietal sulcus and frontal regions around the precentral and superior frontal sulci, the inferior frontal gyrus and the superior frontal gyrus on the medial wall. The occipito-temporal junction and other extrastriate regions exhibited bilateral activation enhancement related to body-centred coding when driven by visual stimuli. We conclude that posterior parietal and frontal regions of humans, as in monkeys, appear to provide multimodal integrated spatial representations in body-centred coordinates, and these data furnish the first indication of such processing networks in the human brain.

Patterns of sensory intermodality relationships in the cerebral cortex of the rat

Patterns of connections underlying cross-modality integration were studied by injecting distinguishable, retrograde tracers (Fluoro-Gold and diamidino yellow) in pairwise manner into different sensory representations (visual, somatosensory, and auditory) in the cerebral cortex of the rat. In agreement with previous single tracer studies, our results indicate that the central core of sensory areas receives projections mainly from a set of association areas located in a ringlike fashion along the margin of the cortical mantle. The visual cortex received projections from areas 48/49, area 29d, posterior agranular medial cortex (AGm), area 11, area 13, and area 35. All these areas were also connected to the auditory cortex with the exception of areas 29d and AGm. However, lateral to area 29d and posterior AGm, a band of neurons projecting to the auditory cortex was present. Somatosensory cortex was connected mainly with the more anterior aspect of the hemisphere, which included primary motor area, area 11, and area 13. The patterns of intermodality relationships revealed in the present study were of two main categories. In the anterior and lateral areas, an intermingling of cells projecting to different sensory modalities was observed. In contrast, in areas located along the medial aspect of the hemisphere, cells connected to different sensory modality representations tended to be segregated from each other. Postsubicular cortex (areas 48/49) contained both intermingled and segregated groups of cells. The incidence of clearly identified double-labeled cells concurrently projecting to two different sensory representations was extremely rare. These patterns may form a substrate for different levels of cross-modal sensory integration in the rat cortex.

Lateral preference patterns and cross-modal sensory integration.

In light of a recent hypothesis, the concordance between patterns of lateralpreference and sensory integration were examined. Specifically, the scoresfor 48 normal preadolescents on a multifactor measure of lateral preferenceand subtests of the Visual Aural Digit Span Test were interrelated. Theresults indicated a tendency for intersensory integration scores to decreaseas patterns of general lateral preference became more mixed and to increaseas foot preference became more stable. No reliable relationships were foundbetween lateral preference and intrasensory scores. The results wereinterpreted as tentative support for problems in visual-verbal integrationconsistent with more mixed patterns of lateral preference.

Intramodal and cross-modal sensory integration by crayfish optomotor neurons

1. 1. Action potentials from crayfish optomotor neurons were recorded extracellularly in situ during natural visual and gravitational stimulation. 2. 2. A small fraction of optomotor neurons apparently received input from statocyst afferents having sensitivities to body orientation in two different geometric planes. 3. 3. One type of optomotor neuron integrates statocyst and visual input in such a way that visual stimulation is much less excitatory at certain body positions than at other positions.