Multisensory Development Research Articles

Development of multisensory processing in ferret parietal cortex.

It is well known that the nervous system adjusts itself to its environment during development. Although a great deal of effort has been directed towards understanding the developmental processes of the individual sensory systems (e.g., vision, hearing, etc.), only one major study has examined the maturation of multisensory processing in cortical neurons. Therefore, the present investigation sought to evaluate multisensory development in a different cortical region and species. Using multiple single-unit recordings in anaesthetised ferrets (n = 18) of different ages (from postnatal day 80 to 300), we studied the responses of neurons from the rostral posterior parietal (PPr) area to presentations of visual, tactile and combined visual-tactile stimulation. The results showed that multisensory neurons were infrequent at the youngest ages (pre-pubertal) and progressively increased through the later ages. Significant response changes that result from multisensory stimulation (defined as multisensory integration [MSI]) were observed in post-pubertal adolescent animals, and the magnitude of these integrated responses also increased across this age group. Furthermore, non-significant multisensory response changes were progressively increased in adolescent animals. Collectively, at the population level, MSI was observed to shift from primarily suppressive levels in infants to increasingly higher levels in later stages. These data indicate that, like the unisensory systems from which it is derived, multisensory processing shows developmental changes whose specific time course may be regionally and species-dependent.

Effects of acoustic-phonetic access on audiovisual speech perception in children

Visual speech helps compensate for degraded acoustic input, especially when redundancy between the accessible acoustic and visual cues is low. Recent findings indicate that children who are hard of hearing benefit more from visual speech than their peers with normal hearing, yet the two groups demonstrate similar speechreading ability. It is unclear whether this increased benefit reflects differences in auditory and multisensory development or differences in redundancy between the visual and acoustic cues available to each group. This study examines the extent to which acoustic-phonetic access (acoustic frequency content) influences auditory and audiovisual word recognition in children with normal hearing. Isolated word and word-in-sentence recognition accuracy are measured for auditory-only and audiovisual speech that were high-pass filtered and low-pass filtered with a 2 kHz cutoff frequency. Data collection is ongoing. We hypothesize that children will demonstrate greater audiovisual benefit when acoustic speech is low-pass filtered than when it is high-pass filtered, because low-frequency acoustic content is less redundant with visual speech cues than high-frequency acoustic content. These findings will elucidate whether acoustic-phonetic access can explain differences in audiovisual benefit between children with normal hearing and children who are hard of hearing. [Work supported by NIH R21DC020544.]

Multisensory Perception and Learning: Linking Pedagogy, Psychophysics, and Human-Computer Interaction.

In this review, we discuss how specific sensory channels can mediate the learning of properties of the environment. In recent years, schools have increasingly been using multisensory technology for teaching. However, it still needs to be sufficiently grounded in neuroscientific and pedagogical evidence. Researchers have recently renewed understanding around the role of communication between sensory modalities during development. In the current review, we outline four principles that will aid technological development based on theoretical models of multisensory development and embodiment to foster in-depth, perceptual, and conceptual learning of mathematics. We also discuss how a multidisciplinary approach offers a unique contribution to development of new practical solutions for learning in school. Scientists, engineers, and pedagogical experts offer their interdisciplinary points of view on this topic. At the end of the review, we present our results, showing that one can use multiple sensory inputs and sensorimotor associations in multisensory technology to improve the discrimination of angles, but also possibly for educational purposes. Finally, we present an application, the 'RobotAngle' developed for primary (i.e., elementary) school children, which uses sounds and body movements to learn about angles.

Late development of audio-visual integration in the vertical plane

• Multisensory audio-visual integration develops later in typical children. • Visual calibration might be needed for auditory localization in the vertical plane. • Multisensory integration can anticipate calibration depending on task conditions. It is not clear how multisensory skills develop and how visual experience impacts on multisensory spatial development. Conflicting results show that visual calibration precedes multisensory integration for the audio-visual spatial bisection task (Gori et al., 2012a, 2012b) while in other tasks such as spatial localization, visual calibration occurs after multisensory development (Rohlf et al., 2020). Results in blind individuals can say something about the role of vision on perceptual development. Scientific evidences show that blind individuals have impairments in bisecting the auditory space (Gori et al., 2014) but not in localizing auditory sources (Lessard et al., 1998). Such results suggest that sensory calibration and impairment are linked. We studied the development of audio-visual multisensory localization in the vertical plane in sighted individuals from 5 years to adulthood to address this hypothesis. We hypothesize that typical children would show late audio-visual integration for the vertical plane, preceded by visual dominance. Unimodal and bimodal audio-visual thresholds and PSEs were measured and compared with the Bayesian optimal-integration model (maximum likelihood estimation). Results show that the development of multisensory integration in the vertical plane is not evident at 5 years, suggesting visual dominance for vertical audio-visual localization. These results support the idea that multisensory perception in the vertical domain depends on sensory calibration. We discuss these scientific results proposing that the process of cross-sensory calibration is task-specific and highlighting the importance of linking the impairment and development to better determine how our brain works.

The effectiveness of literacy stimulation model based on multisensory development of the results of DDST

This research aims to study the effectiveness of literacy stimulation model based on multisensory development of the results of DDST in PAUD Aster 36, Jember. This type of research is Quasi-experimental design which is a form of experimental research that uses a control group but the control group unfunctionally to control external variables that can influence it. DDST assessment was carried out before and after treatment both in children who were given literacy stimulation or not. The research instrument used was the REEDA sign observation sheet with SOP tools. Respondent distribution data included gender, toddler age, toddler parent education, toddler parent age, toddler mother's work, toddler body weight, toddler head circumference and toddler upper arm circumference size. The result is the development of toddler overall assessment using DDST before treatment belongs to the normal category with different percentages. Development of toddler overall assessment using DDST After treatment was included in the normal category, but there was no statistical difference (p-value = 0.083) between the control and treatment groups.

Early Sensory Loss Alters the Dendritic Branching and Spine Density of Supragranular Pyramidal Neurons in Rodent Primary Sensory Cortices.

Multisensory integration in primary auditory (A1), visual (V1), and somatosensory cortex (S1) is substantially mediated by their direct interconnections and by thalamic inputs across the sensory modalities. We have previously shown in rodents (Mongolian gerbils) that during postnatal development, the anatomical and functional strengths of these crossmodal and also of sensory matched connections are determined by early auditory, somatosensory, and visual experience. Because supragranular layer III pyramidal neurons are major targets of corticocortical and thalamocortical connections, we investigated in this follow-up study how the loss of early sensory experience changes their dendritic morphology. Gerbils were sensory deprived early in development by either bilateral sciatic nerve transection at postnatal day (P) 5, ototoxic inner hair cell damage at P10, or eye enucleation at P10. Sholl and branch order analyses of Golgi-stained layer III pyramidal neurons at P28, which demarcates the end of the sensory critical period in this species, revealed that visual and somatosensory deprivation leads to a general increase of apical and basal dendritic branching in A1, V1, and S1. In contrast, dendritic branching, particularly of apical dendrites, decreased in all three areas following auditory deprivation. Generally, the number of spines, and consequently spine density, along the apical and basal dendrites decreased in both sensory deprived and non-deprived cortical areas. Therefore, we conclude that the loss of early sensory experience induces a refinement of corticocortical crossmodal and other cortical and thalamic connections by pruning of dendritic spines at the end of the critical period. Based on present and previous own results and on findings from the literature, we propose a scenario for multisensory development following early sensory loss.

Som tridimensional para deficientes visuais

Neste artigo são apresentadas algumas discussões teóricas que embasam uma das pesquisas em desenvolvimento no designlab/UFSC. O objetivo principal é investigar a construção, de um dispositivo sonoro que facilite na localização e locomoção de deficientes visuais. Parte-se da tridimensionalidade sonora e de seus princípios históricos nas vanguardas artísticas, bem como, alguns seus desdobramentos na arte contemporânea. Na pesquisa são relacionados estudos a respeito do desenvolvimento social e multissensorial do público alvo, com Vygostsky (1997) e (2002) e experiências da arte sonora, a partir de Luigi Russolo (1992) e (1996), John Cage (1985) e (2007) e R. Murray Schafer (1991) e (2001). Por fim, são introduzidos procedimentos técnicos necessários para a construção do protótipo.

The role of multisensory development in early language learning

In typical development, communicative skills such as language emerge from infants’ ability to combine multisensory information into cohesive percepts. For example, the act of associating the visual or tactile experience of an object with its spoken name is commonly used as a measure of early word learning, and social attention and speech perception frequently involve integrating both visual and auditory attributes. Early perspectives once regarded perceptual integration as one of infants’ primary challenges, whereas recent work suggests that caregivers’ social responses contain structured patterns that may facilitate infants’ perception of multisensory social cues. In the current review, we discuss the regularities within caregiver feedback that may allow infants to more easily discriminate and learn from social signals. We focus on the statistical regularities that emerge in the moment-by-moment behaviors observed in studies of naturalistic caregiver–infant play. We propose that the spatial form and contingencies of caregivers’ responses to infants’ looks and prelinguistic vocalizations facilitate communicative and cognitive development. We also explore how individual differences in infants’ sensory and motor abilities may reciprocally influence caregivers’ response patterns, in turn regulating and constraining the types of social learning opportunities that infants experience across early development. We end by discussing implications for neurodevelopmental conditions affecting both multisensory integration and communication (i.e., autism) and suggest avenues for further research and intervention.

Development of the Mechanisms Governing Midbrain Multisensory Integration.

The ability to integrate information across multiple senses enhances the brain's ability to detect, localize, and identify external events. This process has been well documented in single neurons in the superior colliculus (SC), which synthesize concordant combinations of visual, auditory, and/or somatosensory signals to enhance the vigor of their responses. This increases the physiological salience of crossmodal events and, in turn, the speed and accuracy of SC-mediated behavioral responses to them. However, this capability is not an innate feature of the circuit and only develops postnatally after the animal acquires sufficient experience with covariant crossmodal events to form links between their modality-specific components. Of critical importance in this process are tectopetal influences from association cortex. Recent findings suggest that, despite its intuitive appeal, a simple generic associative rule cannot explain how this circuit develops its ability to integrate those crossmodal inputs to produce enhanced multisensory responses. The present neurocomputational model explains how this development can be understood as a transition from a default state in which crossmodal SC inputs interact competitively to one in which they interact cooperatively. Crucial to this transition is the operation of a learning rule requiring coactivation among tectopetal afferents for engagement. The model successfully replicates findings of multisensory development in normal cats and cats of either sex reared with special experience. In doing so, it explains how the cortico-SC projections can use crossmodal experience to craft the multisensory integration capabilities of the SC and adapt them to the environment in which they will be used.SIGNIFICANCE STATEMENT The brain's remarkable ability to integrate information across the senses is not present at birth, but typically develops in early life as experience with crossmodal cues is acquired. Recent empirical findings suggest that the mechanisms supporting this development must be more complex than previously believed. The present work integrates these data with what is already known about the underlying circuit in the midbrain to create and test a mechanistic model of multisensory development. This model represents a novel and comprehensive framework that explains how midbrain circuits acquire multisensory experience and reveals how disruptions in this neurotypic developmental trajectory yield divergent outcomes that will affect the multisensory processing capabilities of the mature brain.

Multisensory Development: Calibrating a Coherent Sensory Milieu in Early Life

A new study reveals the effects of visual deprivation in early life on the development of multisensory simultaneity perception. To understand the developmental processes underlying this we need to consider the multisensory milieu of the newborn infant.

Introduction to the Special Issue on Multisensory Development and Plasticity.

Introduction to the Special Issue on Multisensory Development and Plasticity.

Uni- and crossmodal refractory period effects of event-related potentials provide insights into the development of multisensory processing.

To assess uni- and multisensory development in humans, uni- and crossmodal event-related potential (ERP) refractory period effects were investigated. Forty-one children from 4 to 12 years of age and 15 young adults performed a bimodal oddball task with frequent and rare visual and auditory stimuli presented with two different interstimulus intervals (ISIs). Amplitudes of the visual and auditory ERPs were modulated as a function of the age of the participants, the modality of the preceding stimulus (same vs. different) and the preceding ISI (1000 or 2000 ms). While unimodal refractory period effects were observed in all age groups, crossmodal refractory period effects differed among age groups. Early crossmodal interactions (<150 ms) existing in the youngest age group (4–6 years) disappeared, while later crossmodal interactions (>150 ms) emerged with a parietal topography in older children and adults. Our results are compatible with the intersensory differentiation and the multisensory perceptual narrowing approach of multisensory development. Moreover, our data suggest that uni- and multisensory development run in parallel with unimodal development leading.

Toward a vestibular contribution to social cognition

Social cognition encompasses perception of self and others as well as self-other interactions. Self-other interactions rely on a wide range of cognitive processes such as memory, language, reasoning, and emotion processing (Beer and Ochsner, 2006). Within the last few years, one productive line of research in social neuroscience has investigated the multisensory and motor foundations of self-other interactions, including emotion perception, emotional contagion, empathy, self-other distinction, or self-other knowledge (e.g., Singer et al., 2004; Iacoboni et al., 2005; Ambrosini et al., 2013; Manera et al., 2013). While a strong emphasis has been traditionally put on visual processes, recent research has noted that self-other distinction and mirroring also require processing of auditory (e.g., self-other voice recognition), proprioceptive, and interoceptive signals (Damasio, 2000; Schutz-Bosbach et al., 2006; Tsakiris et al., 2011; Seth, 2013; Xu et al., 2013). In spite of this multisensory development, a vestibular contribution to the embodied mechanisms of social interactions has until now been largely overlooked. This is surprising as the vestibular system has been involved in a growing number of cognitive functions (Smith et al., 2005; Miller and Ngo, 2007; Gurvich et al., 2013), in addition to its crucial role in distinguishing self- and non-self motion. The claim of the present opinion article is that vestibular information should not be ignored when investigating the sensorimotor foundations of social cognition. We present several lines of evidence indicating that vestibular signals may be involved in the sensory bases of self-other distinction and mirroring, emotion perception and perspective taking.

Review: Multisensory Development, How Vision Works: The Physiological Mechanisms behind What We See

Review: Multisensory Development, How Vision Works: The Physiological Mechanisms behind What We See

Children’s Responses to the Rubber-Hand Illusion Reveal Dissociable Pathways in Body Representation

The bodily self is constructed from multisensory information. However, little is known of the relation between multisensory development and the emerging sense of self. We investigated this question by measuring the strength of the rubber-hand illusion in young children (4 to 9 years old) and adults. Intermanual pointing showed that children were as sensitive as adults to visual-tactile synchrony cues for hand position, which indicates that a visual-tactile pathway to the bodily self matures by at least 4 years of age. However, regardless of synchrony cues, children’s perceived hand position was closer to the rubber hand than adults’ perceived hand position was. This indicates a second, later-maturing process based on visual-proprioceptive information. Furthermore, explicit feelings of embodiment were related only to the visual-tactile process. These findings demonstrate two dissociable processes underlying body representation in early life, and they call into question current models of body representation and ownership in adulthood.

Questioning the utility of the concept of amodality: Towards a revised framework for understanding crossmodal relations

The idea that sensory experiences should be distinguished into modal and amodal (in terms of crossmodal redundancy, rather than amodal completion) runs through many areas of psychology; developmental psychologists, in particular, appear to hold opposing views about whether infants start by having modally distinct or amodal experiences of the properties that are commonly considered to be amodal, such as space, duration, and intensity. Many other phenomena in adults — such as ‘crossmodal correspondences’ (Spence, 2011) — are deemed interesting, in part, precisely because they are not simply picking-up on amodal properties or redundantly-coded features. However, while many researchers have, over the years, used the term ‘amodal’, we would argue that there is a great deal of uncertainty over which dimensions of experience should be considered as amodal, and according to which criteria. Here, we argue that virtually nothing is amodal. To the extent that this claim holds, it may help to shed further light on early multisensory development while at the same time calling into question various theoretical accounts about both the acquisition of crossmodal relations in early development and the acquisition of novel associations (or correspondences) in adulthood. This discussion may also help to elucidate the kinds of evidence that can legitimately be taken to support arguments concerning the existence, or otherwise, of neonatal synaesthesia (see Deroy and Spence, submitted).

Basic Multisensory Functions Can Be Acquired After Congenital Visual Pattern Deprivation in Humans

People treated for bilateral congenital cataracts offer a model to study the influence of visual deprivation in early infancy on visual and multisensory development. We investigated cross-modal integration capabilities in cataract patients using a simple detection task that provided redundant information to two different senses. In both patients and controls, redundancy gains were consistent with coactivation models, indicating an integrated processing of modality-specific information. This finding is in contrast with recent studies showing impaired higher-level multisensory interactions in cataract patients. The present results suggest that basic cross-modal integrative processes for simple short stimuli do not depend on visual and/or crossmodal input since birth.

The development of the bodily self: Children’s responses to the Rubber Hand Illusion

The bodily self is constructed from multisensory information. However, little is known of the relationship between multisensory development and the emerging sense of self. We investigated how multisensory processes underpin the developing sense of bodily self by measuring the strength of the ‘Rubber Hand Illusion’ in young children (4 to 9 years old) and adults. Stroking mode (synchronous vs asynchronous) was varied between subjects. Following this visuotactile stimulation, we measured processes of body localisation using intermanual pointing, and feelings of body ownership using questionnaire responses. Intermanual pointing showed that children were as sensitive as adults to visual-tactile synchrony cues for hand position. Similarly, explicit feelings of embodiment, measured by questionnaire items, were sensitive to visual-tactile integration independent of age. These converging results indicate a visual-tactile pathway to the bodily self which matures by at least 4 years of age. However, regardless of synchrony cues, children’s pointing was more captured by the fake hand than adults, indicating a second, later-maturing process based on visual-proprioceptive information. These findings demonstrate two dissociable processes underlying body representation in early life: an early-maturing visual tactile process controlling hand localisation and ownership, and a later-developing visual-proprioceptive process controlling localisation only. The findings further suggest that hand localisation and ownership may not always be so intimately bound together as adult data would suggest, and therefore call for a reassessment of the systems involved in adult own-body perception.

The critical role of experience in the early development of multisensory perception

Human infancy is a time of rapid neural and behavioral development and multisensory perceptual skills emerge during this time. Both animal and human early deprivation studies have shown that experience contributes critically to the development of multisensory perception. Unfortunately, Bodison because the human deprivation studies have only studied adult responsiveness, little is known about the more immediate effects of early experience on multisensory development. Consequently, we have embarked on a program of research to investigate how early experience affects the development of multisensory perception in human infants. To do so, we have focused on multisensory perceptual narrowing, an experience-dependent process where initially broad perceptual tuning is narrowed to match the infant’s native environment. In this talk, I first review our work demonstrating that multisensory narrowing characterizes infants’ response to non-native (i.e., monkey) faces and voices, that the initially broad tuning is present at birth, that narrowing also occurs in the audiovisual speech domain, and that multisensory narrowing is an evolutionarily novel process. In the second part of the talk, I present findings from our most recent studies indicating that experience has a seemingly paradoxical effect on infant response to audio–visual synchrony, that experience narrows infant response to amodal language and intonational prosody cues, and that experience interacts with developmental changes in selective attention during the first year of life resulting in dramatic developmental shifts in human infants’ selective attention to the eyes and mouth of their interlocutors’ talking faces.

Restricted recovery of external remapping of tactile stimuli after restoring vision in a congenitally blind man

People with surgically removed congenital dense bilateral cataracts offer a natural model of visual deprivation and reafferentation in humans to investigate sensitive periods of multisensory development, for example regarding the recruitment of external or anatomical frames of reference for spatial representation. Here we present a single case (HS; male; 33 years; right-handed), born with congenital dense bilateral cataracts. His lenses were removed at the age of two years, but he received optical aids only at age six. At time of testing, his visual acuity was 30% in the best eye. We performed two tasks, a tactile temporal order judgment task (TOJ) in which two tactile stimuli were presented successively to the index fingers located in the two hemifields, adopting a crossed and uncrossed hand posture. The participant judged as precisely as possible which side was stimulated first. Moreover, we used a crossmodal-congruency task in which a tactile stimulus and an irrelevant visual distracter were presented simultaneously but independently to one of four positions. The participant judged the location (index or thumb) of the tactile stimulus with hands crossed or uncrossed. Speed was emphasized. In contrast to sighted controls, HS did not show a decrement of TOJ performance with hands crossed. Moreover, while the congruency gain was equivalent to sighted controls with uncrossed hands, this effect was significantly reduced with hands crossed. Thus, an external remapping of tactile stimuli still develops after a long phase of visual deprivation. However, remapping seems to be less efficient and to only take place in the context of visual stimuli.