Unimodal Conditions Research Articles

Investigating the Kappa Effect Elicited Through Concurrent Visual and Tactile Stimulation.

The experience of time and space in subjective perception is closely connected. The Kappa effect refers to the phenomenon where the perceived duration of the time interval between stimuli is influenced by the spatial distance between them. In this study, we aimed to explore the Kappa effect from a psychophysical perspective. We investigated participants' perception of temporal duration in the sub-second range by delivering visual and tactile inputs through wearable devices placed on both the palm and the forearm. We compared the impact of unimodal sensory stimulation, involving either visual or tactile stimuli, with different bimodal stimulation conditions. Our results revealed that the illusory effect on inter-stimulus duration perception can be observed in both unimodal conditions, although the distortions were significantly more pronounced in vision. In the multimodal stimulation condition, where visual stimuli were presented at non-equidistant spatial locations, the integration of tactile input did not reduce the Kappa effect, regardless of the spatial location of the tactile stimuli. However, when the visual stimuli were equidistant in space, regardless of the spatial location of the tactile stimuli, the Kappa effect disappeared. These results can shed light on the effect played by multimodality on the perception of space and time.

Multisensory integration and motor resonance in the primary motor cortex

Humans are endowed with a motor system that resonates to speech sounds, but whether concurrent visual information from lip movements can improve speech perception at a motor level through multisensory integration mechanisms remains unknown. Therefore, the aim of the study was to explore behavioral and neurophysiological correlates of multisensory influences on motor resonance in speech perception. Motor-evoked potentials (MEPs), by single pulse transcranial magnetic stimulation (TMS) applied over the left lip muscle (orbicularis oris) representation in the primary motor cortex, were recorded in healthy participants during the presentation of syllables in unimodal (visual or auditory) or multisensory (audio-visual) congruent or incongruent conditions. At the behavioral level, subjects showed better syllable identification in the congruent audio-visual condition as compared to the unimodal conditions, hence showing a multisensory enhancement effect. Accordingly, at the neurophysiological level, increased MEPs amplitudes were found in the congruent audio-visual condition, as compared to the unimodal ones. Incongruent audio-visual syllables resulting in illusory percepts did not increase corticospinal excitability, which in fact was comparable to that induced by the real perception of the same syllable. In conclusion, seeing and hearing congruent bilabial syllables increases the excitability of the lip representation in the primary motor cortex, hence documenting that multisensory integration can facilitate speech processing by influencing motor resonance. These findings highlight the modulation role of multisensory processing showing that it can boost speech perception and that multisensory interactions occur not only within higher-order regions, but also within primary motor areas, as shown by corticospinal excitability changes.

EEG Cortical Sources of Theta, Beta and Gamma Frequencies Orchestrate Audio–Visual Interactions

Background The fascinating ability of brain to integrate information from multiple sensory inputs has intrigued many researchers. Audio–visual (AV) interaction is a form of multisensory integration which we encounter to form meaningful representations of the environment around us. There is limited literature related to the underlying neural mechanisms. Purpose Quantitative EEG (QEEG), a tool with high temporal resolution can be used to understand cortical sources of AV interactions. Methods EEG data was recorded using 128 channels from 30 healthy subjects using audio, visual and AV stimuli in ‘object detection task’. Electrical source imaging was performed using s-LORETA across seven frequency bands (lower alpha 1, lower alpha 2, upper alpha, beta, delta, gamma, theta) during AV versus unimodal conditions across 66 gyri. Results The cortical sources were activated in the theta, beta, and gamma bands in cross modal versus unimodal conditions, which we propose, reflect neural communication for AV interaction network. The cortical sources constituted areas involved with visual processing, auditory processing, established multisensory (frontotemporal cortex, parietal cortex, middle temporal gyrus, superior frontal gyrus, inferior frontal gyrus, precentral gyrus) and potential multisensory areas (paracentral, postcentral and subcallosal). Conclusion Together, these results offer an integrative view of cortical areas in frequency oscillations during AV interactions.

Generalisation to novel exemplars of learned shape categories based on visual and auditory spatial cues does not benefit from multisensory information.

Although the integration of information across multiple senses can enhance object representations in memory, how multisensory information affects the formation of categories is uncertain. In particular, it is unclear to what extent categories formed from multisensory information benefit object recognition over unisensory inputs. Two experiments investigated the categorisation of novel auditory and visual objects, with categories defined by spatial similarity, and tested generalisation to novel exemplars. Participants learned to categorise exemplars based on visual-only (geometric shape), auditory-only (spatially defined soundscape) or audio-visual spatial cues. Categorisation to learned as well as novel exemplars was then tested under the same sensory learning conditions. For all learning modalities, categorisation generalised to novel exemplars. However, there was no evidence of enhanced categorisation performance for learned multisensory exemplars. At best, bimodal performance approximated that of the most accurate unimodal condition, although this was observed only for a subset of exemplars within a category. These findings provide insight into the perceptual processes involved in the formation of categories and have relevance for understanding the sensory nature of object representations underpinning these categories.

The dissociating effects of fear and disgust on multisensory integration in autism: evidence from evoked potentials.

Deficits in Multisensory Integration (MSI) in ASD have been reported repeatedly and have been suggested to be caused by altered long-range connectivity. Here we investigate behavioral and ERP correlates of MSI in ASD using ecologically valid videos of emotional expressions. In the present study, we set out to investigate the electrophysiological correlates of audiovisual MSI in young autistic and neurotypical adolescents. We employed dynamic stimuli of high ecological validity (500 ms clips produced by actors) that depicted fear or disgust in unimodal (visual and auditory), and bimodal (audiovisual) conditions. We report robust MSI effects at both the behavioral and electrophysiological levels and pronounced differences between autistic and neurotypical participants. Specifically, neurotypical controls showed robust behavioral MSI for both emotions as seen through a significant speed-up of bimodal response time (RT), confirmed by Miller's Race Model Inequality (RMI), with greater MSI effects for fear than disgust. Adolescents with ASD, by contrast, showed behavioral MSI only for fear. At the electrophysiological level, the bimodal condition as compared to the unimodal conditions reduced the amplitudes of the visual P100 and auditory P200 and increased the amplitude of the visual N170 regardless of group. Furthermore, a cluster-based analysis across all electrodes revealed that adolescents with ASD showed an overall delayed and spatially constrained MSI effect compared to controls. Given that the variables we measured reflect attention, our findings suggest that MSI can be modulated by the differential effects on attention that fear and disgust produce. We also argue that the MSI deficits seen in autistic individuals can be compensated for at later processing stages by (a) the attention-orienting effects of fear, at the behavioral level, and (b) at the electrophysiological level via increased attentional effort.

The audiovisual competition effect induced by temporal asynchronous encoding weakened the visual dominance in working memory retrieval

ABSTRACT Converging evidence suggests a facilitation effect of multisensory interactions on memory performance, reflected in higher accuracy or faster response time under a bimodal encoding condition than a unimodal condition. However, relatively little attention has been given to the effect of multisensory competition on memory. The present study adopted an adaptive staircase test to measure the point of subjective simultaneity (PSS), combined with a delayed matched-to-sample (DMS) task to probe the effect of audiovisual competition during the encoding stage on subsequent unisensory retrieval. The results showed that there was a robust visual dominance effect and multisensory interference effect in WM retrieval, regardless of the subjective synchronous or subjective asynchronous audiovisual presentation. However, a weakened visual dominance effect was observed when the auditory stimulus was presented before the visual stimulus in the encoding period, particularly in the semantically incongruent case. These findings revealed that the prior-entry of sensory information in the early perceptual stage could affect the processing in the late cognitive stage to some extent, and supported the evidence that there is a persistent advantage for visuospatial sketchpad in multisensory WM.

Visuotactile integration in individuals with fibromyalgia.

Our brain constantly integrates afferent information, such as visual and tactile information, to perceive the world around us. According to the maximum-likelihood estimation (MLE) model, imprecise information will be weighted less than precise, making the multisensory percept as precise as possible. Individuals with fibromyalgia (FM), a chronic pain syndrome, show alterations in the integration of tactile information. This could lead to a decrease in their weight in a multisensory percept or a general disruption of multisensory integration, making it less beneficial. To assess multisensory integration, 15 participants with FM and 18 pain-free controls performed a temporal-order judgment task in which they received pairs of sequential visual, tactile (unisensory conditions), or visuotactile (multisensory condition) stimulations on the index and the thumb of the non-dominant hand and had to determine which finger was stimulated first. The task enabled us to measure the precision and accuracy of the percept in each condition. Results indicate an increase in precision in the visuotactile condition compared to the unimodal conditions in controls only, although we found no intergroup differences. The observed visuotactile precision was correlated to the precision predicted by the MLE model in both groups, suggesting an optimal integration. Finally, the weights of the sensory information were not different between the groups; however, in the group with FM, higher pain intensity was associated with smaller tactile weight. This study shows no alterations of the visuotactile integration in individuals with FM, though pain may influence tactile weight in these participants.

Mapping of facial and vocal processing in common marmosets with ultra-high field fMRI

Primate communication relies on multimodal cues, such as vision and audition, to facilitate the exchange of intentions, enable social interactions, avoid predators, and foster group cohesion during daily activities. Understanding the integration of facial and vocal signals is pivotal to comprehend social interaction. In this study, we acquire whole-brain ultra-high field (9.4 T) fMRI data from awake marmosets (Callithrix jacchus) to explore brain responses to unimodal and combined facial and vocal stimuli. Our findings reveal that the multisensory condition not only intensifies activations in the occipito-temporal face patches and auditory voice patches but also engages a more extensive network that includes additional parietal, prefrontal and cingulate areas, compared to the summed responses of the unimodal conditions. By uncovering the neural network underlying multisensory audiovisual integration in marmosets, this study highlights the efficiency and adaptability of the marmoset brain in processing facial and vocal social signals, providing significant insights into primate social communication.

Multisensory integration deficits in Schizophrenia and Autism evidenced in behaviour but not event related potentials

The process of integrating information from different sensory channels, known as multisensory integration (MSI) was assessed in two disorders, Autism Spectrum Disorder (ASD) and Schizophrenia (SCZ). 32 healthy controls (HC), 35 SCZ patients, and 23 ASD patients performed an audiovisual (AV) synchronous target detection task while reaction time (RT) and scalp recorded electrophysiological (EEG) activity were measured. MSI in the AV condition resulted in faster and less variable RTs compared to the unimodal conditions. Using our novel bootstrap method, MSI gain was observed in 78 % of HC, 26 % of ASD, and 48 % of SCZ patients. At the neural level, MSI in the AV condition resulted in larger amplitude of sensory evoked responses and cognitive P3 response compared to the corresponding unimodal conditions. These neural effects of MSI were not related to the behavioural MSI gain identified at the individual level and could not explain the deficits in behavioural MSI of patient groups. In conclusion, a robust MSI gain deficit in RT was observed in both patient groups that was not reflected in early perceptual and cognitive electro-cortical responses, suggesting that behavioural MSI deficits in ASD and SCZ may arise at late processing stages such as response selection.

Distinct neural signatures of multimodal resizing illusions

Illusory body resizing typically uses multisensory integration to change the perceived size of a body part. Previous studies associate these multisensory body illusions with frontal theta oscillations and parietal gamma oscillations for dis-integration and integration of multisensory signals, respectively. However, recent studies also support illusory changes of embodiment from unimodal visual stimuli. This preregistered study (N = 48) investigated differences between multisensory visuo-tactile and unimodal visual resizing illusions using EEG, to gain a more comprehensive understanding of the neural underpinnings of resizing illusions in a healthy population. We hypothesised (1) stronger illusion in multisensory compared to unimodal, and unimodal compared to incongruent (dis-integration) conditions, (2) greater parietal gamma during multisensory compared to unimodal, and (3) greater frontal theta during incongruent compared to baseline conditions. Subjective Illusory results partially support Hypothesis 1, showing a stronger illusion in multisensory compared to unimodal conditions, but finding no significant difference comparing unimodal to incongruent conditions. Results partially supported EEG hypotheses, finding increased parietal gamma activity comparing multisensory to unimodal visual conditions, happening at a later stage of the illusion when compared to previous rubber hand illusion EEG findings, whilst also finding increased parietal theta activity when comparing incongruent to non-illusion conditions. While results demonstrated that only 27% of participants experienced the stretching illusion with unimodal visual stimuli compared to 73% of participants experiencing the stretching illusion in the multisensory condition, further analysis suggested that those who experience visual-only illusions exhibit a different neural signature to those who do not, with activity focussed around frontal and parietal regions early on in the illusory manipulation, compared to activity focussed more over parietal regions and at a later point in the illusory manipulation for the full sample of participants. Our results replicate previous subjective experience findings and support the importance of multisensory integration for illusory changes in perceived body size, whilst adding to our understanding of the temporal onset of multisensory integration within resizing illusions, differing from that of rubber hand illusions.

Differential audiovisual information processing in emotion recognition: An eye-tracking study.

Recent research has suggested that dynamic emotion recognition involves strong audiovisual association; that is, facial or vocal information alone automatically induces perceptual processes in the other modality. We hypothesized that different emotions may differ in the automaticity of audiovisual association, resulting in differential audiovisual information processing. Participants judged the emotion of a talking-head video under audiovisual, video-only (with no sound), and audio-only (with a static neutral face) conditions. Among the six basic emotions, disgust had the largest audiovisual advantage over the unimodal conditions in recognition accuracy. In addition, in the recognition of all the emotions except for disgust, participants' eye-movement patterns did not change significantly across the three conditions, suggesting mandatory audiovisual information processing. In contrast, in disgust recognition, participants' eye movements in the audiovisual condition were less eyes-focused than the video-only condition and more eyes-focused than the audio-only condition, suggesting that audio information in the audiovisual condition interfered with eye-movement planning for important features (eyes) for disgust. In addition, those whose eye-movement pattern was affected less by concurrent disgusted voice information benefited more in recognition accuracy. Disgust recognition is learned later in life and thus may involve a reduced amount of audiovisual associative learning. Consequently, audiovisual association in disgust recognition is less automatic and demands more attentional resources than other emotions. Thus, audiovisual information processing in emotion recognition depends on the automaticity of audiovisual association of the emotion resulting from associative learning. This finding has important implications for real-life emotion recognition and multimodal learning. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Assessing the Value of Multimodal Interfaces: A Study on Human-Machine Interaction in Weld Inspection Workstations.

Multimodal user interfaces promise natural and intuitive human-machine interactions. However, is the extra effort for the development of a complex multisensor system justified, or can users also be satisfied with only one input modality? This study investigates interactions in an industrial weld inspection workstation. Three unimodal interfaces, including spatial interaction with buttons augmented on a workpiece or a worktable, and speech commands, were tested individually and in a multimodal combination. Within the unimodal conditions, users preferred the augmented worktable, but overall, the interindividual usage of all input technologies in the multimodal condition was ranked best. Our findings indicate that the implementation and the use of multiple input modalities is valuable and that it is difficult to predict the usability of individual input modalities for complex systems.

Contralateral hearing aid use in adult cochlear implant recipients: retrospective analysis of auditory outcomes

Objective To investigate retrospectively the frequency of usage of bimodal stimulation among cochlear implant (CI) users, as well its clinical benefit relative to unilateral use. Design All subjects had been monitored with the clinical Minimal Outcome Measurements test battery. Study samples 103 adults with bilateral postlingual profound sensorineural hearing loss and unilateral CI use were extracted from the local database. These were divided into two groups: those who only used a CI and those who used bimodal stimulation. Results The preoperative contralateral residual hearing in the bimodal group was significantly better than that of the CI-only group. In both groups, speech perception in quiet and in noise improved after CI, with no significant difference between postoperative unimodal conditions. For the bimodal group, an additional significant improvement was found for the bimodal condition compared to the unimodal. Conclusion Given the observed auditory benefit of bimodal stimulation in comparison to unimodal stimulation and given the finding that degree of residual hearing is not correlated with bimodal benefits, it is recommended to encourage CI recipients to continue contralateral HA use after CI. As a result of expanding CI criteria worldwide, the population of bimodal users is expected to grow in the near future.

The Development of Cortical Responses to the Integration of Audiovisual Speech in Infancy

In adults, the integration of audiovisual speech elicits specific higher (super-additive) or lower (sub-additive) cortical responses when compared to the responses to unisensory stimuli. Although there is evidence that the fronto-temporal network is active during perception of audiovisual speech in infancy, the development of fronto-temporal responses to audiovisual integration remains unknown. In the current study, 5-month-olds and 10-month-olds watched bimodal (audiovisual) and alternating unimodal (auditory + visual) syllables. In this context we use alternating unimodal to denote alternating auditory and visual syllables that are perceived as separate syllables by adults. Using fNIRS we measured responses over large cortical areas including the inferior frontal and superior temporal regions. We identified channels showing different responses to bimodal than alternating unimodal condition and used multivariate pattern analysis (MVPA) to decode patterns of cortical responses to bimodal (audiovisual) and alternating unimodal (auditory + visual) speech. Results showed that in both age groups integration elicits cortical responses consistent with both super- and sub-additive responses in the fronto-temporal cortex. The univariate analyses revealed that between 5 and 10 months spatial distribution of these responses becomes increasingly focal. MVPA correctly classified responses at 5 months, with key input from channels located in the inferior frontal and superior temporal channels of the right hemisphere. However, MVPA classification was not successful at 10 months, suggesting a potential cortical re-organisation of audiovisual speech perception at this age. These results show the complex and non-gradual development of the cortical responses to integration of congruent audiovisual speech in infancy.

Distributional learning of musical pitch despite tone deafness in individuals with congenital amusia

Congenital amusia is an innate and lifelong deficit of music processing. This study investigated whether adult listeners with amusia were still able to learn pitch-related musical chords based on stimulus frequency of statistical distribution, i.e., via distributional learning. Following a pretest-training-posttest design, 18 amusics and 19 typical, musically intact listeners were assigned to bimodal and unimodal conditions that differed in distribution of the stimuli. Participants' task was to discriminate chord minimal pairs, which were transposed to a novel microtonal scale. Accuracy rates for each test session were collected and compared between the two groups using generalized mixed-effects models. Results showed that amusics were less accurate than typical listeners at all comparisons, thus corroborating previous findings. Importantly, amusics-like typical listeners-demonstrated perceptual gains from pretest to posttest in the bimodal condition (but not the unimodal condition). The findings reveal that amusics' distributional learning of music remains largely preserved despite their deficient music processing. Implications of the results for statistical learning and intervention programs to mitigate amusia are discussed.

Modality predictability modulation confirms the sensorial readiness function of the pre-stimulus activity in sensory brain areas

The auditory Positivity (aP) and the visual Negativity (vN) are recently discovered modality-specific event-related potential (ERP) components associated with sensory readiness, which seems promising to study anticipatory perception and attention. However, a crucial aspect of these waves remains to be determined since it is still unclear if these components are indeed related to sensory readiness or represent the result of stimulus predictably. Indeed, earlier studies found these components in tasks where stimuli were repeatedly presented uniquely in the same sensory modality. To disentangle this issue, we used an experimental design consisting of three passive tasks: a unimodal auditory condition, a unimodal visual condition, and an intermodal condition in which the visual and auditory stimuli were unpredictably alternated. Then, we compared the amplitudes of the aP and vN in the three conditions and performed correlation analyses between pre-stimulus and post-stimulus components. Crucially, results showed that in the intermodal condition the components still occur, but their amplitudes are decreased compared to unimodal condition, providing evidence that they are only partially dependent on the task and that expectancy might modulate them. This result is in line with the “modality-shift effect” costs phenomenon which can occur also for passive tasks even before stimulus presentation. In addition, the amplitude of the post-stimulus components correlated with pre-stimulus ERP. Collectively, the present study confirms that the aP and the vN reflect sensory readiness processes that “boost” post-stimulus auditory N1 and visual P1 components.

Multisensory interactions on auditory and somatosensory information in expert pianists

Fine-tuned sensory functions typically characterize skilled individuals. Although numerous studies demonstrated enhanced unimodal sensory functions at both neural and behavioral levels in skilled individuals, little is known about their multisensory interaction function, especially multisensory integration and selective attention that involve volitional control of information derived from multiple sensory organs. In the current study, expert pianists and musically untrained individuals performed five sets of intensity discrimination tasks at the auditory and somatosensory modalities with different conditions: (1) auditory stimulus, (2) somatosensory stimulus, (3) congruent auditory and somatosensory stimuli (i.e., multisensory integration), (4) auditory and task-irrelevant somatosensory stimuli, and (5) somatosensory and task-irrelevant auditory stimuli. In the fourth and fifth conditions, participants were instructed to ignore a task-irrelevant stimulus and to pay attention to a task-relevant stimulus (i.e., selective attention), respectively. While the discrimination perception was superior in the condition (3) compared to the better one of the individual unimodal conditions only in the pianists, the task-irrelevant somatosensory stimulus worsened the auditory discrimination more in the pianists than the nonmusicians. These findings indicate unique multisensory interactions in expert pianists, which enables pianists to efficiently integrate the auditory and somatosensory information, but exacerbates top-down selective inhibition of somatosensory information during auditory processing.

Crossmodal Correspondence between Music and Ambient Color Is Mediated by Emotion.

The quality of a concert hall primarily depends on its acoustics. But does visual input also have an impact on musical enjoyment? Does the color of ambient lighting modulate the perceived music quality? And are certain colors perceived to fit better than others with a given music piece? To address these questions, we performed three within-subjects experiments. We carried out two pretests to select four music pieces differing in tonality and genre, and 14 lighting conditions of varying hue, brightness, and saturation. In the main experiment, we applied a fully crossed repeated-measures design. Under each of the four lighting conditions, participants rated the musical variables 'Harmonic', 'Powerful', 'Gloomy', 'Lively' and overall liking of the music pieces, as well as the perceived fit of music and lighting. Subsequently, participants evaluated music and lighting separately by rating the same variables as before, as well as their emotional impact (valence, arousal, dominance). We found that music and lighting being similarly rated in terms of valence and arousal in the unimodal conditions were judged to match better when presented together. Accordingly, tonal (atonal) music was rated to fit better with weakly saturated (highly saturated) colors. Moreover, some characteristics of the lighting were carried over to music. That is, just as red lighting was rated as more powerful than green and blue lighting, music was evaluated to be more powerful under red compared to green and blue lighting. We conclude that listening to music is a multisensory process enriched by impressions from the visual domain.

Neurofunctional basis underlying audiovisual integration of print and speech sound in Chinese children.

Effortless print-sound integration is essential to reading development, and the superior temporal cortex (STC) is the most critical brain region. However, to date, the conclusion is almost restricted to alphabetic orthographies. To examine the neural basis in non-alphabetic languages and its relationship with reading abilities, we conducted a functional magnetic resonance imaging study in typically developing Chinese children. Two neuroimaging-based indicators of audiovisual processing-additive enhancement (higher activation in the congruent than the average activation of unimodal conditions) and neural integration (different activations between the congruent versus incongruent conditions)-were used to investigate character-sounds (opaque) and pinyin-sounds (transparent) processing. We found additive enhancement in bilateral STCs processing both character and pinyin stimulations. Moreover, the neural integrations in the left STC for the two scripts were strongly correlated. In terms of differentiation, first, areas beyond the STCs also showed additive enhancement in processing pinyin-sounds. Second, while the bilateral STCs, left inferior/middle frontal and parietal regions manifested a striking neural integration (incongruent > congruent) for character-sounds, no significant clusters were revealed for pinyin-sounds. Finally, the neural integration in the left middle frontal gyrus for characters was specifically associated with silent reading comprehension proficiency, indicating automatic semantic processing during implicit character-sound integration. In contrast, the neural integration in the left STC for pinyin was specifically associated with oral reading fluency that relies on grapho-phonological mapping. To summarize, this study revealed both script-universal and script-specific neurofunctional substrates of print-sound integration as well as their processing- and region-dependent associations with reading abilities in typical Chinese children.

Uniqueness of principal points with respect to p-order distance for a class of univariate continuous distribution

In this paper, the uniqueness of principal points with respect to p-order distance is studied. When the density function of univariate bounded continuous distribution fluctuates in a specified range, the principal points are unique. It is completely different from the strongly unimodal condition.