Unimodal Conditions Research Articles

The effect of distributional training on the production of non-native vowel contrasts

The current study tests production accuracy of a non-native vowel contrast following a modified perceptual learning paradigm. Previous research has found that adults can learn non-native sound categories with sensitivity to distributional properties of their input. In the current study, 34 native-English adults heard stimuli from /o/-/oe/ continuum. Half of the participants heard stimuli drawn from a bimodal distribution and half from a unimodal distribution. To support learning, we incorporated active learning with feedback, lexical support in the form of two images, and overnight consolidation. Production of this contrast was assessed through a repetition task at baseline and at the end of the experiment. Production accuracy is measured as the Euclidean distance between /o/ and /oe/ at baseline and post-training. Preliminary analyses suggest that the distance between these two vowels increased for both groups, indicating that listeners in both conditions were able to transfer perceptual learning to production. These results suggest a way to mitigate the disadvantages previously found for participants in the unimodal condition, by incorporating active engagement with the target stimuli using lexical support, accuracy feedback, and overnight consolidation.

Multimodal perception of interrupted speech and text by younger and older adults

The current study aimed to identify how partial auditory (speech) and visual (text) information are resolved for sentence recognition by younger and older adults. Interrupted speech was created by periodically replacing speech with silent intervals at various interruption rates. Likewise, interrupted text was created by periodically replacing printed text with white space. Speech was further processed by low-pass filtering the stimulus spectrum at 2000 Hz. Speech and text stimuli were interrupted at 0.5, 2, 8, and 32 Hz and presented in unimodal and multimodal conditions. Overall, older listeners demonstrated similar performance to younger listeners on the unimodal conditions. However, there was a marked decrease in performance by older listeners during multimodal testing, suggesting increased difficulty in integrating information across modalities. Performance for both groups improved with higher interruption rates for both text and speech in unimodal and multimodal conditions. Listeners were also able to gain benefit from most multimodal presentations, even when the rate of interruption was mismatched between modalities. Supplementing speech with incomplete visual cues can improve sentence intelligibility and compensate for degraded speech in adverse listening conditions. However, the magnitude of multimodal benefit appears mitigated by age-related changes in integrating information across modalities. [Work supported, in part, by NIH/NIDCD.]

Lack of Cross-Modal Effects in Dual-Modality Implicit Statistical Learning

A current controversy in the area of implicit statistical learning (ISL) is whether this process consists of a single, central mechanism or multiple modality-specific ones. To provide insight into this question, the current study involved three ISL experiments to explore whether multimodal input sources are processed separately in each modality or are integrated together across modalities. In Experiment 1, visual and auditory ISL were measured under unimodal conditions, with the results providing a baseline level of learning for subsequent experiments. Visual and auditory sequences were presented separately, and the underlying grammar used for both modalities was the same. In Experiment 2, visual and auditory sequences were presented simultaneously with each modality using the same artificial grammar to investigate whether redundant multisensory information would result in a facilitative effect (i.e., increased learning) compared to the baseline. In Experiment 3, visual and auditory sequences were again presented simultaneously but this time with each modality employing different artificial grammars to investigate whether an interference effect (i.e., decreased learning) would be observed compared to the baseline. Results showed that there was neither a facilitative learning effect in Experiment 2 nor an interference effect in Experiment 3. These findings suggest that participants were able to track simultaneously and independently two sets of sequential regularities under dual-modality conditions. These findings are consistent with the theories that posit the existence of multiple, modality-specific ISL mechanisms rather than a single central one.

The Time Course of Audio-Visual Phoneme Identification: a High Temporal Resolution Study.

Speech unfolds in time and, as a consequence, its perception requires temporal integration. Yet, studies addressing audio-visual speech processing have often overlooked this temporal aspect. Here, we address the temporal course of audio-visual speech processing in a phoneme identification task using a Gating paradigm. We created disyllabic Spanish word-like utterances (e.g., /pafa/, /paθa/, …) from high-speed camera recordings. The stimuli differed only in the middle consonant (/f/, /θ/, /s/, /r/, /g/), which varied in visual and auditory saliency. As in classical Gating tasks, the utterances were presented in fragments of increasing length (gates), here in 10ms steps, for identification and confidence ratings. We measured correct identification as a function of time (at each gate) for each critical consonant in audio, visual and audio-visual conditions, and computed the Identification Point and Recognition Point scores. The results revealed that audio-visual identification is a time-varying process that depends on the relative strength of each modality (i.e., saliency). In some cases, audio-visual identification followed the pattern of one dominant modality (either A or V), when that modality was very salient. In other cases, both modalities contributed to identification, hence resulting in audio-visual advantage or interference with respect to unimodal conditions. Both unimodal dominance and audio-visual interaction patterns may arise within the course of identification of the same utterance, at different times. The outcome of this study suggests that audio-visual speech integration models should take into account the time-varying nature of visual and auditory saliency.

New sufficient conditions for strong unimodality of multivariate discrete distributions

New sufficient conditions that ensure the strong unimodality of multivariate discrete distributions are obtained by the use of a special simplicial subdivision of multidimensional space. Strong unimodality of multivariate Pólya–Eggenberger distribution is shown.

Electrophysiological evidence for Audio-visuo-lingual speech integration

Recent neurophysiological studies demonstrate that audio-visual speech integration partly operates through temporal expectations and speech-specific predictions. From these results, one common view is that the binding of auditory and visual, lipread, speech cues relies on their joint probability and prior associative audio-visual experience. The present EEG study examined whether visual tongue movements integrate with relevant speech sounds, despite little associative audio-visual experience between the two modalities. A second objective was to determine possible similarities and differences of audio-visual speech integration between unusual audio-visuo-lingual and classical audio-visuo-labial modalities. To this aim, participants were presented with auditory, visual, and audio-visual isolated syllables, with the visual presentation related to either a sagittal view of the tongue movements or a facial view of the lip movements of a speaker, with lingual and facial movements previously recorded by an ultrasound imaging system and a video camera. In line with previous EEG studies, our results revealed an amplitude decrease and a latency facilitation of P2 auditory evoked potentials in both audio-visual-lingual and audio-visuo-labial conditions compared to the sum of unimodal conditions. These results argue against the view that auditory and visual speech cues solely integrate based on prior associative audio-visual perceptual experience. Rather, they suggest that dynamic and phonetic informational cues are sharable across sensory modalities, possibly through a cross-modal transfer of implicit articulatory motor knowledge.

Effect of Linguistic and Musical Experience on Distributional Learning of Nonnative Lexical Tones.

Evidence suggests that extensive experience with lexical tones or musical training provides an advantage in perceiving nonnative lexical tones. This investigation concerns whether such an advantage is evident in learning nonnative lexical tones based on the distributional structure of the input. Using an established protocol, distributional learning of lexical tones was investigated with tone language (Mandarin) listeners with no musical training (Experiment 1) and nontone language (Australian English) listeners with musical training (Experiment 2). Within each experiment, participants were trained on a bimodal (2-peak) or a unimodal (single peak) distribution along a continuum spanning a Thai lexical tone minimal pair. Discrimination performance on the target minimal pair was assessed before and after training. Mandarin nonmusicians exhibited clear distributional learning (listeners in the bimodal, but not those in the unimodal condition, improved significantly as a function of training), whereas Australian English musicians did not (listeners in both the bimodal and unimodal conditions improved as a function of training). Our findings suggest that veridical perception of lexical tones is not sufficient for distributional learning of nonnative lexical tones to occur. Rather, distributional learning appears to be modulated by domain-specific pitch experience and is constrained possibly by top-down interference.

Multimodal Sustained Attention Superiority in Concentrative Meditators Compared to Nonmeditators

Attention regulation plays a central role in both Buddhist and secular forms of meditation. One of the most common meditative practices is a concentrative meditation technique aimed at developing sustained focused attention. Despite a growing body of research showing meditation practice enhances various attention abilities, evidence of sustained attention benefits from meditation has been inconsistent. Also, most studies that tested meditators’ sustained attention abilities used visual tasks. The extent to which the putative superior sustained attention in meditators might generalize to other stimulus modalities (e.g., auditory), and thus, whether meditation is associated with general attentional enhancement, is still unclear. Here, we compared regular meditators’ sustained attention performance to nonmeditator controls using the response switching task (RST) in unimodal visual and auditory conditions (Exp 1) as well as bimodal visual-auditory conditions (Exp 2). The RST involves continuous responding to frequent stimuli and switching button-responses with an infrequent target stimulus. Errors in responding to the target signify difficulties in sustaining attention. Our main results showed that meditators made significantly fewer errors, indicating fewer attentional lapses in sustained attention, than nonmeditators across all unimodal and bimodal RST conditions, a finding that has not been reported before in previous studies. These findings provide further evidence of a positive association between meditators and enhanced sustained attention and new evidence that suggest that meditation is associated with general, modality nonspecific, enhancement of attentional control.

Impaired facial and vocal emotion decoding in schizophrenia is underpinned by basic perceptivo-motor deficits

ABSTRACTIntroduction: Emotional decoding impairments have been largely demonstrated in schizophrenia for facial and prosodic stimuli, when presented separately. Nevertheless, the exploration of crossmodal integration has been far less considered, despite its omnipresence in daily social interactions. Moreover, the role played by basic visuo-motor impairments in unimodal and crossmodal decoding remains unexplored.Methods: Thirty-two patients were compared with 32 matched controls in an emotional decoding task including unimodal (visual and auditory) and crossmodal (congruent and incongruent) conditions. A control perceptive task was also conducted to take potential low-level perceptual deficits into account.Results: Schizoprenic patients presented lower performance and higher reaction times for both unimodal tasks (visual and auditory) and crossmodal conditions. Moreover, reaction times for the visuo-perceptive task were also significantly longer for patients compared to controls.Conclusions: The consistency of the results across unimodal and crossmodal tasks suggests a globalised emotional impairment in schizophrenia, independent of the sensorial modality and crossmodal nature of the stimuli. Centrally, given the results in the visuo-perceptive task, the impairments observed for emotional recognition appears at least partly explained by primary cognitive deficits, namely reduced processing speed.

Multisensory integration in orienting behavior: Pupil size, microsaccades, and saccades

Signals from different sensory modalities are integrated in the brain to optimize behavior. Although multisensory integration has been demonstrated in saccadic eye movements, its influence on other orienting responses, including pupil size and microsaccades, is still poorly understood. We examined human gaze orienting responses following presentation of visual, auditory, or combined audiovisual stimuli. Transient pupil dilation and microsaccade inhibition were evoked shortly after the appearance of a salient stimulus. Audiovisual stimuli evoked larger pupil dilation, greater microsaccade inhibition, and faster saccade reaction times compared to unimodal conditions. Trials with faster saccadic reaction times were accompanied with greater pupil dilation responses. Similar modulation of pre-stimulus pupil-size-change rate was observed between stimulus-evoked saccadic and pupillary responses. Thus, multisensory integration impacts multiple components of orienting, with coordination between saccade and pupil responses, implicating the superior colliculus in coordinating these responses because of its central role in both orienting behavior and multisensory integration.

Age differences in emotion recognition: A question of modality?

Previous research has suggested age deficits in unimodal emotion recognition tasks. In 2 studies with independent samples, we tested the idea that older adults' performance will be enhanced in multimodal emotion recognition tasks. In each study, participants were presented with newly developed film clips, each portraying a young or an old woman while she relived an emotional memory. As a first step, participants received the film clips in each of 3 unimodal conditions (facial, lexical, prosodic). As a second step, participants were presented with tasks that contained different combinations of these modalities. Consistent with prior research, younger adults outperformed older adults in unimodal facial and prosodic tasks (Study 1) or in all 3 unimodal tasks (Study 2). As predicted, older adults profited more from multimodal stimuli than young adults so that there were no age deficits in the multimodal tasks (Studies 1 and 2). The findings support the idea that the ecological validity of laboratory tasks influence age differences in emotion recognition. (PsycINFO Database Record

Electrophysiological evidence for a self-processing advantage during audiovisual speech integration.

Previous electrophysiological studies have provided strong evidence for early multisensory integrative mechanisms during audiovisual speech perception. From these studies, one unanswered issue is whether hearing our own voice and seeing our own articulatory gestures facilitate speech perception, possibly through a better processing and integration of sensory inputs with our own sensory-motor knowledge. The present EEG study examined the impact of self-knowledge during the perception of auditory (A), visual (V) and audiovisual (AV) speech stimuli that were previously recorded from the participant or from a speaker he/she had never met. Audiovisual interactions were estimated by comparing N1 and P2 auditory evoked potentials during the bimodal condition (AV) with the sum of those observed in the unimodal conditions (A+V). In line with previous EEG studies, our results revealed an amplitude decrease of P2 auditory evoked potentials in AV compared to A+V conditions. Crucially, a temporal facilitation of N1 responses was observed during the visual perception of self speech movements compared to those of another speaker. This facilitation was negatively correlated with the saliency of visual stimuli. These results provide evidence for a temporal facilitation of the integration of auditory and visual speech signals when the visual situation involves our own speech gestures.

Does facilitation by phonological cuing in picture naming depend on the modality of the cue?

ABSTRACTBackground: Phonological cuing in spoken picture naming is widely used to improve word production accuracy in aphasic participants and to investigate word planning processes in psycholinguistic experiments with healthy participants. However, different paradigms and different modalities have been used across studies and populations, with some contradictory results.Aims: Here, we investigate whether and how the modality in which a phonological cue is delivered impacts on word production in picture naming tasks using the exact same paradigm with healthy and with aphasic participants.Methods & Procedure: In experiment 1, behavioural and electroencephalographic event-related potential (ERP) responses to auditory, visual and audiovisual cues in a picture naming task were investigated in healthy participants. In experiment 2, the impact of the same cue modalities was studied on errors and latencies in 15 aphasic participants.Outcomes & Results: The modality in which the cue is delivered influenced phonological facilitation effects in both populations, the audio modality being the most facilitative on production latencies in healthy participants and in participants with mild aphasia. No further gains were obtained by a double modality audiovisual cue on production latencies, whereas the double cues interfered with production accuracy in aphasic participants. ERP results not only were compatible with facilitation of phonological encoding in all experimental conditions but also showed modulation of a late encoding time period for both the visual and audiovisual cues, likely reflecting interference of the visual modality during monitoring or motor speech encoding. The latter observation is indeed coherent with the interference of the audiovisual modality that was observed on error rates in the more severely impaired aphasic participants.Conclusions: These results suggest that unimodal auditory cuing is the facilitatory condition, while visual cues in unimodal or bimodal conditions give rise to both facilitation and interference which manifest behaviourally depending on the balance of these opposing effects.

Cross-modal processing of voices and faces in developmental prosopagnosia and developmental phonagnosia

ABSTRACTConspecifics can be recognized from either the face or the voice alone. However, person identity information is rarely encountered in purely unimodal situations and there is increasing evidence that the face and voice interact in neurotypical identity processing. Conversely, developmental deficits have been observed that seem to be selective for face and voice recognition, developmental prosopagnosia and developmental phonagnosia, respectively. To date, studies on developmental prosopagnosia and phonagnosia have largely centred on within modality testing. Here, we review evidence from a small number of behavioural and neuroimaging studies which have examined the recognition of both faces and voices in these cohorts. A consensus from the findings is that, when tested in purely unimodal conditions, voice-identity processing appears normal in most cases of developmental prosopagnosia, as does face-identity processing in developmental phonagnosia. However, there is now first evidence that the multisensory nature of person identity impacts on identity recognition abilities in these cohorts. For example, unlike neurotypicals, auditory-only voice recognition is not enhanced in developmental prosopagnosia for voices which have been previously learned together with a face. This might also explain why the recognition of personally familiar voices is poorer in developmental prosopagnosics, compared to controls. In contrast, there is evidence that multisensory interactions might also lead to compensatory mechanisms in these disorders. For example, in developmental phonagnosia, voice recognition may be enhanced if voices have been learned with a corresponding face. Taken together, the reviewed findings challenge traditional models of person recognition which have assumed independence between face-identity and voice-identity processing and rather support an audio-visual model of human communication that assumes direction interactions between voice and face processing streams. In addition, the reviewed findings open up novel empirical research questions and have important implications for potential training regimes for developmental prosopagnosia and phonagnosia.

Weighted integration suggests that visual and tactile signals provide independent estimates about duration.

Humans might possess either a single (amodal) internal clock or multiple clocks for different sensory modalities. Sensitivity could be improved by the provision of multiple signals. Such improvements can be predicted quantitatively, assuming estimates are combined by summation, a process described as optimal when summation is weighted in accordance with the variance associated with each of the initially independent estimates. This possibility was assessed for visual and tactile information regarding temporal intervals. In Experiment 1, 12 musicians and 12 nonmusicians judged durations of 300 and 600 ms, compared to test values spanning these standards. Bimodal precision increased relative to unimodal conditions, but not to the extent predicted by optimally weighted summation. In Experiment 2, 6 musicians and 6 other participants each judged 6 standards, ranging from 100 ms to 600 ms, with conflicting cues providing a measure of the weight assigned to each sensory modality. A weighted integration model best fitted these data, with musicians more likely to select near-optimal weights than nonmusicians. Overall, data were consistent with the existence of separate visual and tactile clock components at either the counter/integrator or memory stages. Independent estimates are passed to a decisional process, but not always combined in a statistically optimal fashion. (PsycINFO Database Record

Activation in the angular gyrus and in the pSTS is modulated by face primes during voice recognition.

The aim of the present study was to better understand the interaction of face and voice processing when identifying people. In a S1-S2 crossmodal priming fMRI experiment, the target (S2) was a disyllabic voice stimulus, whereas the modality of the prime (S1) was manipulated blockwise and consisted of the silent video of a speaking face in the crossmodal condition or of a voice stimulus in the unimodal condition. Primes and targets were from the same speaker (person-congruent) or from two different speakers (person-incongruent). Participants had to classify the S2 as either an old or a young person. Response times were shorter after a congruent than after an incongruent face prime. The right posterior superior temporal sulcus (pSTS) and the right angular gyrus showed a significant person identity effect (person-incongruent > person-congruent) in the crossmodal condition but not in the unimodal condition. In the unimodal condition, a person identity effect was observed in the bilateral inferior frontal gyrus. Our data suggest that both the priming with a voice and with a face result in a preactivated voice representation of the respective person, which eventually facilitates (person-congruent trials) or hampers (person-incongruent trials) the processing of the identity of a subsequent voice. This process involves activation in the right pSTS and in the right angular gyrus for voices primed by faces, but not for voices primed by voices. Hum Brain Mapp 38:2553-2565, 2017. © 2017 Wiley Periodicals, Inc.

Multisensory Bayesian Inference Depends on Synapse Maturation during Training: Theoretical Analysis and Neural Modeling Implementation.

Recent theoretical and experimental studies suggest that in multisensory conditions, the brain performs a near-optimal Bayesian estimate of external events, giving more weight to the more reliable stimuli. However, the neural mechanisms responsible for this behavior, and its progressive maturation in a multisensory environment, are still insufficiently understood. The aim of this letter is to analyze this problem with a neural network model of audiovisual integration, based on probabilistic population coding-the idea that a population of neurons can encode probability functions to perform Bayesian inference. The model consists of two chains of unisensory neurons (auditory and visual) topologically organized. They receive the corresponding input through a plastic receptive field and reciprocally exchange plastic cross-modal synapses, which encode the spatial co-occurrence of visual-auditory inputs. A third chain of multisensory neurons performs a simple sum of auditory and visual excitations. The work includes a theoretical part and a computer simulation study. We show how a simple rule for synapse learning (consisting of Hebbian reinforcement and a decay term) can be used during training to shrink the receptive fields and encode the unisensory likelihood functions. Hence, after training, each unisensory area realizes a maximum likelihood estimate of stimulus position (auditory or visual). In cross-modal conditions, the same learning rule can encode information on prior probability into the cross-modal synapses. Computer simulations confirm the theoretical results and show that the proposed network can realize a maximum likelihood estimate of auditory (or visual) positions in unimodal conditions and a Bayesian estimate, with moderate deviations from optimality, in cross-modal conditions. Furthermore, the model explains the ventriloquism illusion and, looking at the activity in the multimodal neurons, explains the automatic reweighting of auditory and visual inputs on a trial-by-trial basis, according to the reliability of the individual cues.

Predicting the Multisensory Consequences of One's Own Action: BOLD Suppression in Auditory and Visual Cortices.

Predictive mechanisms are essential to successfully interact with the environment and to compensate for delays in the transmission of neural signals. However, whether and how we predict multisensory action outcomes remains largely unknown. Here we investigated the existence of multisensory predictive mechanisms in a context where actions have outcomes in different modalities. During fMRI data acquisition auditory, visual and auditory-visual stimuli were presented in active and passive conditions. In the active condition, a self-initiated button press elicited the stimuli with variable short delays (0-417ms) between action and outcome, and participants had to detect the presence of a delay for auditory or visual outcome (task modality). In the passive condition, stimuli appeared automatically, and participants had to detect the number of stimulus modalities (unimodal/bimodal). For action consequences compared to identical but unpredictable control stimuli we observed suppression of the blood oxygen level depended (BOLD) response in a broad network including bilateral auditory and visual cortices. This effect was independent of task modality or stimulus modality and strongest for trials where no delay was detected (undetected<detected). In bimodal vs. unimodal conditions we found activation differences in the left cerebellum for detected vs. undetected trials and an increased cerebellar-sensory cortex connectivity. Thus, action-related predictive mechanisms lead to BOLD suppression in multiple sensory brain regions. These findings support the hypothesis of multisensory predictive mechanisms, which are probably conducted in the left cerebellum.

Speech Recognition and Cognitive Skills in Bimodal Cochlear Implant Users

To examine the relation between speech recognition and cognitive skills in bimodal cochlear implant (CI) and hearing aid users. Seventeen bimodal CI users (28-74 years) were recruited to the study. Speech recognition tests were carried out in quiet and in noise. The cognitive tests employed included the Reading Span Test and the Trail Making Test (Daneman & Carpenter, 1980; Reitan, 1958, 1992), measuring working memory capacity and processing speed and executive functioning, respectively. Data were analyzed using paired-sample t tests, Pearson correlations, and partial correlations controlling for age. The results indicate that performance on some cognitive tests predicts speech recognition and that bimodal listening generates a significant improvement in speech in quiet compared to unilateral CI listening. However, the current results also suggest that bimodal listening requires different cognitive skills than does unimodal CI listening. This is likely to relate to the relative difficulty of having to integrate 2 different signals and then map the integrated signal to representations stored in the long-term memory. Even though participants obtained speech recognition benefit from bimodal listening, the results suggest that processing bimodal stimuli involves different cognitive skills than does unimodal conditions in quiet. Thus, clinically, it is important to consider this when assessing treatment outcomes.

Learning novel musical pitch via distributional learning.

Because different musical scales use different sets of intervals and, hence, different musical pitches, how do music listeners learn those that are in their native musical system? One possibility is that musical pitches are acquired in the same way as phonemes, that is, via distributional learning, in which learners infer knowledge from the distributional structure of their input. In this study, we investigate whether novel musical pitch can be acquired distributionally. Nonmusician adults were trained on a continuum spanning a novel musical chord minimal pair (i.e., a novel chord and a mistuned version of that chord) in which the continuum was presented either in a bimodal distribution, with a modal peak at each end of the continuum, or in a unimodal distribution, with a single central modal peak. Discrimination of target minimal pairs was assessed before and after exposure to the distribution. Distributional learning would be said to occur if learners in the bimodal condition, but not those in the unimodal condition, showed evidence of learning, as indexed by improvement in discriminating the minimal pair from pretest to posttest. This indeed was the outcome, suggesting that the building blocks of musical melody-musical pitch-can be acquired using distributional learning. (PsycINFO Database Record