Visual-only Research Articles

EXPRESS: Memory for health information: Influences of age, hearing aids, and multisensory presentation.

We investigated how presenting online health information in different modalities can influence memory, as this may be particularly important for older adults who may need to make regular decisions about health, and could also face additional challenges such as memory deficits and sensory impairment (hearing loss). We tested whether, as predicted by some literature, older adults would disproportionately benefit from audio-visual (AV) information compared with visual-only (VO) or auditory-only (AO) information, relative to young adults. Participants were 78 young adults (aged 18-30 years old, mean=25.50 years), 78 older adults with normal hearing (aged 65-80 years old, mean=68.34 years), and 78 older adults who wear hearing aids (aged 65-79 years old, mean=70.89 years). There were no significant differences in the amount of information remembered across modalities (AV, VO, AO), no differences across participant groups, and we did not find the predicted interaction between participant group and modality. The older-adult groups performed worse than young adults on background measures of cognition, with the exception of a vocabulary test, suggesting that they may have been using strategies based on prior knowledge and experience to compensate for cognitive and/or sensory deficits. The findings indicate that cost-effective, text-based websites may be just as useful as those with edited videos for conveying health information to all age groups, and hearing aid users.

Speech perception: Auditory and visual cue integration in children with and without phonological disorder in voiceless fricatives

ABSTRACT The literature reports contradictory results regarding the influence of visual cues on speech perception tasks in children with phonological disorder (PD). This study aimed to compare the performance of children with (n = 15) and without PD (n = 15) in audiovisual perception task in voiceless fricatives. Assuming that PD could be associated with an inability to integrate phonological information from two sensory sources, we presumed that children with PD would present difficulties in integrating auditory and visual cues compared to typical children. A syllable identification task was conducted. The stimuli were presented according to four conditions: auditory-only (AO); visual-only (VO); audiovisual congruent (AV+); and audiovisual incongruent (AV-). The percentages of correct answers and the respective reaction times in the AO, VO, and AV+ conditions were considered for the analysis. The correct percentage of auditory stimuli was considered for the AV- condition, as well as the percentage of perceptual preference: auditory, visual, and/or illusion (McGurk effect), with the respective reaction time. In comparing the four conditions, children with PD presented a lower number of correct answers and longer reaction time than children with typical development, mainly for the VO. Both groups showed a preference for auditory stimuli for the AV- condition. However, children with PD showed higher percentages for visual perceptual preference and the McGurk effect than typical children. The superiority of typical children over PD children in auditory-visual speech perception depends on type of stimuli and condition of presentation.

Visual Hallucination Elevates Speech Recognition

Due to the detrimental impact of noise on the conventional audio speech recognition (ASR) task, audio-visual speech recognition~(AVSR) has been proposed by incorporating both audio and visual video signals. Although existing methods have demonstrated that the aligned visual input of lip movements can enhance the robustness of AVSR systems against noise, the paired videos are not always available during inference, leading to the problem of the missing visual modality, which restricts their practicality in real-world scenarios. To tackle this problem, we propose a Discrete Feature based Visual Generative Model (DFVGM) which exploits semantic correspondences between the audio and visual modalities during training, generating visual hallucinations in lieu of real videos during inference. To achieve that, the primary challenge is to generate the visual hallucination given the noisy audio while preserving semantic correspondences with the clean speech. To tackle this challenge, we start with training the audio encoder in the Audio-Only (AO) setting, which generates continuous semantic features closely associated with the linguistic information. Simultaneously, the visual encoder is trained in the Visual-Only (VO) setting, producing visual features that are phonetically related. Next, we employ K-means to discretize the continuous audio and visual feature spaces. The discretization step allows DFVGM to capture high-level semantic structures that are more resilient to noise and generate visual hallucinations with high quality. To evaluate the effectiveness and robustness of our approach, we conduct extensive experiments on two publicly available datasets. The results demonstrate that our method achieves a remarkable 53% relative reduction (30.5%->12.9%) in Word Error Rate (WER) on average compared to the current state-of-the-art Audio-Only (AO) baselines while maintaining comparable results (< 5% difference) under the Audio-Visual (AV) setting even without video as input.

Comparison between two scoring methods to assess tail damage of docked pig carcasses during postmortem inspection in Ireland.

Tail inspection in the abattoir is a tool to help determine the welfare status of pigs. However, methodologies vary widely. Moreover, meat inspection is moving from palpation and incision towards visual-only (VIS) examination. This study investigated whether a VIS examination was sufficient to detect tail damage compared to handling (HAND), which ensures examination of all aspects of the tail. The severity of tail skin damage (0 [undamaged] - 4 [partial/full loss of tail]) and presence/absence of bruises was scored using both methods after scalding/dehairing of 5498 pig carcasses. There was a good relationship between methods when evaluating tail skin damage (sensitivity, 82.48%; specificity, 99.98%; accuracy, 98.98%; correlation ρ = 0.84). The results were similar for the presence of bruises (sensitivity, 74.98%; specificity, 99.09%; accuracy, 89.94%; correlation ρ = 0.79). However, the percentage of tails classified as undamaged was higher using VIS (69.9%) than HAND (63.55%) examination. Conversely, VIS detected fewer mild lesions (score 1 - 13.64%; score 2 - 11.73%) than HAND (score 1 - 15.21%; score 2 - 15.53%). A higher percentage of bruises was detected using HAND than VIS (37.96% vs. 29.03%). Visual evaluation is a valid alternative to handling evaluation of carcass tail damage and bruising.

Older adults' emotion recognition: No auditory-visual benefit for less clear expressions.

The ability to recognise emotion from faces or voices appears to decline with advancing age. However, some studies have shown that emotion recognition of auditory-visual (AV) expressions is largely unaffected by age, i.e., older adults get a larger benefit from AV presentation than younger adults resulting in similar AV recognition levels. An issue with these studies is that they used well-recognised emotional expressions that are unlikely to generalise to real-life settings. To examine if an AV emotion recognition benefit generalizes across well and less well recognised stimuli, we conducted an emotion recognition study using expressions that had clear or unclear emotion information for both modalities, or clear visual, but unclear auditory information. Older (n = 30) and younger (n = 30) participants were tested on stimuli of anger, happiness, sadness, surprise, and disgust (expressed in spoken sentences) in auditory-only (AO), visual-only (VO), or AV format. Participants were required to respond by choosing one of 5 emotion options. Younger adults were more accurate in recognising emotions than older adults except for clear VO expressions. Younger adults showed an AV benefit even when unimodal recognition was poor. No such AV benefit was found for older adults; indeed, AV was worse than VO recognition when AO recognition was poor. Analyses of confusion responses indicated that older adults generated more confusion responses that were common between AO and VO conditions, than younger adults. We propose that older adults' poorer AV performance may be due to a combination of weak auditory emotion recognition and response uncertainty that resulted in a higher cognitive load.

Seeing lexical tone: Head and face motion in production and perception of Cantonese lexical tones

Previous studies show that lexical tones can be discriminated visually, but the locus of this information is unknown. Here we investigate the role of visual face and head information in the production and perception of the six Cantonese lexical tones. Experiment 1 (speech production) showed that tone category membership is predicted independently by each of three visual components, Head-only, Face-only, and combined Face&Head motion, that Face&Head combined is a better predictor than the sum of Head-only and Face-only, that Head-only was a better predictor than Face-only, and that up-down head motion may be involved. In Experiment 2 (speech perception) discrimination of Cantonese tones and phones (consonants and vowels) was investigated in Auditory-Only (AO), Visual-Only (VO), and Auditory-Visual (AV) presentations of Face-only, Head-only or combined Face&Head information. There was successful visual discrimination of both phones and tones, but in different manners. For VO, face movement (Face-only, Face&Head) was sufficient for Phone discrimination, whereas both head and face (Face&Head) were necessary for Tone discrimination; and Phone discrimination was better for Face-only than Head-only, whereas Tone discrimination was the better for Head-only than Face-only. For visual augmentation of auditory information, (AV-AO)/AO), for Phones there was significant facilitation in Head-only and Face-only, but not Face&Head, whereas for Tones there was significant facilitation only in the Face&Head condition. Overall, head motion is integrally involved in the differential production of Cantonese tones and, as for consonants and vowels, there is visual perception of lexical tone, albeit based on entirely different cues: face motion is sufficient for phone perception (head motion is not necessary), whereas head motion is necessary but not sufficient for tone perception (Face&Head required). Head motion is integral both to the differential production of Cantonese tones and to the visual discrimination of those tones.

Seeing a talking face matters: The relationship between cortical tracking of continuous auditory‐visual speech and gaze behaviour in infants, children and adults

An auditory-visual speech benefit, the benefit that visual speech cues bring to auditory speech perception, is experienced from early on in infancy and continues to be experienced to an increasing degree with age. While there is both behavioural and neurophysiological evidence for children and adults, only behavioural evidence exists for infants – as no neurophysiological study has provided a comprehensive examination of the auditory-visual speech benefit in infants. It is also surprising that most studies on auditory-visual speech benefit do not concurrently report looking behaviour especially since the auditory-visual speech benefit rests on the assumption that listeners attend to a speaker's talking face and that there are meaningful individual differences in looking behaviour. To address these gaps, we simultaneously recorded electroencephalographic (EEG) and eye-tracking data of 5-month-olds, 4-year-olds and adults as they were presented with a speaker in auditory-only (AO), visual-only (VO), and auditory-visual (AV) modes. Cortical tracking analyses that involved forward encoding models of the speech envelope revealed that there was an auditory-visual speech benefit [i.e., AV > (A + V)], evident in 5-month-olds and adults but not 4-year-olds. Examination of cortical tracking accuracy in relation to looking behaviour, showed that infants’ relative attention to the speaker's mouth (vs. eyes) was positively correlated with cortical tracking accuracy of VO speech, whereas adults’ attention to the display overall was negatively correlated with cortical tracking accuracy of VO speech. This study provides the first neurophysiological evidence of auditory-visual speech benefit in infants and our results suggest ways in which current models of speech processing can be fine-tuned.

During Lipreading Training With Sentence Stimuli, Feedback Controls Learning and Generalization to Audiovisual Speech in Noise

This study investigated the effects of external feedback on perceptual learning of visual speech during lipreading training with sentence stimuli. The goal was to improve visual-only (VO) speech recognition and increase accuracy of audiovisual (AV) speech recognition in noise. The rationale was that spoken word recognition depends on the accuracy of sublexical (phonemic/phonetic) speech perception; effective feedback during training must support sublexical perceptual learning. Normal-hearing (NH) adults were assigned to one of three types of feedback: Sentence feedback was the entire sentence printed after responding to the stimulus. Word feedback was the correct response words and perceptually near but incorrect response words. Consonant feedback was correct response words and consonants in incorrect but perceptually near response words. Six training sessions were given. Pre- and posttraining testing included an untrained control group. Test stimuli were disyllable nonsense words for forced-choice consonant identification, and isolated words and sentences for open-set identification. Words and sentences were VO, AV, and audio-only (AO) with the audio in speech-shaped noise. Lipreading accuracy increased during training. Pre- and posttraining tests of consonant identification showed no improvement beyond test-retest increases obtained by untrained controls. Isolated word recognition with a talker not seen during training showed that the control group improved more than the sentence group. Tests of untrained sentences showed that the consonant group significantly improved in all of the stimulus conditions (VO, AO, and AV). Its mean words correct scores increased by 9.2 percentage points for VO, 3.4 percentage points for AO, and 9.8 percentage points for AV stimuli. Consonant feedback during training with sentences stimuli significantly increased perceptual learning. The training generalized to untrained VO, AO, and AV sentence stimuli. Lipreading training has potential to significantly improve adults' face-to-face communication in noisy settings in which the talker can be seen.

Recognition of Accented Speech by Cochlear-Implant Listeners: Benefit of Audiovisual Cues.

When auditory and visual speech information are presented together, listeners obtain an audiovisual (AV) benefit or a speech understanding improvement compared with auditory-only (AO) or visual-only (VO) presentations. Cochlear-implant (CI) listeners, who receive degraded speech input and therefore understand speech using primarily temporal information, seem to readily use visual cues and can achieve a larger AV benefit than normal-hearing (NH) listeners. It is unclear, however, if the AV benefit remains relatively large for CI listeners when trying to understand foreign-accented speech when compared with unaccented speech. Accented speech can introduce changes to temporal auditory cues and visual cues, which could decrease the usefulness of AV information. Furthermore, we sought to determine if the AV benefit was relatively larger in CI compared with NH listeners for both unaccented and accented speech. AV benefit was investigated for unaccented and Spanish-accented speech by presenting English sentences in AO, VO, and AV conditions to 15 CI and 15 age- and performance-matched NH listeners. Performance matching between NH and CI listeners was achieved by varying the number of channels of a noise vocoder for the NH listeners. Because of the differences in age and hearing history of the CI listeners, the effects of listener-related variables on speech understanding performance and AV benefit were also examined. AV benefit was observed for both unaccented and accented conditions and for both CI and NH listeners. The two groups showed similar performance for the AO and AV conditions, and the normalized AV benefit was relatively smaller for the accented than the unaccented conditions. In the CI listeners, older age was associated with significantly poorer performance with the accented speaker compared with the unaccented speaker. The negative impact of age was somewhat reduced by a significant improvement in performance with access to AV information. When auditory speech information is degraded by CI sound processing, visual cues can be used to improve speech understanding, even in the presence of a Spanish accent. The AV benefit of the CI listeners closely matched that of the NH listeners presented with vocoded speech, which was unexpected given that CI listeners appear to rely more on visual information to communicate. This result is perhaps due to the one-to-one age and performance matching of the listeners. While aging decreased CI listener performance with the accented speaker, access to visual cues boosted performance and could partially overcome the age-related speech understanding deficits for the older CI listeners.

Modality Effects on Lexical Encoding and Memory Representations of Spoken Words.

The present study investigated presentation modality differences in lexical encoding and working memory representations of spoken words of older, hearing-impaired adults. Two experiments were undertaken: a memory-scanning experiment and a stimulus gating experiment. The primary objective of experiment 1 was to determine whether memory encoding and retrieval and scanning speeds are different for easily identifiable words presented in auditory-visual (AV), auditory-only (AO), and visual-only (VO) modalities. The primary objective of experiment 2 was to determine if memory encoding and retrieval speed differences observed in experiment 1 could be attributed to the early availability of AV speech information compared with AO or VO conditions. Twenty-six adults over age 60 years with bilateral mild to moderate sensorineural hearing loss participated in experiment 1, and 24 adults who took part in experiment 1 participated in experiment 2. An item recognition reaction-time paradigm (memory-scanning) was used in experiment 1 to measure (1) lexical encoding speed, that is, the speed at which an easily identifiable word was recognized and placed into working memory, and (2) retrieval speed, that is, the speed at which words were retrieved from memory and compared with similarly encoded words (memory scanning) presented in AV, AO, and VO modalities. Experiment 2 used a time-gated word identification task to test whether the time course of stimulus information available to participants predicted the modality-related memory encoding and retrieval speed results from experiment 1. The results of experiment 1 revealed significant differences among the modalities with respect to both memory encoding and retrieval speed, with AV fastest and VO slowest. These differences motivated an examination of the time course of stimulus information available as a function of modality. Results from experiment 2 indicated the encoding and retrieval speed advantages for AV and AO words compared with VO words were mostly driven by the time course of stimulus information. The AV advantage seen in encoding and retrieval speeds is likely due to a combination of robust stimulus information available to the listener earlier in time and lower attentional demands compared with AO or VO encoding and retrieval. Significant modality differences in lexical encoding and memory retrieval speeds were observed across modalities. The memory scanning speed advantage observed for AV compared with AO or VO modalities was strongly related to the time course of stimulus information. In contrast, lexical encoding and retrieval speeds for VO words could not be explained by the time-course of stimulus information alone. Working memory processes for the VO modality may be impacted by greater attentional demands and less information availability compared with the AV and AO modalities. Overall, these results support the hypothesis that the presentation modality for speech inputs (AV, AO, or VO) affects how older adult listeners with hearing loss encode, remember, and retrieve what they hear.

Audiovisual Speech Enhancement via Cross-Modal Suppression of Auditory Association Cortex by Visual Speech

Abstract INTRODUCTION Speech is multisensory. The addition of visual speech to auditory speech greatly improves comprehension, especially under noisy auditory conditions. However, the neural mechanism for this visual enhancement of auditory speech is poorly understood. We used electrocorticography (ECoG) to study how auditory, visual, and audiovisual speech is processed in the posterior superior temporal gyrus (pSTG), an area of auditory association cortex involved in audiovisual speech integration. We hypothesized that early visual mouth movements modulate audiovisual speech integration through a mechanism of cross-modal suppression, suggesting that the pSTG response to early mouth movements should correlate with comprehension benefits gained by the addition of visual speech to auditory speech. METHODS Words were presented under auditory-only (AUD), visual-only (VIS), and audiovisual (AV) conditions to epilepsy patients (n = 8) implanted with intracranial electrodes for phase-2 monitoring. We measured high-frequency broadband activity (75-150 Hz), a marker for local neuronal firing, in 28 electrodes over the pSTG. RESULTS The early neural response to visual-only words was compared to the reduction in neural response seen from AUD to AV words, a reduction correlated with an improvement in speech comprehension that occurs with the addition of visual to auditory speech. In words that showed a comprehension benefit with the addition of visual speech, there was a strong early response to visual speech and a correlation between early visual response and the AUD-AV difference (r = 0.64, P = 104). In words where visual speech did not provide any comprehension benefit, there was a weak early visual response and no correlation (r = 0.18, P = .35). CONCLUSION Words with a visual speech comprehension benefit also elicit a strong neural response to early visual speech in pSTG, while words with no comprehension benefit do not cause a strong early response. This suggests that cross-modal suppression of auditory association cortex (pSTG) by early visual plays an important role in audiovisual speech perception.

Audiovisual benefit in the perception of foreign-accented speech by cochlear-implant users

When visual and auditory speech cues are presented together, listeners typically obtain an audiovisual (AV) benefit, or a boost in speech perception accuracy compared to auditory-only (AO) or visual-only (VO) presentations. Visual cues are particularly important for the speech perception of cochlear-implant (CI) users, who show a greater AV benefit than normal-hearing (NH) listeners. However, whether CI users can utilize visual cues for an AV benefit when the speaker has a foreign accent is unknown. Accented speech can alter auditory (primarily temporal) and visual cues, and is significantly more difficult for CI users in AO listening conditions. We investigated the AV benefit in CI users for unaccented and accented speech, presenting IEEE sentences in VO, AO, and AV conditions. The sentences were spoken by one native English speaker and one non-native English speaker who is a native Spanish-speaker from Peru. Preliminary results reveal that CI users show AV benefits in both unaccented and accented conditions, with substantially larger AV benefits for accented speech. Although visual cues alone may be insufficient for speech perception, CI users may necessitate more reliance on visual cues when presented with auditory cues that have been altered in accented speech.

Effects of clear speech and language background on multimodal perception of English fricatives

Research shows that acoustic modifications in clearly enunciated fricative consonants (relative to the plain, conversational productions) facilitate auditory fricative perception. However, clear-speech effects on visual fricative perception have received less attention. A comparison of auditory and visual (facial) clear-fricative perception is particularly interesting since sibilant fricatives in English are more auditorily salient while non-sibilants are more visually salient. This study thus examines clear-speech effects on audio-visual perception of English sibilant and non-sibilant fricatives. Native English perceivers and non-native perceivers with different L1 fricative inventories (Mandarin, Korean) identified clear and plain fricative-vowel syllables in audio-only (AO), visual-only (VO), and audio-visual (AV) modes. The results across perceiver groups and speech styles showed an overall visual benefit and auditory dominance (AV>AO>VO). Comparisons of styles revealed clear-speech benefits in AO across fricatives and groups, but different patterns were noted in the visual conditions. In VO and AV, clear speech helped the more visually salient non-sibilant identification for native perceivers; however, clear-speech benefits were less prominent in non-natives’ perception of the non-sibilants, which are non-existent in their L1s. These findings are discussed in terms of the relative audio-visual weighting that benefits perception in clear speech as a function of input saliency and perceiver experience.

The influence of auditory-visual speech and clear speech on cross-language perceptual assimilation

The perception of non-native speech is influenced by prior attunement to the native language. Evidence from auditory–only (AO) citation speech research indicates that non-native consonants are perceptually assimilated to native language categories, often causing difficulties in discrimination of non-native speech contrasts. But, as auditory-visual (AV) speech and clear speech have been shown to benefit non-native speech perception, we reasoned that modality and speaking style may also assist to clarify categorical distinctions, and goodness-of-fit differences between contrasting non-native consonants. This was tested by comparing the perceptual assimilation of Sindhi consonants by Australian English monolinguals in AO, AV, and visual-only (VO), clear and citation speech conditions. Although the consonants were perceptually assimilated to native categories similarly across AO and AV conditions, in both speaking styles, categorization consistency was dependent on whether the consonants were categorized in the VO condition. Consonants that were uncategorized in VO were more consistently categorized in AO than AV, and citation speech was more consistently categorized than clear. However, these patterns were reversed for consonants that were categorized in VO, where participants used the visual articulatory information to make more consistent categorization judgments for AV than AO, and clear speech was more consistently categorized than citation. These results suggest that the impact of AV and clear speech on cross-language perception depends on the similarities and differences between the articulatory characteristics of native and non-native consonants, and may have important implications for second language learning, as well as human and machine interaction, via the robustness of automatic speech recognition systems.

Cross-linguistic perception of clearly spoken English tense and lax vowels based on auditory, visual, and auditory-visual information

Studies on acoustic and visual characteristics of English tense and lax vowels show consistent enhancement of tensity contrasts in clear speech. However, the degree to which listeners utilize these enhancements in speech perception remains unclear. The present study addresses this issue by testing speech style effects on tense and lax vowel perception by 23 native English and 30 non-native Mandarin-Chinese listeners in audio-only (AO), audio-visual (AV), and visual-only (VO) stimulus modes. English and Chinese listeners showed similar relative differences in performance by mode (VO < AO < AV) and style (plain < clear). However, the two groups differed in the nature of the interaction between tensity, style, and stimulus mode. English listeners showed advantages for clear speech for both tense and lax vowels in all but VO stimuli, whereas Chinese listeners showed a clear speech advantage only for tense vowels, while clear lax vowels showed no improvement in AO and reduced accuracy in AV and VO. While temporal and spectral acoustic cues may coordinate to preserve or improve tense-lax category identity in clear speech, non-native listeners may not be attending to both dimensions. Further, Chinese listeners' greater reliance on visual information may account for their less accurate lax vowel identification.

Perceptual assimilation of lexical tone: the roles of language experience and visual information.

Using Best’s (1995) perceptual assimilation model (PAM), we investigated auditory–visual (AV), auditory-only (AO), and visual-only (VO) perception of Thai tones. Mandarin and Cantonese (tone-language) speakers were asked to categorize Thai tones according to their own native tone categories, and Australian English (non-tone-language) speakers to categorize Thai tones into their native intonation categories—for instance, question or statement. As comparisons, Thai participants completed a straightforward identification task, and another Australian English group identified the Thai tones using simple symbols. All of the groups also completed an AX discrimination task. Both the Mandarin and Cantonese groups categorized AO and AV Thai falling tones as their native level tones, and Thai rising tones as their native rising tones, although the Mandarin participants found it easier to categorize Thai level tones than did the Cantonese participants. VO information led to very poor categorization for all groups, and AO and AV information also led to very poor categorizations for the English intonation categorization group. PAM’s predictions regarding tone discriminability based on these category assimilation patterns were borne out for the Mandarin group’s AO and AV discriminations, providing support for the applicability of the PAM to lexical tones. For the Cantonese group, however, PAM was unable to account for one specific discrimination pattern—namely, their relatively good performance on the Thai high–rising contrast in the auditory conditions—and no predictions could be derived for the English groups. A full account of tone assimilation will likely need to incorporate considerations of phonetic, and even acoustic, similarity and overlap between nonnative and native tone categories.

Multisensory training can promote or impede visual perceptual learning of speech stimuli: visual-tactile vs. visual-auditory training

In a series of studies we have been investigating how multisensory training affects unisensory perceptual learning with speech stimuli. Previously, we reported that audiovisual (AV) training with speech stimuli can promote auditory-only (AO) perceptual learning in normal-hearing adults but can impede learning in congenitally deaf adults with late-acquired cochlear implants. Here, impeder and promoter effects were sought in normal-hearing adults who participated in lipreading training. In Experiment 1, visual-only (VO) training on paired associations between CVCVC nonsense word videos and nonsense pictures demonstrated that VO words could be learned to a high level of accuracy even by poor lipreaders. In Experiment 2, visual-auditory (VA) training in the same paradigm but with the addition of synchronous vocoded acoustic speech impeded VO learning of the stimuli in the paired-associates paradigm. In Experiment 3, the vocoded AO stimuli were shown to be less informative than the VO speech. Experiment 4 combined vibrotactile speech stimuli with the visual stimuli during training. Vibrotactile stimuli were shown to promote visual perceptual learning. In Experiment 5, no-training controls were used to show that training with visual speech carried over to consonant identification of untrained CVCVC stimuli but not to lipreading words in sentences. Across this and previous studies, multisensory training effects depended on the functional relationship between pathways engaged during training. Two principles are proposed to account for stimulus effects: (1) Stimuli presented to the trainee’s primary perceptual pathway will impede learning by a lower-rank pathway. (2) Stimuli presented to the trainee’s lower rank perceptual pathway will promote learning by a higher-rank pathway. The mechanisms supporting these principles are discussed in light of multisensory reverse hierarchy theory (RHT).

Enhanced audiovisual integration with aging in speech perception: a heightened McGurk effect in older adults

Two experiments compared young and older adults in order to examine whether aging leads to a larger dependence on visual articulatory movements in auditory-visual speech perception. These experiments examined accuracy and response time in syllable identification for auditory-visual (AV) congruent and incongruent stimuli. There were also auditory-only (AO) and visual-only (VO) presentation modes. Data were analyzed only for participants with normal hearing. It was found that the older adults were more strongly influenced by visual speech than the younger ones for acoustically identical signal-to-noise ratios (SNRs) of auditory speech (Experiment 1). This was also confirmed when the SNRs of auditory speech were calibrated for the equivalent AO accuracy between the two age groups (Experiment 2). There were no aging-related differences in VO lipreading accuracy. Combined with response time data, this enhanced visual influence for the older adults was likely to be associated with an aging-related delay in auditory processing.

Audiovisual Speech Integration Does Not Rely on the Motor System: Evidence from Articulatory Suppression, the McGurk Effect, and fMRI

Visual speech influences the perception of heard speech. A classic example of this is the McGurk effect, whereby an auditory /pa/ overlaid onto a visual /ka/ induces the fusion percept of /ta/. Recent behavioral and neuroimaging research has highlighted the importance of both articulatory representations and motor speech regions of the brain, particularly Broca's area, in audiovisual (AV) speech integration. Alternatively, AV speech integration may be accomplished by the sensory system through multisensory integration in the posterior STS. We assessed the claims regarding the involvement of the motor system in AV integration in two experiments: (i) examining the effect of articulatory suppression on the McGurk effect and (ii) determining if motor speech regions show an AV integration profile. The hypothesis regarding experiment (i) is that if the motor system plays a role in McGurk fusion, distracting the motor system through articulatory suppression should result in a reduction of McGurk fusion. The results of experiment (i) showed that articulatory suppression results in no such reduction, suggesting that the motor system is not responsible for the McGurk effect. The hypothesis of experiment (ii) was that if the brain activation to AV speech in motor regions (such as Broca's area) reflects AV integration, the profile of activity should reflect AV integration: AV > AO (auditory only) and AV > VO (visual only). The results of experiment (ii) demonstrate that motor speech regions do not show this integration profile, whereas the posterior STS does. Instead, activity in motor regions is task dependent. The combined results suggest that AV speech integration does not rely on the motor system.

Rating of Intensity of Emotions Across Auditory, Visual, and Auditory-Visual Sensory Domains

The perceived intensity for happy and sad emotions in three sensory domains-AV (auditory-visual), VO (visual-only) and AO (auditory-only)-was compared with 58 undergraduate college students using a within-subject design. Both happy and sad emotions were rated higher in intensity in the AV (auditory-visual) condition compared to the VO (visual-only) and AO (auditory-only) conditions separately. There was no significant difference observed between VO (visual-only) and AO (auditory-only) conditions for either happy or sad emotional stimuli. The implications for these findings are discussed especially with respect to the possibly decreased impact of intensity of emotions in different modes of communication like phone or email.