Speechreading Performance Research Articles

Teaching Children With Hearing Loss to Recognize Speech: Gains Made With Computer-Based Auditory and/or Speechreading Training.

Transfer appropriate processing (TAP) refers to a general finding that training gains are maximized when training and testing are conducted under the same conditions. The present study tested the extent to which TAP applies to speech perception training in children with hearing loss. Specifically, we assessed the benefits of computer-based speech perception training games for enhancing children's speech recognition by comparing three training groups: auditory training (AT), audiovisual training (AVT), and a combination of these two (AT/AVT). We also determined whether talker-specific training, as might occur when children train with the speech of a next year's classroom teacher, leads to better recognition of that talker's speech and if so, the extent to which training benefits generalize to untrained talkers. Consistent with TAP theory, we predicted that children would improve their ability to recognize the speech of the trained talker more than that of three untrained talkers and, depending on their training group, would improve more on an auditory-only (listening) or audiovisual (speechreading) speech perception assessment, that matched the type of training they received. We also hypothesized that benefit would generalize to untrained talkers and to test modalities in which they did not train, albeit to a lesser extent. Ninety-nine elementary school aged children with hearing loss were enrolled into a randomized control trial with a repeated measures A-A-B experimental mixed design in which children served as their own control for the assessment of overall benefit of a particular training type and three different groups of children yielded data for comparing the three types of training. We also assessed talker-specific learning and transfer of learning by including speech perception tests with stimuli spoken by the talker with whom a child trained and stimuli spoken by three talkers with whom the child did not train and by including speech perception tests that presented both auditory (listening) and audiovisual (speechreading) stimuli. Children received 16 hr of gamified training. The games provided word identification and connected speech comprehension training activities. Overall, children showed significant improvement in both their listening and speechreading performance. Consistent with TAP theory, children improved more on their trained talker than on the untrained talkers. Also consistent with TAP theory, the children who received AT improved more on the listening than the speechreading. However, children who received AVT improved on both types of assessment equally, which is not consistent with our predictions derived from a TAP perspective. Age, language level, and phonological awareness were either not predictive of training benefits or only negligibly so. The findings provide support for the practice of providing children who have hearing loss with structured speech perception training and suggest that future aural rehabilitation programs might include teacher-specific speech perception training to prepare children for an upcoming school year, especially since training will generalize to other talkers. The results also suggest that benefits of speech perception training were not significantly related to age, language level, or degree of phonological awareness. The findings are largely consistent with TAP theory, suggesting that the more aligned a training task is with the desired outcome, the more likely benefit will accrue.

Phonological store and speechreading performance of Chinese students with hearing impairment

ABSTRACT This study presents three experiments to examine the role of the phonological store component of working memory in the speechreading performance of students with hearing impairment (HI) in China. In Experiment 1, 86 high school students with HI completed an immediate serial recall task with four lists of monosyllabic words that differed in phonological and visual similarities. In Experiment 2 and Experiment 3, 40 participants divided into high or low phonological store capacity (PS) and 40 participants divided into high or low visual phonological story capacity (VPS) completed a speechreading test at the word, phrase and sentence levels. Results revealed that (1) immediate serial recall showed effects of phonological and visual similarity and their interaction; (2) there was no significant effect of phonological store capacities on speechreading; and (3) there was a significant effect of visual phonological store capacities on accuracy but not speed of speechreading. These findings point to a general phonological store system for visual orthographic coding and phonological coding that students with HI engage in speechreading in Chinese. It provides evidence for the contention that the visual-based coding has a more direct impact on speechreading performance of Chinese students with HI than the speech-based coding.

The effect of speech training experiences on speechreading skills of Chinese children with hearing impairment

ABSTRACTSpeechreading contributes significantly to effective communication, and persons with hearing impairment (HI) may need to rely more on speechreading. However, whether they may benefit from training/practice to improve their speechreading performance remains unclear. In this study, we examine the effect of speech training on speechreading performance of children with HI in China, and how such effect, if any, may be influenced by age. Fifty-nine HI children with speech training experiences, and fifty-eight HI children without speech training experiences completed tasks measuring their ability to speechread Chinese at the linguistic levels of words, phrases, and sentences. The children ranged from 7 to 14 years old, and were divided into four age groups by two-year age intervals. Both accuracy rate and response time data were collected. Results revealed three findings: (1) HI children with speech training experiences speechread more accurately but more slowly than those without speech training experiences; (2) while speechreading performance generally improved with age, age didn’t alter the relative performance between the two groups; and (3) speechreading performance was best for phrases, and worst for sentences. These findings suggest that HI children benefit from speech training to improve their speechreading performance, and their speechreading performance is influenced by age, linguistic level, and the specific measure used.

Enhanced Speechreading Performance in Young Hearing Aid Users in China.

Purpose The study was designed primarily to determine if the use of hearing aids (HAs) in individuals with hearing impairment in China would affect their speechreading performance. Method Sixty-seven young adults with hearing impairment with HAs and 78 young adults with hearing impairment without HAs completed newly developed Chinese speechreading tests targeting 3 linguistic levels (i.e., words, phrases, and sentences). Results Groups with HAs were more accurate at speechreading than groups without HA across the 3 linguistic levels. For both groups, speechreading accuracy was higher for phrases than words and sentences, and speechreading speed was slower for sentences than words and phrases. Furthermore, there was a positive correlation between years of HA use and the accuracy of speechreading performance; longer HA use was associated with more accurate speechreading. Conclusions Young HA users in China have enhanced speechreading performance over their peers with hearing impairment who are not HA users. This result argues against the perceptual dependence hypothesis that suggests greater dependence on visual information leads to improvement in visual speech perception.

The effect of hearing status on speechreading performance of Chinese adolescents

ABSTRACT The effect of hearing status on the ability to speechread is poorly understood, and current findings are inconclusive regarding differences in speechreading performance between children and adults with hearing impairment and those with normal hearing. In this study, we investigated the effect of hearing status on speechreading skills in Chinese adolescents. Thirty seven severely deaf students with a mean pure-tone average of 93 dB hearing threshold level and 21 hearing controls aged 16 completed tasks measuring their speechreading of simplex finals (monophthongs), complex finals (diphthongs or vowel + nasal constellations) and initials (consonants) in Chinese. Both accuracy rate and response time data were collected. Results showed no significant difference in accuracy between groups. By contrast, deaf individuals were significantly faster at speechreading than their hearing controls. In addition, for both groups, performance on speechreading simplex finals was faster and more accurate than complex finals, which in turn was better than initial consonants. We conclude that speechreading skills in Chinese adolescents are influenced by hearing status, characteristics of sounds to be identified, as well as the measures used.

Eye Movements During Visual Speech Perception in Deaf and Hearing Children.

For children who are born deaf, lipreading (speechreading) is an important source of access to spoken language. We used eye tracking to investigate the strategies used by deaf (n = 33) and hearing 5–8‐year‐olds (n = 59) during a sentence speechreading task. The proportion of time spent looking at the mouth during speech correlated positively with speechreading accuracy. In addition, all children showed a tendency to watch the mouth during speech and watch the eyes when the model was not speaking. The extent to which the children used this communicative pattern, which we refer to as social‐tuning, positively predicted their speechreading performance, with the deaf children showing a stronger relationship than the hearing children. These data suggest that better speechreading skills are seen in those children, both deaf and hearing, who are able to guide their visual attention to the appropriate part of the image and in those who have a good understanding of conversational turn‐taking.

Speechreading in Deaf Adults with Cochlear Implants: Evidence for Perceptual Compensation.

Previous research has provided evidence for a speechreading advantage in congenitally deaf adults compared to hearing adults. A ‘perceptual compensation’ account of this finding proposes that prolonged early onset deafness leads to a greater reliance on visual, as opposed to auditory, information when perceiving speech which in turn results in superior visual speech perception skills in deaf adults. In the current study we tested whether previous demonstrations of a speechreading advantage for profoundly congenitally deaf adults with hearing aids, or no amplificiation, were also apparent in adults with the same deafness profile but who have experienced greater access to the auditory elements of speech via a cochlear implant (CI). We also tested the prediction that, in line with the perceptual compensation account, receiving a CI at a later age is associated with superior speechreading skills due to later implanted individuals having experienced greater dependence on visual speech information. We designed a speechreading task in which participants viewed silent videos of 123 single words spoken by a model and were required to indicate which word they thought had been said via a free text response. We compared congenitally deaf adults who had received CIs in childhood or adolescence (N = 15) with a comparison group of hearing adults (N = 15) matched on age and education level. The adults with CI showed significantly better scores on the speechreading task than the hearing comparison group. Furthermore, within the group of adults with CI, there was a significant positive correlation between age at implantation and speechreading performance; earlier implantation was associated with lower speechreading scores. These results are both consistent with the hypothesis of perceptual compensation in the domain of speech perception, indicating that more prolonged dependence on visual speech information in speech perception may lead to improvements in the perception of visual speech. In addition our study provides metrics of the ‘speechreadability’ of 123 words produced in British English: one derived from hearing adults (N = 61) and one from deaf adults with CI (N = 15). Evidence for the validity of these ‘speechreadability’ metrics come from correlations with visual lexical competition data.

Do gender differences in audio-visual benefit and visual influence in audio-visual speech perception emerge with age?

Gender and age have been found to affect adults’ audio-visual (AV) speech perception. However, research on adult aging focuses on adults over 60 years, who have an increasing likelihood for cognitive and sensory decline, which may confound positive effects of age-related AV-experience and its interaction with gender. Observed age and gender differences in AV speech perception may also depend on measurement sensitivity and AV task difficulty. Consequently both AV benefit and visual influence were used to measure visual contribution for gender-balanced groups of young (20–30 years) and middle-aged adults (50–60 years) with task difficulty varied using AV syllables from different talkers in alternative auditory backgrounds. Females had better speech-reading performance than males. Whereas no gender differences in AV benefit or visual influence were observed for young adults, visually influenced responses were significantly greater for middle-aged females than middle-aged males. That speech-reading performance did not influence AV benefit may be explained by visual speech extraction and AV integration constituting independent abilities. Contrastingly, the gender difference in visually influenced responses in middle adulthood may reflect an experience-related shift in females’ general AV perceptual strategy. Although young females’ speech-reading proficiency may not readily contribute to greater visual influence, between young and middle-adulthood recurrent confirmation of the contribution of visual cues induced by speech-reading proficiency may gradually shift females AV perceptual strategy toward more visually dominated responses.

Speech-reading performance of hearing-impaired students in China

This study investigated speech-reading performance of hearing-impaired (HI) students in China. Eighteen students [mean age = 16;4 (age;month)] with HI (averages of 89.5 dB, SD = 13.21) in primarily oral-teaching classrooms and 47 students (mean age = 15;6) with HI (averages of 92 dB, SD = 16.77) from primarily sign-teaching classrooms were asked to identify initials (consonants), simplex finals (monophthongs), and complex finals in standard Chinese (the common language) via speech reading. Both accuracy rate and response time data were collected. Results show that in comparison to students instructed in the sign-based approach, students from oral approach classrooms showed more accurate identification of vowels, but not of consonants. However, the two groups of students did not differ in their speed of identification. In addition, the overall levels of identification accuracy rose, whereas the response times fell, for simplex finals, complex finals, and initial consonants. These results suggest that oral approaches to education for hearing-impaired children facilitate their speech-reading performance for vowels but not for consonants, providing initial evidence from a non-European language for vowel-based visual speech perception training protocols.

Don’t Listen With Your Mouth Full: The Role of Facial Motor Action in Visual Speech Perception

Theories of speech perception agree that visual input enhances the understanding of speech but disagree on whether physically mimicking the speaker improves understanding. This study investigated whether facial motor mimicry facilitates visual speech perception by testing whether blocking facial motor action impairs speechreading performance. Thirty-five typically developing children (19 boys; 16 girls; M age = 7 years) completed the Revised Craig Lipreading Inventory under two conditions. While observing silent videos of 15 words being spoken, participants either held a tongue depressor horizontally with their teeth (blocking facial motor action) or squeezed a ball with one hand (allowing facial motor action). As hypothesized, blocking motor action resulted in fewer correctly understood words than that of the control task. The results suggest that facial mimicry or other methods of facial action support visual speech perception in children. Future studies on the impact of motor action on the typical and atypical development of speech perception are warranted.

What is the McGurk effect?

What is the McGurk effect?

Comparison of informational vs. energetic masking effects on speechreading performance.

The effects of two types of auditory distracters (steady-state noise vs. four-talker babble) on visual-only speechreading accuracy were tested against a baseline (silence) in 23 participants with above-average speechreading ability. Their task was to speechread high frequency Swedish words. They were asked to rate their own performance and effort, and report how distracting each type of auditory distracter was. Only four-talker babble impeded speechreading accuracy. This suggests competition for phonological processing, since the four-talker babble demands phonological processing, which is also required for the speechreading task. Better accuracy was associated with lower self-rated effort in silence; no other correlations were found.

Longitudinal Speech Perception and Language Performance in Pediatric Cochlear Implant Users

Few studies have examined the long-term effect of age at implantation on outcomes using multiple data points in children with cochlear implants. The goal of this study was to determine whether age at implantation has a significant, lasting impact on speech perception, language, and reading performance for children with prelingual hearing loss. A linear mixed-model framework was used to determine the effect of age at implantation on speech perception, language, and reading abilities in 83 children with prelingual hearing loss who received cochlear implants by the age of 4 years. The children were divided into two groups based on their age at implantation: (1) under 2 years of age and (2) between 2 and 3.9 years of age. Differences in model-specified mean scores between groups were compared at annual intervals from 5 to 13 years of age for speech perception, and 7 to 11 years of age for language and reading. After controlling for communication mode, device configuration, and preoperative pure-tone average, there was no significant effect of age at implantation for receptive language by 8 years of age, expressive language by 10 years of age, reading by 7 years of age. In terms of speech-perception outcomes, significance varied between 7 and 13 years of age, with no significant difference in speech-perception scores between groups at ages 7, 11, and 13 years. Children who used oral communication (OC) demonstrated significantly higher speech-perception scores than children who used total communication (TC). OC users tended to have higher expressive language scores than TC users, although this did not reach significance. There was no significant difference between OC and TC users for receptive language or reading scores. Speech perception, language, and reading performance continue to improve over time for children implanted before 4 years of age. The present results indicate that the effect of age at implantation diminishes with time, particularly for higher-order skills such as language and reading. Some children who receive cochlear implants after the age of 2 years have the capacity to approximate the language and reading skills of their earlier-implanted peers, suggesting that additional factors may moderate the influence of age at implantation on outcomes over time.

Investigating the impact of lip visibility and talking style on speechreading performance

It has long been known that visual information from a talker’s mouth and face plays an important role in the perception and understanding of spoken language. The reported experiments explore the impact of lip visibility (Experiments 1 & 2) and speaking style (Experiment 2) on talker speechreadability. Specifically we compare speechreading performance (words in Experiment 1; sentences in Experiment 2 with low level auditory input) from talkers with natural lips, with brightly coloured lips and with concealed lips. Results reveal that highlighting the lip area by the application of lipstick or concealer improves speechreading, relative to natural lips. Furthermore, speaking in a clear (rather than conversational) manner improves speechreading performance, with no interaction between lip visibility and speaking style. Results are discussed in relation to practical methods of improving speechreading and in relation to attention and movement parameters.

Early experience of Cued Speech enhances speechreading performance in deaf

It is known that deaf individuals usually outperform normal hearing subjects in speechreading; however, the underlying reasons remain unclear. In the present study, speechreading performance was assessed in normal hearing participants (NH), deaf participants who had been exposed to the Cued Speech (CS) system early and intensively, and deaf participants exposed to oral language without Cued Speech (NCS). Results show a gradation in performance with highest performance in CS, then in NCS, and finally NH participants. Moreover, error analysis suggests that speechreading processing is more accurate in the CS group than in the other groups. Given that early and intensive CS has been shown to promote development of accurate phonological processing, we propose that the higher speechreading results in Cued Speech users are linked to a better capacity in phonological decoding of visual articulators.

An analysis of British regional accent and contextual cue effects on speechreading performance

The aim of this paper was to examine the effect of regional accent on speechreading accuracy and the utility of contextual cues in reducing accent effects. Study 1: Participants were recruited from Nottingham ( n = 24) and Glasgow ( n = 17). Their task was to speechread 240 visually presented sentences spoken by 12 talkers, half with a Glaswegian accent, half a Nottingham accent. Both participant groups found the Glaswegian talkers less intelligible ( p < 0.05). A significant interaction between participant location and accent type ( p < 0.05) indicated that both participant groups showed an advantage for speechreading talkers with their own accent over the opposite group. Study 2: Participants were recruited from Nottingham ( n = 15). The same visual sentences were used, but each one was presented with a contextual cue. The results showed that speechreading performance was significantly improved when a contextual cue was used ( p < 0.05). However the Nottingham observers still found the Glaswegian talkers less intelligible than the Nottingham talkers ( p < 0.05). The findings of this paper suggest that accent type may have an influence upon visual speech intelligibility and as such may impact upon the design, and results, of tests of speechreading ability.

Developmental factors and the non-native speaker effect in auditory-visual speech perception

The study investigated language and developmental factors in the use of visual information in audiovisual speech perception in speakers from different language backgrounds. Mandarin-Chinese and English adults and 8- to 9-year-old children were presented with /ba/, /da/, and /ga/ tokens spoken by two English and two Mandarin-Chinese speakers. A syllable identification task was presented in auditory-only, visual-only, and audiovisual (congruent and incongruent) conditions in clear and in noise. The results showed an increase in the use of visual information in adults relative to children in both the Chinese and English groups. In addition, a positive correlation between the total visual effect and speechreading performance was found, suggesting that the smaller visual influence in the bimodal condition for children might be accounted by their less sophisticated speechreading ability. In regard to the language factor, it was found that Chinese perceivers use visual information in their audiovisual speech processing to the same extent as English perceivers. Finally, there was evidence for a "non-native speaker effect" (i.e., stronger visual effect for non-native speech stimuli), but only for the English participants. Results from the current study suggest that the visual appearance of individual speakers and the acoustic-phonetic properties of specific languages should be considered in future cross-language studies.

Does face familiarity influence speechreadability?

The impact of face familiarity on the speechreadability of faces is examined. First we measured the baseline speechreading performance of participants, from unfamiliar faces. Next, participants were familiarized with the face and voice of either the same or a different speaker, or were asked to take part in a word puzzle instead. Speechreading performance was measured again, before participants completed a further period of familiarization (or puzzle completion) and a final speechreading performance task. Results showed that speechreading performance increased overall with practice but that performance increased significantly more as participants became increasingly familiar with the same speaker. Our findings demonstrate the importance of talker-specific variations or other instance-based characteristics and suggest that these are a useful source of information for speechreading.

Evidence that cochlear-implanted deaf patients are better multisensory integrators.

The cochlear implant (CI) is a neuroprosthesis that allows profoundly deaf patients to recover speech intelligibility. This recovery goes through long-term adaptative processes to build coherent percepts from the coarse information delivered by the implant. Here we analyzed the longitudinal postimplantation evolution of word recognition in a large sample of CI users in unisensory (visual or auditory) and bisensory (visuoauditory) conditions. We found that, despite considerable recovery of auditory performance during the first year postimplantation, CI patients maintain a much higher level of word recognition in speechreading conditions compared with normally hearing subjects, even several years after implantation. Consequently, we show that CI users present higher visuoauditory performance when compared with normally hearing subjects with similar auditory stimuli. This better performance is not only due to greater speechreading performance, but, most importantly, also due to a greater capacity to integrate visual input with the distorted speech signal. Our results suggest that these behavioral changes in CI users might be mediated by a reorganization of the cortical network involved in speech recognition that favors a more specific involvement of visual areas. Furthermore, they provide crucial indications to guide the rehabilitation of CI patients by using visually oriented therapeutic strategies.

Evaluation of an Audiovisual-FM System: Investigating the Interaction Between Illumination Level and a Talker’s Skin Color on Speech-Reading Performance

A program designed to evaluate the benefits of an audiovisual-frequency modulated (FM) system led to some questions concerning the effects of illumination level and a talker's skin color on speech-reading performance. To address those issues, the speech of a Caucasian female was videotaped under 2 conditions: a light skin color condition and a dark skin color condition. For the latter condition, makeup was applied to the talker's face. For both skin color conditions, the talker was recorded while speaking sentences under 7 different levels of illumination: 2, 3, 4, 16, 60, 256, and 600 footcandles (fc). Fifteen participants completed the speech perception task in a visual-only modality. The results revealed a significant interaction of illumination level and skin color. For the light skin color condition, speech-reading performance improved systematically as the illumination level increased from 3 to 16 fc. For the dark skin color condition, no differences in speech-reading performance were observed between the 2-fc and the 3-fc conditions. However, a large improvement in speech-reading performance was observed as the illumination level increased from 4 fc to 16 fc. It is speculated that in addition to an overall effect of illumination level, the contrast in luminance at the level of the talker's face has an effect on speech-reading performance.