Information In Speech Perception Research Articles

Multisensory and lexical information in speech perception.

Both multisensory and lexical information are known to influence the perception of speech. However, an open question remains: is either source more fundamental to perceiving speech? In this perspective, we review the literature and argue that multisensory information plays a more fundamental role in speech perception than lexical information. Three sets of findings support this conclusion: first, reaction times and electroencephalographic signal latencies indicate that the effects of multisensory information on speech processing seem to occur earlier than the effects of lexical information. Second, non-auditory sensory input influences the perception of features that differentiate phonetic categories; thus, multisensory information determines what lexical information is ultimately processed. Finally, there is evidence that multisensory information helps form some lexical information as part of a phenomenon known as sound symbolism. These findings support a framework of speech perception that, while acknowledging the influential roles of both multisensory and lexical information, holds that multisensory information is more fundamental to the process.

Internal Cognitive Load Differentially Influences Acoustic and Lexical Context Effects in Speech Perception: Evidence From a Population With Attention-Deficit/Hyperactivity Disorder.

To overcome variability in spoken language, listeners utilize various types of context information for disambiguating speech sounds. Context effects have been shown to be affected by cognitive load. However, previous results are mixed regarding the influence of cognitive load on the use of context information in speech perception. We tested a population characterized by an attention-deficit/hyperactivity disorder (ADHD) to better understand the relationship between attention (or internal cognitive load) and context effects. The use of acoustic versus lexical properties of the surrounding signal to disambiguate speech sounds was examined in listeners with ADHD and neurotypical listeners. Compared to neurotypicals, individuals with ADHD relied more strongly on lexical context for speech perception; however, reliance on acoustic context information from speech rate did not differ. These findings confirm that cognitive load impacts the use of high-level but not low-level context information in speech and imply that speech recognition deficits in ADHD likely arise due to impaired higher order cognitive processes.

Processing of linguistic and indexical information in adult cochlear implant users

Normal-hearing (NH) listeners simultaneously make use of both linguistic (speech content) and indexical (talker-specific) information in speech perception. For adult cochlear implant (CI) users, limitations in the talker-specific details conveyed by their devices may impact processing dependences between linguistic and indexical information. The current study examined the extent to which linguistic and indexical information are processed interdependently in adult CI users, using speeded phoneme and talker gender classification tasks. Higher-performing adult CI users (open-set word recognition >65% correct) reported the target vowel (phoneme classification) or target talker gender (gender classification). The non-target dimension varied by the number of talkers (phoneme classification) and the number of words (gender classification). Largely consistent with NH findings, adult CI users displayed asymmetric interference between linguistic and indexical dimensions. Phoneme classification was less accurate and slower with high talker variability. In contrast, gender classification was not affected by word variability. Processing dependencies also varied across individual CI users and related to auditory and cognitive-linguistic abilities. These preliminary findings suggest that adult CI users may make use of degraded linguistic and indexical information during speech processing. However, the extent to which they do so may depend on individual auditory and cognitive-linguistic factors.

The clear speech intelligibility benefit for text-to-speech voices: Effects of speaking style and visual guise

This study examined how speaking style and guise influence the intelligibility of text-to-speech (TTS) and naturally produced human voices. Results showed that TTS voices were less intelligible overall. Although using a clear speech style improved intelligibility for both human and TTS voices (using "newscaster" neural TTS), the clear speech effect was stronger for TTS voices. Finally, a visual device guise decreased intelligibility, regardless of voice type. The results suggest that both speaking style and visual guise affect intelligibility of human and TTS voices. Findings are discussed in terms of theories about the role of social information in speech perception.

The role of cue weighting and contextual information in speech perception by younger and older adults

Normal-hearing older listeners are as accurate as younger listeners when perceiving native English words in quiet despite challenges in temporal processing. Older listeners may compensate for the declined use of fine-grained temporal cues by reducing the weight of temporal cues (VOT) and increase the reliance on other acoustic correlates (F0) of the sound contrast. In Experiment 1, younger (age 18–25) and older (age 55–65) normal-hearing listeners participate in an online 2AFC identification task with /d/-/t/ contrast varying in both VOT and F0. We predict that, while both younger and older listeners rely more on VOT than on F0, older listeners, because of their reduced temporal processing abilities, rely on F0 to a larger degree than younger listeners. Temporal processing not only involves local durational cues of the target segments, but also global contextual cues such as speaking rate. In Experiment 2, the same listeners complete another online 2AFC identification task with /dɑ/-/tɑ/ syllables that vary in VOT and vowel duration (short versus long). We predict that older listeners exhibit a smaller shift in the /d/-/t/ category boundary between the long and short vowel durations than younger listeners since older adults are less sensitive to contextual temporal information.

Integration of contextual-pragmatic and phonetic information in speech perception: An eye-tracking study

Pragmatic information, such as inferences regarding upcoming coreference, has been shown to influence phonetic perception (Rohde & Ettlinger, 2012). Pragmatic information, however, comes in many forms. Using a Visual World Paradigm, tracking listeners’ categorical responses and the time-course of information integration via eye movements, we investigated whether and how a different kind of pragmatic information, the contrastive function of prenominal adjectives (Sedivy, Tanenhaus, Chambers, & Carlson, 1999), can affect listeners’ perception of voicing in initial plosives. Our results suggest that the pragmatic contrast inference did not affect the behavioral judgments on phonetic categorization, but it did have an (albeit limited) influence during the online processing of voice onset time (VOT). Our findings suggest that different kinds of higher-level pragmatic inferences are not uniform in how (successfully) they are integrated with low-level phonetic properties in real-time comprehension.

The influence of tone language experience and speech style on the use of intonation in language discrimination.

This work tests whether listeners' use of suprasegmental information in speech perception is modulated by language background and speech style. Native Mandarin (tone language) and Malay (non-tone language) listeners completed an AX language discrimination task with four levels of signal degradation and two speech styles. Listeners in both groups showed more benefit from pitch information in read than in spontaneous speech. Mandarin listeners showed a greater benefit than Malay listeners from the inclusion of f0 information in a segmentally degraded signal, suggesting that experience with lexical tone may extend to increased attention and/or sensitivity to phrase-level pitch contours.

The Important Role of Contextual Information in Speech Perception in Cochlear Implant Users and Its Consequences in Speech Tests.

This study investigated the role of contextual information in speech intelligibility, the influence of verbal working memory on the use of contextual information, and the suitability of an ecologically valid sentence test containing contextual information, compared with a CNC (Consonant-Nucleus-Consonant) word test, in cochlear implant (CI) users. Speech intelligibility performance was assessed in 50 postlingual adult CI users on sentence lists and on CNC word lists. Results were compared with a normal-hearing (NH) group. The influence of contextual information was calculated from three different context models. Working memory capacity was measured with a Reading Span Test. CI recipients made significantly more use of contextual information in recognition of CNC words and sentences than NH listeners. Their use of contextual information in sentences was related to verbal working memory capacity but not to age, indicating that the ability to use context is dependent on cognitive abilities, regardless of age. The presence of context in sentences enhanced the sensitivity to differences in sensory bottom-up information but also increased the risk of a ceiling effect. A sentence test appeared to be suitable in CI users if word scoring is used and noise is added for the best performers.

Mandarin-Speaking Children's Speech Recognition: Developmental Changes in the Influences of Semantic Context and F0 Contours.

The goal of this developmental speech perception study was to assess whether and how age group modulated the influences of high-level semantic context and low-level fundamental frequency (F0) contours on the recognition of Mandarin speech by elementary and middle-school-aged children in quiet and interference backgrounds. The results revealed different patterns for semantic and F0 information. One the one hand, age group modulated significantly the use of F0 contours, indicating that elementary school children relied more on natural F0 contours than middle school children during Mandarin speech recognition. On the other hand, there was no significant modulation effect of age group on semantic context, indicating that children of both age groups used semantic context to assist speech recognition to a similar extent. Furthermore, the significant modulation effect of age group on the interaction between F0 contours and semantic context revealed that younger children could not make better use of semantic context in recognizing speech with flat F0 contours compared with natural F0 contours, while older children could benefit from semantic context even when natural F0 contours were altered, thus confirming the important role of F0 contours in Mandarin speech recognition by elementary school children. The developmental changes in the effects of high-level semantic and low-level F0 information on speech recognition might reflect the differences in auditory and cognitive resources associated with processing of the two types of information in speech perception.

Speechreading in Deaf Adults with Cochlear Implants: Evidence for Perceptual Compensation.

Previous research has provided evidence for a speechreading advantage in congenitally deaf adults compared to hearing adults. A ‘perceptual compensation’ account of this finding proposes that prolonged early onset deafness leads to a greater reliance on visual, as opposed to auditory, information when perceiving speech which in turn results in superior visual speech perception skills in deaf adults. In the current study we tested whether previous demonstrations of a speechreading advantage for profoundly congenitally deaf adults with hearing aids, or no amplificiation, were also apparent in adults with the same deafness profile but who have experienced greater access to the auditory elements of speech via a cochlear implant (CI). We also tested the prediction that, in line with the perceptual compensation account, receiving a CI at a later age is associated with superior speechreading skills due to later implanted individuals having experienced greater dependence on visual speech information. We designed a speechreading task in which participants viewed silent videos of 123 single words spoken by a model and were required to indicate which word they thought had been said via a free text response. We compared congenitally deaf adults who had received CIs in childhood or adolescence (N = 15) with a comparison group of hearing adults (N = 15) matched on age and education level. The adults with CI showed significantly better scores on the speechreading task than the hearing comparison group. Furthermore, within the group of adults with CI, there was a significant positive correlation between age at implantation and speechreading performance; earlier implantation was associated with lower speechreading scores. These results are both consistent with the hypothesis of perceptual compensation in the domain of speech perception, indicating that more prolonged dependence on visual speech information in speech perception may lead to improvements in the perception of visual speech. In addition our study provides metrics of the ‘speechreadability’ of 123 words produced in British English: one derived from hearing adults (N = 61) and one from deaf adults with CI (N = 15). Evidence for the validity of these ‘speechreadability’ metrics come from correlations with visual lexical competition data.

Spatial congruence in multimodal speech perception

A growing literature provides evidence for the importance of synchronicity of cross-modal information in speech perception [e.g., audio-visual, Munhall et al. 1996, Perception & Psychophysics 58 : 351-362; audio-aerotactile, Gick et al. 2010, JASA 128: 342-346; visual-aerotactile, Bicevskis et al. submitted ms]. While considerable work has investigated this role of temporal congruence, no research has directly explored the role of spatial congruence (i.e., co-directionality) of stimulus sources. If perceivers are picking up a localized distal speech event [e.g., Fowler 1986, Status Report of Speech Research: 139-169] cross-modal sources of information are predicted to be more likely to integrate when presented codirectionally than contradirectionally. An audio-aerotactile pairing lends itself well to this question as both modalities can easily be presented laterally. The current study draws on methodology from previous work [Gick & Derrick 2009, Nature 462: 502-504] to ask whether cross-modal integration persists when cross-modal cues are spatially incongruent. Native English perceivers were presented with syllables contrasting in aspiration and embedded in noise, with some tokens accompanied by inaudible air puffs applied to the neck; aerotactile source locations either matched or opposed the spatial direction of the acoustic signal. Implications of results for multimodal integration theories will be discussed. [Funded by NIH Grant DC-02717 and NSERC.]

Compensation for visually specified coarticulation in liquid-stop contexts.

The question of whether speech perceivers use visual coarticulatory information in speech perception remains unanswered, despite numerous past studies. Across different coarticulatory contexts, studies have both detected (e.g., Mitterer in Perception & Psychophysics, 68, 1227-1240, 2006) and failed to detect (e.g., Vroomen & de Gelder in Language and Cognitive Processes, 16, 661-672. doi: 10.1080/01690960143000092 , 2001) visual effects. In this study, we focused on a liquid-stop coarticulatory context and attempted to resolve the contradictory findings of Fowler, Brown, and Mann (Journal of Experimental Psychology: Human Perception and Performance, 26, 877-888. doi: 10.1037/0096-1523.26.3.877 , 2000) and Holt, Stephens, and Lotto (Perception & Psychophysics, 67, 1102-1112. doi: 10.3758/BF03193635 , 2005). We used the original stimuli of Fowler et al. with modifications to the experimental paradigm to examine whether visual compensation can occur when acoustic coarticulatory information is absent (rather than merely ambiguous). We found that perceivers' categorizations of the target changed when coarticulatory information was presented visually using a silent precursor, suggesting that visually presented coarticulatory information can induce compensation. However, we failed to detect this effect when the same visual information was accompanied by an ambiguous auditory precursor, suggesting that these effects are weaker and less robust than auditory compensation. We discussed why this might be the case and examined implications for accounts of coarticulatory compensation.

Neural sensitivity to syllable frequency and mutual information in speech perception and production

Many factors affect our ability to decode the speech signal, including its quality, the complexity of the elements that compose it, as well as their frequency of occurrence and co-occurrence in a language. Syllable frequency effects have been described in the behavioral literature, including facilitatory effects during speech production and inhibitory effects during word recognition, but the neural mechanisms underlying these effects remain largely unknown. The objective of this study was to examine, using functional neuroimaging, the neurobiological correlates of three different distributional statistics in simple 2-syllable nonwords: the frequency of the first and second syllables, and the mutual information between the syllables. We examined these statistics during nonword perception and production using a powerful single-trial analytical approach. We found that repetition accuracy was higher for nonwords in which the frequency of the first syllable was high. In addition, brain responses to distributional statistics were widespread and almost exclusively cortical. Importantly, brain activity was modulated in a distinct manner for each statistic, with the strongest facilitatory effects associated with the frequency of the first syllable and mutual information. These findings show that distributional statistics modulate nonword perception and production. We discuss the common and unique impact of each distributional statistic on brain activity, as well as task differences.

Auditory and visual information in speech perception: A developmental perspective

ABSTRACTThis study investigates the development of audiovisual speech perception from age 4 to 80, analysing the contribution of modality, context and special features of specific language being tested. Data of 77 participants in five age groups is presented in the study. Speech stimuli were introduced via auditory, visual and audiovisual modalities. Monosyllabic meaningful and nonsense words were included in a signal-to-noise ratio of 0 dB. Speech perception accuracy in audiovisual and auditory modalities by age resulted in an inverse U-shape, with lowest performance at ages 4–5 and 65–80. In the visual modality, a clear difference was shown between performance of children (ages 4–5 and 8–9) and adults (age 20 and above). The findings of the current study have important implications for strategic planning in rehabilitation programmes for child and adult speakers of different languages with hearing difficulties.

Predictability effects on speech perception in noise (SPIN) in Korean

This study investigates speech perception in noise (SPIN) in Korean. A new type of Korean SPIN test was developed by adopting a similar format to the English SPIN test. The predictability effects, noise effects and their interactions were examined in order to verify the previous findings based on English. The data from 14 Korean adults collected with this new type of Korean SPIN test confirmed the previous findings: first, the participants’ overall performance was better in low noise conditions than in high noise conditions. Secondly, there was a tendency for highly predictable words to be more accurately perceived than less predictable words especially in high noise conditions. The results were interpreted in such a way that the listeners actively used both types of information: acoustic information and contextual information in speech perception. When the acoustic property of the speech sound was degraded with noise, the listeners took advantage of the linguistic contextual information in their processing of the speech sound. The findings of this study conform to those of the previous studies based on the English SPIN test. In addition, a possible effect of the frequency of target word was also found, calling for further investigation in this field of research in Korean. Implications of the results were also discussed.

The Role of Somatosensory Information in Speech Perception: Imitation Improves Recognition of Disordered Speech.

Perceptual learning paradigms involving written feedback appear to be a viable clinical tool to reduce the intelligibility burden of dysarthria. The underlying theoretical assumption is that pairing the degraded acoustics with the intended lexical targets facilitates a remapping of existing mental representations in the lexicon. This study investigated whether ties to mental representations can be strengthened by way of a somatosensory motor trace. Following an intelligibility pretest, 100 participants were assigned to 1 of 5 experimental groups. The control group received no training, but the other 4 groups received training with dysarthric speech under conditions involving a unique combination of auditory targets, written feedback, and/or a vocal imitation task. All participants then completed an intelligibility posttest. Training improved intelligibility of dysarthric speech, with the largest improvements observed when the auditory targets were accompanied by both written feedback and an imitation task. Further, a significant relationship between intelligibility improvement and imitation accuracy was identified. This study suggests that somatosensory information can strengthen the activation of speech sound maps of dysarthric speech. The findings, therefore, implicate a bidirectional relationship between speech perception and speech production as well as advance our understanding of the mechanisms that underlie perceptual learning of degraded speech.

Audio–visual integration during speech perception in prelingually deafened Japanese children revealed by the McGurk effect

ObjectiveTo investigate the McGurk effect in profoundly deafened Japanese children with cochlear implants (CI) and in normal-hearing children. This was done to identify how children with profound deafness using CI established audiovisual integration during the speech acquisition period. MethodsTwenty-four prelingually deafened children with CI and 12 age-matched normal-hearing children participated in this study. Responses to audiovisual stimuli were compared between deafened and normal-hearing controls. Additionally, responses of the children with CI younger than 6 years of age were compared with those of the children with CI at least 6 years of age at the time of the test. ResultsResponses to stimuli combining auditory labials and visual non-labials were significantly different between deafened children with CI and normal-hearing controls (p<0.05). Additionally, the McGurk effect tended to be more induced in deafened children older than 6 years of age than in their younger counterparts. ConclusionsThe McGurk effect was more significantly induced in prelingually deafened Japanese children with CI than in normal-hearing, age-matched Japanese children. Despite having good speech-perception skills and auditory input through their CI, from early childhood, deafened children may use more visual information in speech perception than normal-hearing children. As children using CI need to communicate based on insufficient speech signals coded by CI, additional activities of higher-order brain function may be necessary to compensate for the incomplete auditory input. This study provided information on the influence of deafness on the development of audiovisual integration related to speech, which could contribute to our further understanding of the strategies used in spoken language communication by prelingually deafened children.

Top-down influences in perception with spectrally degraded resynthesized natural speech

Lexical processes can influence perception of ambiguous phonemes (e.g., Ganong, 1980). To date, research has focused on these influences in quiet conditions with stimuli that have not been degraded. This project examines the interplay between lexical and acoustic information in speech perception, with stimuli in non-degraded and spectrally degraded conditions. Continua of /t, k/ onsets were developed by wavelet resynthesis of natural speech using TandemSTRAIGHT software, concatenated to the vowel × coda portion of words/nonwords, and distorted using AngelSim (TigerCIS) vocoding software. Normal hearing adult participants identified the onset phonemes in non-degraded and spectrally degraded conditions. Identification functions and the effects of lexical status in phoneme perception with spectrally degraded stimuli will be discussed.

Are articulatory commands automatically and involuntarily activated during speech perception?

A recent PNAS article by Yuen et al. (1) presented evidence for articulation-specific effects of auditory speech on subsequent production of spoken syllables (Expt 1). This was interpreted as evidence that motor routines are “automatically” and “involuntarily” activated by heard speech, and the authors concluded their paradigm provides “a behavioral diagnostic for the activation of articulatory information in speech perception.” Electropalatography is an appealing approach to investigating the nature of motor involvement in speech perception. The authors demonstrated that an incongruent alveolar /t/ sound led to a greater proportion of alveolar tongue contact in the initial phase of subsequent production of /s/ and /k/ sounds, relative to a congruent condition. However, the claim that hearing speech evokes automatic motor activation is undermined by the inclusion of an overt phoneme monitoring task on the auditory speech. In Experiment 1, participants heard a distractor syllable, then received a written cue to produce a rhyming target aloud, after which they were visually prompted to indicate whether or not a specific phoneme had occurred in the original distractor. Phoneme monitoring requires segmentation of heard speech into its constituent elements, and it has long been argued that such tasks engage neural systems beyond those associated with normal speech comprehension (2, 3). For example, illiterate individuals can understand speech but cannot perform simple phoneme segmentation (4). Consistent with this perspective, a recent study showed that repetitive transcranial magnetic stimulation (TMS) applied to premotor cortex has a detrimental effect on performance of a task requiring phoneme segmentation, but has no effect on tasks that relied on simpler acoustic processing (e.g., identification of isolated phonemes) (5).

Activation of articulatory information in speech perception

Emerging neurophysiologic evidence indicates that motor systems are activated during the perception of speech, but whether this activity reflects basic processes underlying speech perception remains a matter of considerable debate. Our contribution to this debate is to report direct behavioral evidence that specific articulatory commands are activated automatically and involuntarily during speech perception. We used electropalatography to measure whether motor information activated from spoken distractors would yield specific distortions on the articulation of printed target syllables. Participants produced target syllables beginning with /k/ or /s/ while listening to the same syllables or to incongruent rhyming syllables beginning with /t/. Tongue-palate contact for target productions was measured during the articulatory closure of /k/ and during the frication of /s/. Results revealed "traces" of the incongruent distractors on target productions, with the incongruent /t/-initial distractors inducing greater alveolar contact in the articulation of /k/ and /s/ than the congruent distractors. Two further experiments established that (i) the nature of this interference effect is dependent specifically on the articulatory properties of the spoken distractors; and (ii) this interference effect is unique to spoken distractors and does not arise when distractors are presented in printed form. Results are discussed in terms of a broader emerging framework concerning the relationship between perception and action, whereby the perception of action entails activation of the motor system.