Understanding Speech Perception Research Articles

The Accent Atlas: Effects of long-term exposure to nonnative accents on adaptive speech perception

As mobility increases and virtual interactions grow, quickly adapting to diverse accents is essential for effective communication. While lab-based accent training promotes rapid adaptation to unfamiliar accents, whether prolonged environmental exposure to linguistic diversity yields similar benefits is unknown. We conducted a large-scale perceptual experiment involving 600 + participants across 15 U.S. states, utilizing a cross-modal word matching task to measure speech recognition in both non-native accented English and native English with background noise. Participants were recruited from linguistically diverse (LD) or linguistically homogeneous (LH) states. Linguistic diversity was defined based on the prevalence of non-English speaking residents (U.S. census data). Our findings were twofold: firstly, consistent with prior studies, only exposure to non-native accented speech—but not exposure to speech in noise—improved recognition of novel talkers with the same accent; secondly, individuals from the LD states outperformed those from LH areas in initial accented speech perception. However, by the end of the experiment, benefits from in-lab accent exposure did not differ significantly between the two groups, suggesting that short-term training could mitigate long-term environmental influences. These results illuminate the interplay between daily linguistic environments and accent adaptation, enhancing our understanding of speech perception's adaptivity in a multilingual context.

Age-related rollover for spectrally degraded temporal speech contrasts

Some speech sounds, or phonemes, are primarily distinguished acoustically from other phonemes by temporal cues. A decline in temporal-processing abilities is a common consequence of aging that might cause a listener to have difficulty differentiating between sounds, especially when the speech is degraded. One type of signal degradation occurs in a cochlear-implant vocoder simulation, which eliminates most of the spectral information in speech while maintaining the temporal envelope. In addition, temporal processing abilities may be challenged at higher presentation levels causing a decrease in performance (i.e., rollover), particularly in older listeners. This study measures the phoneme classification of younger and older listeners of four spectro-temporal contrasts when presented in quiet with and without vocoding at a range of presentation levels. We hypothesized that, due to declines in temporal processing abilities, older listeners would be more susceptible to rollover, measured by a decrease in phoneme classification performance for degraded temporal contrasts as presentation levels increase. We further hypothesized that performance rollover would occur more often for vocoded speech than for unprocessed speech, especially in older listeners. Preliminary data show interactions between speech contrasts, presentation levels, and vocoding. Our results will help facilitate understanding speech perception difficulties in older cochlear-implant users.

Distinct Neural Resource Involvements but Similar Hemispheric Lateralization Patterns in Pre-Attentive Processing of Speaker's Identity and Linguistic Information.

The speaker's identity (who the speaker is) and linguistic information (what the speaker is saying) are essential to daily communication. However, it is unclear whether and how listeners process the two types of information differently in speech perception. The present study adopted a passive oddball paradigm to compare the identity and linguistic information processing concerning neural resource involvements and hemispheric lateralization patterns. We used two female native Mandarin speakers' real and pseudo-Mandarin words to differentiate the identity from linguistic (phonological and lexical) information. The results showed that, in real words, the phonological-lexical variation elicited larger MMN amplitudes than the identity variation. In contrast, there were no significant MMN amplitude differences between the identity and phonological variation in pseudo words. Regardless of real or pseudo words, the identity and linguistic variation did not elicit MMN amplitudes differences between the left and right hemispheres. Taken together, findings from the present study indicated that the identity information recruited similar neural resources to the phonological information but different neural resources from the lexical information. However, the identity and linguistic information processing did not show a particular hemispheric lateralization pattern at an early pre-attentive speech perception stage. The findings revealed similarities and differences between linguistic and non-linguistic information processing, contributing to a better understanding of speech perception and spoken word recognition.

Oddball paradigm complexity in multi-token auditory evoked potentials

We developed and tested a series of novel and increasingly complex multi-token electrophysiology paradigms for evoking the auditory P3 response. The primary goal was to evaluate the degree to which more complex discrimination tasks and listening environments – which are more likely to engage the types of neural processing used in real-world speech-in-noise situations – could still evoke a robust P3 response. If so, this opens the possibility of such a paradigm making up part of the toolkit for a brain-behavioral approach to improve understanding of speech processing. Fourteen normal-hearing adults were tested using four different auditory paradigms consisting of 5 tokens, 20 tokens, 160 tokens, or 160 tokens with background babble. Stimuli were naturally produced consonant-vowel tokens varying in consonant (/d/, /b/, /g/, /v/, and /ð/; all conditions), vowel (/ɑ/, /u/, /i/, and /ɜr/; 20- and 160-token conditions), and talker (4 female, 4 male; 160-token conditions only). All four conditions evoked robust neural responses, and all peaks had visible differences across conditions. However, the more exogenous auditory evoked potentials (N1 and P2) were primarily affected not by overall complexity but by the presence of background noise specifically, the presence of which was associated with longer latencies and smaller amplitudes. The more endogenous P3 peak, as well as the paradigm behavioral measures, revealed a more graded effect of overall paradigm complexity, rather than the background noise dominating the other factors. Our conclusion was that all four complex auditory paradigms, including the most complex (160 distinct consonant-vowel tokens presented in background babble), are viable means of stimulating N1-P2 and N2b-P3 auditory evoked responses and may therefore be useful in brain-behavioral approaches to understanding speech perception in noise.

Variability in the perception of race through voices and faces

Although perception of race is known to influence speech intelligibility, speech has often been studied without careful attention to this variability. It has been common to study language in an audio-only modality, which fails to capture the multimodal natures of language and person perception. We have developed a set of audiovisual study materials consisting of participants producing individual sentences selected from two different sets of standard materials. The corpus we have created is intentionally diverse with respect to racial self-report. However, there is extraordinary diversity in how persons who share a racial identification may present. The complex and variable relationship between physical features and the construction of social status demands that we reject reliance on self-reports when characterizing the diversity of our sample. In two experiments, participants viewed faces and listened to voices of people who contributed to the audiovisual corpus. The results allow us to explicitly characterize how the participants in the corpus may be racialized by observers, independent of their self-report. We argue that examining cognitive processes, which give rise to racialized and gendered perceptions, is foundational to advancing our understanding of speech perception in realistic environments with high degrees of variability in personal appearance and linguistic behavior.

Effects of noise, native language, age, and speaker gender on intelligibility in a large corpus of read speech

Because speech communication takes place under noisy conditions, investigating the effect of noise on intelligibility is an essential part of understanding speech perception. One challenge is having sufficient numbers of speakers and unique sentences to control for sentence and speaker effects. In an online experiment, we used a corpus of 720 IEEE sentences read by 20 native English speakers (10 male, 10 female) from the Pacific Northwest (WA, OR, ID), which were embedded in three corpus-shaped noise conditions (−2, 0, + 2 dB). Stimuli were presented to undergraduate students at the Universities of Alberta and Washington using a design matrix which ensured that no listener heard a sentence more than once. Participants entered responses into a form which appeared immediately after the stimulus completed playing. Data collection is ongoing, but the current number of listeners is 1269, who responded to 120 items each. We use Levenshtein and Jaro distance measures to compare listener transcription accuracy. We investigate the effects of listener native language, age, and gender. We also investigate effects of the speaker’s gender on transcription accuracy. [Work supported by NIH NIDCD R01 DC006014.]

The evolution of vowel perception in second language learners of Spanish

Given the importance of understanding speech perception in language acquisition research (Best, 1991) and building on the methodology of Garcia de las Bayonas (2004), this remotely conducted study explores the acquisition of correct Spanish vowel perception by English-speaking learners with special attention paid to the role of syllable stress. Participants include Spanish speakers (n = 4), English speakers (n = 7), and beginning Spanish students (n = 9). In a discrimination task, learners performed most accurately with Spanish vowels in stressed syllables (88.6%) with less accuracy discriminating those in unstressed syllables (82.2%). This is likely due to the tendency of American English to reduce many unstressed vowels. In an identification task, participants selected vowel sounds (from English and Spanish inventories) believed to belong in Spanish words. High accuracy was shown identifying Spanish vowels /i/, /o/, and /u/ but all groups struggled to correctly identify /e/ and /a/, likely because these vowels occupy similar acoustic spaces as English vowels /ɛ/ and /æ/. Despite imperfect perception, learners report high confidence in their responses, indicating under-developed vowel inventories in the target language. At the time of presentation, these results will include larger and more diverse participant groups, including students at different stages of experience with the target language.

Weak observer–level correlation and strong stimulus-level correlation between the McGurk effect and audiovisual speech-in-noise: A causal inference explanation

The McGurk effect is a widely used measure of multisensory integration during speech perception. Two observations have raised questions about the validity of the effect as a tool for understanding speech perception. First, there is high variability in perception of the McGurk effect across different stimuli and observers. Second, across observers there is low correlation between McGurk susceptibility and recognition of visual speech paired with auditory speech-in-noise, another common measure of multisensory integration. Using the framework of the causal inference of multisensory speech (CIMS) model, we explored the relationship between the McGurk effect, syllable perception, and sentence perception in seven experiments with a total of 296 different participants. Perceptual reports revealed a relationship between the efficacy of different McGurk stimuli created from the same talker and perception of the auditory component of the McGurk stimuli presented in isolation, both with and without added noise. The CIMS model explained this strong stimulus-level correlation using the principles of noisy sensory encoding followed by optimal cue combination within a common representational space across speech types. Because the McGurk effect (but not speech-in-noise) requires the resolution of conflicting cues between modalities, there is an additional source of individual variability that can explain the weak observer–level correlation between McGurk and noisy speech. Power calculations show that detecting this weak correlation requires studies with many more participants than those conducted to-date. Perception of the McGurk effect and other types of speech can be explained by a common theoretical framework that includes causal inference, suggesting that the McGurk effect is a valid and useful experimental tool.

Recognizing speech in the context of talker variability

The relationship of the acoustic patterns of speech to phonetic categories varies across talkers. A change in talker therefore requires listeners use this relationship which depends on determining the acoustic-phonetic mapping for a new talker. The work from our group has demonstrated that tuning the perceptual system to the acoustic-phonetic system of a new talker is associated with a redirection of attention and a momentary increased load on working memory. These empirical findings support the hypothesis that speech perception is best thought of as an active cognitive process, where the ambiguity of the acoustic signal determines the mobilization of cognitive resources needed to support accurate perception. In this talk, I will (1) discuss the implication of these results for both cognitive and neural models of speech perception and (2) make the argument that extant theories of speech perception must address recognition in the context of acoustic-linguistic pattern variability, such as talker variability, as it is central to the computational problem of understanding speech perception.

Speech communication in older adults: an acoustic and perceptual investigation – ESRC

Speech communication can be difficult for older people, due to the combined effects of age-related hearing loss, which is common over the age of 65, age-related decline in the quality of phonation and speech articulation, and cognitive problems such as poorer short-term memory and processing speed. Past studies of how older individuals perceive and produce speech sounds have tended to consider these abilities independently of each other using controlled materials, such as read words or sentences. These studies tell us little about how older speakers function when using speech for communicative purposes, and how these various factors interact. In our project, we gained a comprehensive account of older people's speech production and perception in situations involving communication with another individual. Older adults with age-related hearing loss and the rarer group of older adults with normal hearing were included in the study, as well as younger adult controls. In Study 1, participants were recorded while they completed a problem-solving task with another person. Communication between the participants was either in good listening conditions, where both speakers could hear each other normally, or in adverse conditions. There, one of the participants had to get their message across to another speaker who has a simulated hearing loss or when both were speaking in a noisy background. These comparisons enabled us to get a sense of the degree to which an older person is able to adapt their speech to overcome difficult listening conditions, a skill which is of paramount importance in speech communication in everyday life. High-quality recordings were made of these interactions and these were used to carry out acoustic analyses of their speech and also analyses of interactional features such as eye gaze and turn-taking. In Study 2, older and younger listeners with normal and impaired hearing were presented some of the materials recorded in Study 1 in good and adverse listening conditions. Intelligibility tests will be run to see what impact age and hearing status have on speech understanding and to see whether the 'clear speech' adaptations made by older speakers to counter the effects of poor communication conditions gave the same benefit to that of younger speakers. This project aims to lead to a better understanding of the effects of ageing on speech communication and of the various contributing factors to potentially degraded speech communication in a population of 'healthy aged' individuals. These benchmarks will be of use for practitioners such as speech and language therapists and audiologists who work on aspects of communication with older people who have health complications. A better understanding of communication difficulties that older individuals experience and of their strategies to overcome these difficulties also assists professionals such as social workers and care professionals who work to improve quality of life for older people, as well as developers of speech technology devices for telemedicine and remote monitoring. Importantly, this research also contributes to our basic understanding of speech perception and production development across the lifespan.

Toward a taxonomic model of attention in effortful listening

In recent years, there has been increasing interest in studying listening effort. Research on listening effort intersects with the development of active theories of speech perception and contributes to the broader endeavor of understanding speech perception within the context of neuroscientific theories of perception, attention, and effort. Due to the multidisciplinary nature of the problem, researchers vary widely in their precise conceptualization of the catch-all term listening effort. Very recent consensus work stresses the relationship between listening effort and the allocation of cognitive resources, providing a conceptual link to current cognitive neuropsychological theories associating effort with the allocation of selective attention. By linking listening effort to attentional effort, we enable the application of a taxonomy of external and internal attention to the characterization of effortful listening. More specifically, we use a vectorial model to decompose the demand causing listening effort into its mutually orthogonal external and internal components and map the relationship between demanded and exerted effort by means of a resource-limiting term that can represent the influence of motivation as well as vigilance and arousal. Due to its quantitative nature and easy graphical interpretation, this model can be applied to a broad range of problems dealing with listening effort. As such, we conclude that the model provides a good starting point for further research on effortful listening within a more differentiated neuropsychological framework.

Challenges in understanding and modeling speech perception

Speech perception performance is one of the most widely used measures in audiology and auditory research. Nevertheless, our understanding of how speech is processed by the auditory system is limited. This is not only due to the subject—the complexity of the speech signal itself and the intricacy of disentangling target speech from interfering sounds—but also to the way it is approached: different aspects of speech perception are studied by largely separate research communities. Also when only early processing of speech is considered, excluding psycholinguistic aspects and effects of memory and learning, at least three relevant research fields can be discriminated: audibility and masking, auditory scene analysis, and attention. Although there is increasing interest in the crosslinks between these fields, approaches are still insufficiently integrated and important questions remain unanswered. Furthermore, modeling efforts are lagging behind and are only partly driven by advances in neurophysiological research. After reviewing progress in the three fields, a conceptual model will be presented that attempts to tie results together. Finally, knowledge gaps and open research issues will be highlighted.

A Phase Transition in Theories of Speech: A Metalogue

ABSTRACTThis article is a metalogue tracing one of Carol Fowler's theoretical contributions to understanding speech perception and, more broadly, cognition. Metalogues, a term coined by Bateson (1972), are conversations in which the content is illustrated by the structure. Similarly in this piece, the structure of the exposition embodies the dynamic progression of ideas. In particular, the structure of the text mirrors the perturbations that Fowler's and others' arguments brought to the 1980s view of speech segments. We start with an early, generally accepted theoretical approach (not endorsed by the authors and admittedly oversimplified), of speech segments as abstract and timeless. The text then moves toward and through a phase transition in which ideas, prose, and font become increasingly noisy and unstable until a very different theoretical outlook, espoused by Carol Fowler and colleagues, is established.

Τhe pattern of relations between cognitive control and Theory of Mind in older adults having vascular risk factors and older adults with Mild Cognitive Impairment: are there any differences?

Τhe pattern of relations between cognitive control and Theory of Mind in older adults having vascular risk factors and older adults with Mild Cognitive Impairment: are there any differences?

A challenge to Articulation Theory: Narrow-band acoustic signals and visual speech cues

Given Kryter’s extensive work on the physiological and subjective effects of noise on the human auditory system it is not surprising that at some point he would turn his attention to the impact of noise on speech intelligibility. In 1962, Kryter published “Methods for the Calculation and Use of the Articulation Index” greatly simplifying the methods originally proposed by Fletcher in the 20s and 30s and French and Steinberg in 1947, ultimately leading to the ANSI 1969 standard. Over the years, the AI and the revised Speech Intelligibility Index (SII) have become the leading metric for predicting speech recognition in noise and in quantifying the audibility provided by amplification. However, the scope of the AI/SII is still limited to communication conditions, which do not include multiple, narrow bands of speech or noise. Similarly, predictions for conditions of auditory-visual speech are only roughly approximated. Data from a variety of studies suggest that rather than ignore these communication conditions they should be studied closely for what they reveal. In this talk, I will show data that expose chinks in the armor of our understanding of speech perception in noise and under conditions where visual speech cues are present.

Using the lens of phonetic experience to resolve phonological forms

This special issue of the Journal contains a selection of papers developed from original presentations at the 2nd ASA Special Workshop on Speech with the theme of Cross-Language Speech Perception and Variations in Linguistics Experience. The papers represent major theoretical and empirical contributions that converge upon the common theme of how our perception of phonological forms is guided and constrained by our experience with the phonetic details of the language(s) we have learned. Several of the papers presented here offer key theoretical advances and lay out novel or newly expanded frameworks that increase our understanding of speech perception as shaped by universal, first language acquisition abilities, general learning mechanisms, and language-specific perceptual tuning. Others offer careful empirical investigations of language learning by simultaneous bilinguals, as well as by later second language learners, and discuss their new findings in light of the theoretical proposals. The work presented here will provide a stimulating and thoughtful impetus toward further progress on the fundamentally significant issue of understanding of how language experience shapes our perception of phonetic details and phonological structure in spoken language.

Improvement of intelligibility of ideal binary-masked noisy speech by adding background noise

When a target-speech/masker mixture is processed with the signal-separation technique, ideal binary mask (IBM), intelligibility of target speech is remarkably improved in both normal-hearing listeners and hearing-impaired listeners. Intelligibility of speech can also be improved by filling in speech gaps with un-modulated broadband noise. This study investigated whether intelligibility of target speech in the IBM-treated target-speech/masker mixture can be further improved by adding a broadband-noise background. The results of this study show that following the IBM manipulation, which remarkably released target speech from speech-spectrum noise, foreign-speech, or native-speech masking (experiment 1), adding a broadband-noise background with the signal-to-noise ratio no less than 4 dB significantly improved intelligibility of target speech when the masker was either noise (experiment 2) or speech (experiment 3). The results suggest that since adding the noise background shallows the areas of silence in the time-frequency domain of the IBM-treated target-speech/masker mixture, the abruption of transient changes in the mixture is smoothed and the perceived continuity of target-speech components becomes enhanced, leading to improved target-speech intelligibility. The findings are useful for advancing computational auditory scene analysis, hearing-aid/cochlear-implant designs, and understanding of speech perception under "cocktail-party" conditions.

P1-N1-P2 Complex and Acoustic Change Complex Elicited by Speech Sounds: Current Research and Applications

Auditory cortical-evoked potentials reflect neural activity related to the detection, perception, discrimination, and cognition of incoming sounds from their timing (latency) and magnitude (amplitude) at the levels of primary auditory cortex and associated areas of the temporal lobe. Researchers have become interested in cortical-evoked potentials elicited by speech stimuli, referred to as speech- evoked potentials (SEPs). SEPs can provide information about a listener’s speech perception abilities at the auditory cortical level, primarily regarding features within speech signals that yield a greater chance of extracting essential cues for human auditory/oral communication. However, to date, relatively little data exist based on naturally produced speech stimuli, which include the listener’s perceptual information, such as specific acoustic/phonetic features, and unmodified spectral/temporal cues produced in natural speech. This tutorial paper reviews current SEP studies and provides feasible applications, focusing on two components of auditory potentials: P1-N1-P2 complex and acoustic change complex (ACC). SEPs offer a powerful objective technique for understanding speech perception of both normal and hearing-impaired listeners, particularly infants and elderly patients who cannot be assessed reliably using typical behavioral speech perception measures; SEPs also provide a means to investigate how acoustical characteristics of various naturally spoken speech stimuli are reflected in the evoked responses for both populations.

Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing

Speech and language are considered uniquely human abilities: animals have communication systems, but they do not match human linguistic skills in terms of recursive structure and combinatorial power. Yet, in evolution, spoken language must have emerged from neural mechanisms at least partially available in animals. In this paper, we will demonstrate how our understanding of speech perception, one important facet of language, has profited from findings and theory in nonhuman primate studies. Chief among these are physiological and anatomical studies showing that primate auditory cortex, across species, shows patterns of hierarchical structure, topographic mapping and streams of functional processing. We will identify roles for different cortical areas in the perceptual processing of speech and review functional imaging work in humans that bears on our understanding of how the brain decodes and monitors speech. A new model connects structures in the temporal, frontal and parietal lobes linking speech perception and production.

Speech intelligibility in background noise with ideal binary time-frequency masking

Ideal binary time-frequency masking is a signal separation technique that retains mixture energy in time-frequency units where local signal-to-noise ratio exceeds a certain threshold and rejects mixture energy in other time-frequency units. Two experiments were designed to assess the effects of ideal binary masking on speech intelligibility of both normal-hearing (NH) and hearing-impaired (HI) listeners in different kinds of background interference. The results from Experiment 1 demonstrate that ideal binary masking leads to substantial reductions in speech-reception threshold for both NH and HI listeners, and the reduction is greater in a cafeteria background than in a speech-shaped noise. Furthermore, listeners with hearing loss benefit more than listeners with normal hearing, particularly for cafeteria noise, and ideal masking nearly equalizes the speech intelligibility performances of NH and HI listeners in noisy backgrounds. The results from Experiment 2 suggest that ideal binary masking in the low-frequency range yields larger intelligibility improvements than in the high-frequency range, especially for listeners with hearing loss. The findings from the two experiments have major implications for understanding speech perception in noise, computational auditory scene analysis, speech enhancement, and hearing aid design.