Single Talker Research Articles

Speaker-Attributed Training for Multi-Speaker Speech Recognition Using Multi-Stage Encoders and Attention-Weighted Speaker Embedding

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).

Partner-Specific Adaptation in Disfluency Processing.

Speakers tend to produce disfluencies when naming unexpected or complex items; in turn, when perceiving disfluency, listeners tend to expect upcoming reference to items that are unexpected or complex to name. In two experiments, we examined if these disfluency-based expectations are routine, or instead, if they adapt to the way the speaker uses disfluency in the current context in a talker-specific manner. Participants listened to instructions to look at objects in contexts with several images, some of which lacked conventional names. We manipulated the co-occurrence of disfluency and reference to novel versus familiar objects in a single talker situation (Experiment 1) and in a multi-talker situation (Experiment 2). In the predictive condition, disfluent expressions referred to novel objects, and fluent expressions referred to familiar objects. In the nonpredictive condition, fluent and disfluent trials referred to either familiar or novel objects. Participants' gaze revealed that listeners more readily predicted familiar images for fluent trials and novel images for disfluent trials in the predictive condition than in the nonpredictive condition. In sum, listeners adapted their expectations about upcoming words based on recent experience with disfluency. Disfluency is not invariably processed, but instead a cue that is flexibly interpreted depending on the local context even in a multi-talker setting.

Gender stereotypes and social perception of vocal confidence is mitigated by salience of socio-indexical cues to gender.

Socio-indexical cues to gender and vocal affect often interact and sometimes lead listeners to make differential judgements of affective intent based on the gender of the speaker. Previous research suggests that rising intonation is a common cue that both women and men produce to communicate lack of confidence, but listeners are more sensitive to this cue when it is produced by women. Some speech perception theories assume that listeners will track conditional statistics of speech and language cues (e.g., frequency of the socio-indexical cues to gender and affect) in their listening and communication environments during speech perception. It is currently less clear if these conditional statistics will impact listener ratings when context varies (e.g., number of talkers). To test this, we presented listeners with vocal utterances from one female and one male-pitched voice (single talker condition) or many female/male-pitched voices (4 female voices; 4 female voices pitch-shifted to a male range) to examine how they impacted perceptions of talker confidence. Results indicated that when one voice was evaluated, listeners defaulted to the gender stereotype that the female voice using rising intonation (a cue to lack of confidence) was less confident than the male-pitched voice (using the same cue). However, in the multi-talker condition, this effect went away and listeners equally rated the confidence of the female and male-pitched voices. Findings support dual process theories of information processing, such that listeners may rely on heuristics when speech perception is devoid of context, but when there are no differentiating qualities across talkers (regardless of gender), listeners may be ideal adapters who focus on only the relevant cues.

Statistical learning at a virtual cocktail party

Statistical learning – the ability to extract distributional regularities from input – is suggested to be key to language acquisition. Yet, evidence for the human capacity for statistical learning comes mainly from studies conducted in carefully controlled settings without auditory distraction. While such conditions permit careful examination of learning, they do not reflect the naturalistic language learning experience, which is replete with auditory distraction – including competing talkers. Here, we examine how statistical language learning proceeds in a virtual cocktail party environment, where the to-be-learned input is presented alongside a competing speech stream with its own distributional regularities. During exposure, participants in the Dual Talker group concurrently heard two novel languages, one produced by a female talker and one by a male talker, with each talker virtually positioned at opposite sides of the listener (left/right) using binaural acoustic manipulations. Selective attention was manipulated by instructing participants to attend to only one of the two talkers. At test, participants were asked to distinguish words from part-words for both the attended and the unattended languages. Results indicated that participants’ accuracy was significantly higher for trials from the attended vs. unattended language. Further, the performance of this Dual Talker group was no different compared to a control group who heard only one language from a single talker (Single Talker group). We thus conclude that statistical learning is modulated by selective attention, being relatively robust against the additional cognitive load provided by competing speech, emphasizing its efficiency in naturalistic language learning situations.

Word Learning in Deaf Adults Who Use Cochlear Implants: The Role of Talker Variability and Attention to the Mouth.

Although cochlear implants (CIs) facilitate spoken language acquisition, many CI listeners experience difficulty learning new words. Studies have shown that highly variable stimulus input and audiovisual cues improve speech perception in CI listeners. However, less is known whether these two factors improve perception in a word learning context. Furthermore, few studies have examined how CI listeners direct their gaze to efficiently capture visual information available on a talker's face. The purpose of this study was two-fold: (1) to examine whether talker variability could improve word learning in CI listeners and (2) to examine how CI listeners direct their gaze while viewing a talker speak. Eighteen adults with CIs and 10 adults with normal hearing (NH) learned eight novel word-object pairs spoken by a single talker or six different talkers (multiple talkers). The word learning task comprised of nonsense words following the phonotactic rules of English. Learning was probed using a novel talker in a two-alternative forced-choice eye gaze task. Learners' eye movements to the mouth and the target object (accuracy) were tracked over time. Both groups performed near ceiling during the test phase, regardless of whether they learned from the same talker or different talkers. However, compared to listeners with NH, CI listeners directed their gaze significantly more to the talker's mouth while learning the words. Unlike NH listeners who can successfully learn words without focusing on the talker's mouth, CI listeners tended to direct their gaze to the talker's mouth, which may facilitate learning. This finding is consistent with the hypothesis that CI listeners use a visual processing strategy that efficiently captures redundant audiovisual speech cues available at the mouth. Due to ceiling effects, however, it is unclear whether talker variability facilitated word learning for adult CI listeners, an issue that should be addressed in future work using more difficult listening conditions.

Cognitive factors in nonnative phonetic learning: Impacts of inhibitory control and working memory on the benefits and costs of talker variability

Talker variability has been reported to facilitate generalization and retention of speech learning, but is also shown to place demands on cognitive resources. Our recent study provided evidence that phonetically-irrelevant acoustic variability in single-talker (ST) speech is sufficient to induce equivalent amounts of learning to the use of multiple-talker (MT) training. This study is a follow-up contrasting MT versus ST training with varying degrees of temporal exaggeration to examine how cognitive measures of individual learners may influence the role of input variability in immediate learning and long-term retention. Native Chinese-speaking adults were trained on the English /i/-/ɪ/ contrast. We assessed the trainees’ working memory and inhibition control before training. The two trained groups showed comparable long-term retention of training effects in terms of word identification performance and more native-like cue weighting in both perception and production regardless of talker variability condition. The results demonstrate the role of phonetically-irrelevant variability in robust speech learning and modulatory functions of nonlinguistic domain-general inhibitory control and working memory, highlighting the necessity to consider the interaction between input characteristics, task difficulty, and individual differences in cognitive abilities in assessing learning outcomes.

Talker adaptation or "talker" adaptation? Musical instrument variability impedes pitch perception.

Listeners show perceptual benefits (faster and/or more accurate responses) when perceiving speech spoken by a single talker versus multiple talkers, known as talker adaptation.While near-exclusively studied in speech and with talkers, some aspects of talker adaptation might reflect domain-general processes. Music, like speech, is a sound class replete with acoustic variation, such as a multitude of pitch and instrument possibilities. Thus, it was hypothesized that perceptual benefits from structure in the acoustic signal (i.e., hearing the same sound source on every trial) are not specific to speech but rather a general auditory response. Forty nonmusician participants completed a simple musical task that mirrored talker adaptation paradigms. Low- or high-pitched notes were presented in single- and mixed-instrument blocks. Reflecting both music research on pitch and timbre interdependence and mirroring traditional "talker" adaptation paradigms, listeners were faster to make their pitch judgments when presented with a single instrument timbre relative to when the timbre was selected from one of four instruments from trial to trial. A second experiment ruled out the possibility that participants were responding faster to the specific instrument chosen as the single-instrument timbre. Consistent with general theoretical approaches to perception, perceptual benefits from signal structure are not limited to speech.

Asymmetries in vowel perception align with focalization patterns, not peripherality: Acoustic evidence from Canadian French /e/ and /Ø/

The question of whether asymmetries in vowel perception stem from peripherality (location in F1/F2 vowel space) or focalization patterns (formant convergence) was examined. Prior studies with Danish and Dutch listeners reported asymmetric perception of /e-Ø/ such that discriminating a change from /e/ to /Ø/ was easier compared to the reverse direction (/Ø/ to /e/). Acoustic analyses of test stimuli suggest that focalization patterns predict this asymmetry whereas a peripherality-based prediction fails. Although this interpretation is compelling, it is based on analyses of several vowel tokens produced by a single talker. To confirm the reliability of these patterns we obtained acoustic and electromagnetic articulography recordings of Canadian French /e/ and /Ø/ productions from a broader sample of talkers (n = 20): 400 vowels/10 productions per vowel. Acoustic analyses revealed a robust pattern for all talkers: F1 and F2 (also F1 and F3) were spectrally closer for /Ø/ compared to /e/, and /e/ was also more peripheral than /Ø/ in F1/F2 space. These data align with the earlier (single-talker) analyses and confirm that /Ø/ is a more focal vowel than /e/. We will also discuss articulatory analyses of these French vowel productions that focus on how vocal-tract constrictions relate to focalization and peripherality.

Influences of talker variability, reading ability, and language ability on word learning

Studies show that talker variability is beneficial for learning second language contrasts, though less is known about variability in processing speech in a speaker’s native language. Talker variability benefits have also been found for word learning studies with children, with better performance for children in the high variability condition than those in the low variability condition. Furthermore, it has been shown that individuals with a range of reading and language abilities perform differently on tasks that tap into phonological processes. Therefore, the current study examines how individual differences in reading and language abilities influence word learning and the extent to which talker variability improves performance. One hundred and forty listeners were randomly assigned to three conditions: (1) single talker, (2) multiple talker (randomized), and (3) multiple talker (blocked). Our results showed that individuals with poorer reading or language skills performed worse than individuals with higher reading and language skills. Surprisingly, variability was not beneficial and instead, listeners in the single talker condition performed better than those in the multiple talker conditions. These results suggest that talker variability benefits in word learning may be dependent on other aspects such as the amount of difference between talkers or the task.

Speech Understanding in Complex Environments by School-Age Children with Mild Bilateral or Unilateral Hearing Loss.

Numerous studies have shown that children with mild bilateral (MBHL) or unilateral hearing loss (UHL) experience speech perception difficulties in poor acoustics. Much of the research in this area has been conducted via laboratory studies using speech-recognition tasks with a single talker and presentation via earphones and/or from a loudspeaker located directly in front of the listener. Real-world speech understanding is more complex, however, and these children may need to exert greater effort than their peers with normal hearing to understand speech, potentially impacting progress in a number of developmental areas. This article discusses issues and research relative to speech understanding in complex environments for children with MBHL or UHL and implications for real-world listening and understanding.

Performance with a new bone conduction implant audio processor in patients with single-sided deafness

PurposeThe SAMBA 2 BB audio processor for the BONEBRIDGE bone conduction implant features a new automatic listening environment detection to focus on target speech and to reduce interfering speech and background noises. The aim of this study was to evaluate the audiological benefit of the SAMBA 2 BB (AP2) and to compare it with its predecessor SAMBA BB (AP1).MethodsProspective within-subject comparison study. We compared the aided sound field hearing thresholds, speech understanding in quiet (Freiburg monosyllables), and speech understanding in noise (Oldenburg sentence test) with the AP1 and AP2. Each audio processor was worn for 2 weeks before assessment and seven users with single-sided sensorineural deafness (SSD) participated in the study. For speech understanding in noise, two complex noise scenarios with multiple noise sources including single talker interfering speech were used. The first scenario included speech presented from the front (S0NMIX), while in the second scenario speech was presented from the side of the implanted ear (SIPSINMIX). In addition, subjective evaluation using the SSQ12, APSQ, and the BBSS questionnaires was performed.ResultsWe found improved speech understanding in quiet with the AP2 compared to the AP1 aided condition (on average + 17%, p = 0.007). In both noise scenarios, the AP2 lead to improved speech reception thresholds by 1.2 dB (S0NMIX, p = 0.032) and 2.1 dB (SIPSINMIX, p = 0.048) compared to the AP1. The questionnaires revealed no statistically significant differences, except an improved APSQ usability score with the AP2.ConclusionClinicians can expect that patients with SSD will benefit from the SAMBA 2 BB by improved speech understanding in both quiet and in complex noise scenarios, when compared to the older SAMBA BB.

Group conversation enhancement using distributed microphone arrays with adaptive binauralization

Hearing aids and other listening devices perform poorly in noisy, reverberant venues like restaurants and conference centers with numerous active sound sources. Microphone arrays can use array processing techniques like beamforming to isolate talkers from a specific region in the room while attenuating undesired sound sources. However, beamforming often removes spatial cues and is typically restricted to isolating a single talker at a time. Previous work has shown the effectiveness of remote microphones worn by talkers and adapting the signal at the earpiece to improve the intelligibility of group conversations. Due to the increase in hybrid meetings and classrooms, many spaces are equipped with high throughput, low latency devices including large microphone arrays. In this work, we present a system that aggregates information collected by microphone arrays distributed in a room to enhance the intelligibility of talkers in a group conversation. The beamformed signal from the microphone arrays is adapted to match the magnitude and phase of the earpiece microphones. The filters are continuously updated in order to track motion of both the listeners and talkers.

Perceptual learning of multiple talkers: Determinants, characteristics, and limitations.

Research suggests that listeners simultaneously update talker-specific generative models to reflect structured phonetic variation. Because past investigations exposed listeners to talkers of different genders, it is unknown whether adaptation is talker specific or rather linked to a broader sociophonetic class. Here, we test determinants of listeners' ability to update and apply talker-specific models for speech perception. In six experiments (n = 480), listeners were first exposed to the speech of two talkers who produced ambiguous fricative energy. The talkers' speech was interleaved during exposure, and lexical context differentially biased interpretation of the ambiguity as either /s/ or /ʃ/ for each talker. At test, listeners categorized tokens from ashi-asi continua, one for each talker. Across conditions and experiments, we manipulated exposure quantity, talker gender, blocked versus interleaved talker structure at test, and the degree to which fricative acoustics differed between talkers. When test was blocked by talker, learning was observed for different but not same gender talkers. When talkers were interleaved at test, learning was observed for both different and same gender talkers, which was attenuated when fricative acoustics were constant across talkers. There was no strong evidence to suggest that adaptation to multiple talkers required increased quantity of exposure beyond that required to adapt to a single talker. These results suggest that perceptual learning for speech is achieved via a mechanism that represents a context-dependent, cumulative integration of experience with speech input and identity critical constraints on listeners' ability to dynamically apply multiple generative models in mixed talker listening environments.

Cortical tracking of voice pitch in the presence of multiple speakers depends on selective attention.

Voice pitch carries linguistic and non-linguistic information. Previous studies have described cortical tracking of voice pitch in clean speech, with responses reflecting both pitch strength and pitch value. However, pitch is also a powerful cue for auditory stream segregation, especially when competing streams have pitch differing in fundamental frequency, as is the case when multiple speakers talk simultaneously. We therefore investigated how cortical speech pitch tracking is affected in the presence of a second, task-irrelevant speaker. We analyzed human magnetoencephalography (MEG) responses to continuous narrative speech, presented either as a single talker in a quiet background or as a two-talker mixture of a male and a female speaker. In clean speech, voice pitch was associated with a right-dominant response, peaking at a latency of around 100 ms, consistent with previous electroencephalography and electrocorticography results. The response tracked both the presence of pitch and the relative value of the speaker’s fundamental frequency. In the two-talker mixture, the pitch of the attended speaker was tracked bilaterally, regardless of whether or not there was simultaneously present pitch in the speech of the irrelevant speaker. Pitch tracking for the irrelevant speaker was reduced: only the right hemisphere still significantly tracked pitch of the unattended speaker, and only during intervals in which no pitch was present in the attended talker’s speech. Taken together, these results suggest that pitch-based segregation of multiple speakers, at least as measured by macroscopic cortical tracking, is not entirely automatic but strongly dependent on selective attention.

Spelling provides a precise (but sometimes misplaced) phonological target. Orthography and acoustic variability in second language word learning

L1 French participants learned novel L2 English words over two days of learning sessions, with half of the words presented with their orthographic forms (Audio-Ortho) and half without (Audio only). One group heard the words pronounced by a single talker, while another group heard them pronounced by multiple talkers. On the third day, they completed a variety of tasks to evaluate their learning. Our results show a robust influence of orthography, with faster response times in both production (Picture naming) and recognition (Picture mapping) tasks for words learned in the Audio-Ortho condition. Moreover, formant analyses of the Picture naming responses show that orthographic input pulls pronunciations of English novel words towards a non-native (French) phonological target. Words learned with their orthographic forms were pronounced more precisely (with smaller Dispersion Scores), but were misplaced in the vowel space (as reflected by smaller Euclidian distances with respect to French vowels). Finally, we found only limited evidence of an effect of talker-based acoustic variability: novel words learned with multiple talkers showed faster responses times in the Picture naming task, but only in the Audio-only condition, which suggests that orthographic information may have overwhelmed any advantage of talker-based acoustic variability.

Talker and sentence variability attenuates speaking rate normalization in speech perception

Speech perception is shaped by acoustic context effects, where acoustic properties of earlier (context) sounds influence categorization of later (target) sounds. These context effects are sensitive to talker characteristics. When context sentences were spoken by a different talker on each trial, spectral contrast effect (SCE) magnitudes affecting categorization of the target vowel were smaller than when sentences were spoken by a single talker [Assgari and Stilp, J. Ascoust. Soc. Am. (2015)]. Here, perceptual consequences of talker variability were examined in Temporal Contrast Effects (TCEs; also termed rate normalization). Each trial presented a context sentence (spoken at a fast or slow rate) followed by a target word varying from “deer” to “tier.” Context sentences were the same stimuli from Assgari and Stilp (2015): one talker speaking the same sentence on each trial, one talker speaking a different sentence on each trial, and 200 talkers each speaking a different sentence on each trial. TCE magnitudes significantly decreased between One Talker/One Sentence and One Talker/200 Sentences, and again between One Talker/200 Sentences and 200 Talkers/200 Sentences. Conversely, in Assgari and Stilp, talker variability diminished SCEs but sentence variability for a single talker did not. These results suggest that variability in different acoustic domains restricts speech perception.

The use of piecewise linear regression to explore development of children's listening-in-noise ability

To explore the development of speech-in-noise comprehension, we tested children 15–48 months in a preferential-looking paradigm; children saw two images and were told which to attend to (“Find the keys!”). The target voice was masked either by a 9-voice blend (primarily energetic masking) or a single talker (providing informational masking; IM). We used piecewise linear-mixed-effects modeling to identify phases of developmental growth and their corresponding changepoints. Based on 51 participants to date, performance in multitalker babble was best described by a linear model with gradual improvement over time, but the 1-voice-masker condition was well-fit by a combination of three phases: a period of rapid growth until 25.3 months, followed by a period of relative stagnation until 43.4 months, then a second growth period. Bonino et al. (2021) argued that substantial development in IM ability occurs prior to 30 months, which corresponds well with our first phase. The second growth period may represent an increase in the ability to integrate “acoustic glimpses”; prior studies found that 5-year-olds, but not 30-month-olds, can restore interrupted speech, and our elbow at 43 months fits with this timeline. This demonstrates the usefulness of piecewise linear-mixed-effects as an approach to studying development.

Talker adaptation or “talker” adaptation? Musical instrument variability impedes pitch perception

When perceiving speech spoken by a single talker versus multiple talkers, listeners show perceptual benefits such as increased accuracy and/or decreased response time. There are several theoretical explanations for this talker adaptation phenomenon; one way to distinguish among these is to test whether adapting to stimulus structure is speech-specific or general to auditory perception. Music, like speech, is a sound class replete with acoustic variation. Here, participants completed a musical task that mirrored talker adaptation paradigms. On each trial, participants heard a tone and reported whether it was the lower (D4, 294 Hz) or higher pitched tone (F#4, 370 Hz). Tones were presented in a single- or mixed-instrument block. We predicted that perceptual benefits from structure in the acoustic signal are not specific to speech but are a general auditory response. Accordingly, we hypothesized participants would respond faster in the single-instrument block, consistent with speech studies that used a similar paradigm. Pitch judgments were faster (and more accurate) in the single instrument block, parallel to results from talker adaptation studies. In agreement with general theoretical approaches to auditory perception, perceptual benefits from signal structure are not limited to speech.

Clearly, fame isn’t everything: Talker familiarity does not mitigate the perceptual consequences of talker variability

Familiarity with a talker’s voice provides numerous benefits to speech perception, including faster response times and improved intelligibility both in quiet and amidst competing talkers. However, it is unclear whether familiarity provides resilience against talker variability, or the increased processing costs attributed to hearing speech from multiple different talkers compared to a single talker. Here, listeners completed a speeded word recognition task where stimuli were the words “do” and “to” excised from political speeches, presented in either single- or multiple-talker blocks. Talkers were either famous (the last five U.S. Presidents) or non-famous (other male politicians of similar ages). No information about the talkers was provided. As predicted, listeners categorized words more quickly when spoken by famous talkers. However, talker familiarity did not provide any resiliency to talker variability; reaction times increased (from single- to multiple-talker blocks) by greater amounts for famous talkers than non-famous talkers. In a post-task questionnaire, participants recognized famous talkers better than non-famous talkers from the “do”/”to” target words, a full sentence, or when shown their names and asked if they knew who the talker was. Results were not correlated with self-rated political interest. Thus, familiarity might not alleviate the perceptual consequences of talker variability.

Listen Carefully protocol: an exploratory case–control study of the association between listening effort and cognitive function

IntroductionA growing body of evidence suggests that hearing loss is a significant and potentially modifiable risk factor for cognitive impairment. Although the mechanisms underlying the associations between cognitive decline and...