Target Talker Research Articles

Acoustic scene complexity affects motion behavior during speech perception in audio-visual multi-talker virtual environments

In real-world listening situations, individuals typically utilize head and eye movements to receive and enhance sensory information while exploring acoustic scenes. However, the specific patterns of such movements have not yet been fully characterized. Here, we studied how movement behavior is influenced by scene complexity, varied in terms of reverberation and the number of concurrent talkers. Thirteen normal-hearing participants engaged in a speech comprehension and localization task, requiring them to indicate the spatial location of a spoken story in the presence of other stories in virtual audio-visual scenes. We observed delayed initial head movements when more simultaneous talkers were present in the scene. Both reverberation and a higher number of talkers extended the search period, increased the number of fixated source locations, and resulted in more gaze jumps. The period preceding the participants’ responses was prolonged when more concurrent talkers were present, and listeners continued to move their eyes in the proximity of the target talker. In scenes with more reverberation, the final head position when making the decision was farther away from the target. These findings demonstrate that the complexity of the acoustic scene influences listener behavior during speech comprehension and localization in audio-visual scenes.

Band importance for speech-in-speech recognition in the presence of extended high-frequency cues.

Band importance functions for speech-in-noise recognition, typically determined in the presence of steady background noise, indicate a negligible role for extended high frequencies (EHFs; 8-20 kHz). However, recent findings indicate that EHF cues support speech recognition in multi-talker environments, particularly when the masker has reduced EHF levels relative to the target. This scenario can occur in natural auditory scenes when the target talker is facing the listener, but the maskers are not. In this study, we measured the importance of five bands from 40 to 20 000 Hz for speech-in-speech recognition by notch-filtering the bands individually. Stimuli consisted of a female target talker recorded from 0° and a spatially co-located two-talker female masker recorded either from 0° or 56.25°, simulating a masker either facing the listener or facing away, respectively. Results indicated peak band importance in the 0.4-1.3 kHz band and a negligible effect of removing the EHF band in the facing-masker condition. However, in the non-facing condition, the peak was broader and EHF importance was higher and comparable to that of the 3.3-8.3 kHz band in the facing-masker condition. These findings suggest that EHFs contain important cues for speech recognition in listening conditions with mismatched talker head orientations.

Neural tracking of speech acoustics in noise is coupled with lexical predictability as estimated by large language models.

Adults heard recordings of two spatially separated speakers reading newspaper and magazine articles. They were asked to listen to one of them and ignore the other, and EEG was recorded to assess their neural processing. Machine learning extracted neural sources that tracked the target and distractor speakers at three levels: the acoustic envelope of speech (delta- and theta-band modulations), lexical frequency for individual words, and the contextual predictability of individual words estimated by GPT-4 and earlier lexical models. To provide a broader view of speech perception, half of the subjects completed a simultaneous visual task, and the listeners included both native and non-native English speakers. Distinct neural components were extracted for these levels of auditory and lexical processing, demonstrating that native English speakers had greater target-distractor separation compared to non-native English speakers on most measures, and that lexical processing was reduced by the visual task. Moreover, there was a novel interaction of lexical predictability and frequency with auditory processing; acoustic tracking was stronger for lexically harder words, suggesting that people listened harder to the acoustics when needed for lexical selection. This demonstrates that speech perception is not simply a feedforward process from acoustic processing to the lexicon. Rather, the adaptable context-sensitive processing long known to occur at a lexical level has broader consequences for perception, coupling with the acoustic tracking of individual speakers in noise.Significance Statement In challenging listening conditions, people use focused attention to help understand individual talkers and ignore others, which changes their neural processing for speech at auditory through lexical levels. However, lexical processing for natural materials (e.g., conversations, audiobooks) has been difficult to measure, because of limitations of tools to estimate the predictability of individual words in longer discourses. The present investigation uses a contemporary large language model, GPT-4, to estimate word predictability, and demonstrates that listeners make online adaptations to their auditory neural processing in accord with these predictions; neural activity more closely tracks the acoustics of the target talker when words are less predictable from the context.

The role of long-term target and masker talker familiarity in children's speech-in-speech recognition.

This study investigated the influence of long-term talker familiarity on speech-in-speech recognition in school-age children, with a specific emphasis on the role of familiarity with the mother's voice as either the target or masker speech. Open-set sentence recognition was measured adaptively in a two-talker masker. Target and masker sentences were recorded by the adult mothers of the child participants. Each child heard sentences spoken by three adult female voices during testing; their own mother's voice (familiar voice) and two unfamiliar adult female voices. Twenty-four school age children (8-13 years) with normal hearing. When the target speech was spoken by a familiar talker (the mother), speech recognition was significantly better compared to when the target was unfamiliar. When the masker was spoken by the familiar talker, there was no difference in performance relative to the unfamiliar masker condition. Across all conditions, younger children required a more favorable signal-to-noise ratio than older children. Implicit long-term familiarity with a talker consistently improves children's speech-in-speech recognition across the age range tested, specifically when the target talker is familiar. However, performance remains unaffected by masker talker familiarity. Additionally, while target familiarity is advantageous, it does not entirely eliminate children's increased susceptibility to competing speech.

Long- and short-term implicit talker familiarity: Effects on children's word and sentence recognition

While previous research has highlighted the positive effects of talker familiarity on children’s word recognition in quiet, its role in challenging listening conditions remains poorly understood for this age group. Additionally, the amount of familiarity required to yield benefits remains unclear. The current study explored implicitly acquired short and long-term talker familiarity effects in school-age children (8–12 years), utilizing speech or noise maskers. Experiment 1 assessed open-set sentence recognition in a two-talker-female masker, with the child's mother's voice serving as the target or masker speech. Experiment 2 measured closed-set word recognition in a noise masker, both before and after a 5-day exposure to a female talker. Results from Experiment 1 indicate that children benefit from familiarity when their mother is the target talker, but not when she is the masking speech. In Experiment 2, children exhibited familiarity effects after only a 5-day exposure period to a previously unfamiliar voice, demonstrating a significant impact of implicit short-term familiarity. Working memory and selective attention did not predict performance or benefit. Together, these findings underscore the importance of talker familiarity in children's speech recognition in challenging listening situations. The implications of these findings for understanding the mechanisms underlying familiarity effects will be discussed.

Role of spatial positioning strategy for speech recognition in a simulated environment: Effects of age and hearing loss

In realistic listening environments, the physical position of the listener and target talker, relative to competing talkers, affects the signal-to-noise ratio and availability of spatial cues for release from masking. While the importance of signal-to-noise ratio and spatial cues for speech recognition has been demonstrated in the laboratory, a listener’s ability to leverage these acoustic cues by manipulating their position in an environment has received less attention. This study evaluated how changes in listener position within a simulated environment effect speech recognition for four listener groups that differed in age (younger or older) and hearing status (normal or impaired hearing). The simulated environment consisted of 8–10 speech maskers, individually rendered under headphones for specific locations within the space, with target and listener positions that varied across conditions. The intensity of the target varied throughout testing to estimate the psychometric function for keyword recognition of each listener in each position. Subjective reports of preferred listening position were collected before and after testing to determine the extent to which these listener groups strategically position themselves to maximize their speech recognition in noise. Results will be discussed in terms of the potential gains in speech recognition associated with optimal positioning strategies.

Factors influencing the minimum audible change in head orientation of a talker

Humans can detect changes in the head orientation (a.k.a. facing angle) of a talker using only auditory cues. Head orientation cues are beneficial for determining whether one is the intended recipient of an utterance and for segregating a target talker from background talkers. Because human talkers are directional sound sources, binaural cues and monoaural spectral cues can both contribute to talker head orientation perception. In this study, we assessed listeners’ sensitivity to talker head orientation changes using only monaural cues. We tested several factors that could influence the minimum audible change in head orientation: talker and gender (two male, two female), stimulus bandwidth (full-band versus low-pass filtered at 8 or 10 kHz), transducer (loudspeaker versus headphone), stimulus uncertainty (interleaved versus blocked presentation of four talkers), and vocal production mode (speech versus singing). Bandlimiting at 8 or 10 kHz led to worse performance, as did increasing stimulus uncertainty. The effect of transducer was very limited. Performance with speech was better than that with singing. The effect of talker was large and consistent, suggesting individual variability in speech directivity patterns may affect head orientation cues. [Work supported by NIH grant R01DC019745.]

A mechanistic investigation of processing costs associated with cross-talker acoustic-phonetic variability

The acoustic realization of speech sounds may differ substantially across talkers, such that (for example) a sound interpreted as “s” in one voice might be considered a “sh” in another. In this talk, I will review select neurobiological and computational data illustrating how listeners cope with this variability. I will also review behavioral data showing how talker variability can hinder processing, with slower and/or less accurate processing when the listening environment contains multiple voices compared to one voice. I argue that at least three mechanisms underlie these multi-talker processing costs. First, multi-talker processing costs may arise because talker changes disrupt a listener’s ability to attend to the target talker. Second, I make the novel claim that talker changes might impose processing penalties by inducing uncertainty about whether the extant acoustic-to-phonetic mapping is appropriate, motivated by recent data that when multi-talker processing costs are attenuated by a preceding carrier phrase, this manifests as slowed responses to single-talker speech rather than improved processing of mixed-talker speech. Finally, talker variability costs may result from making on-the-fly adjustments to the mapping between acoustics and phonetic categories. Overall, this talk highlights the ways in which listeners contend with and are hindered by talker variability.

Children's use of spatial and visual cues for release from perceptual masking.

This study examined the role of visual speech in providing release from perceptual masking in children by comparing visual speech benefit across conditions with and without a spatial separation cue. Auditory-only and audiovisual speech recognition thresholds in a two-talker speech masker were obtained from 21 children with typical hearing (7-9 years of age) using a color-number identification task. The target was presented from a loudspeaker at 0° azimuth. Masker source location varied across conditions. In the spatially collocated condition, the masker was also presented from the loudspeaker at 0° azimuth. In the spatially separated condition, the masker was presented from the loudspeaker at 0° azimuth and a loudspeaker at -90° azimuth, with the signal from the -90° loudspeaker leading the signal from the 0° loudspeaker by 4 ms. The visual stimulus (static image or video of the target talker) was presented at 0° azimuth. Children achieved better thresholds when the spatial cue was provided and when the visual cue was provided. Visual and spatial cue benefit did not differ significantly depending on the presence of the other cue. Additional studies are needed to characterize how children's preferential use of visual and spatial cues varies depending on the strength of each cue.

The Right-Ear Advantage in Static and Dynamic Cocktail-Party Situations.

When presenting two competing speech stimuli, one to each ear, a right-ear advantage (REA) can often be observed, reflected in better speech recognition compared to the left ear. Considering the left-hemispheric dominance for language, the REA has been explained by superior contralateral pathways (structural models) and language-induced shifts of attention to the right (attentional models). There is some evidence that the REA becomes more pronounced, as cognitive load increases. Hence, it is interesting to investigate the REA in static (constant target talker) and dynamic (target changing pseudo-randomly) cocktail-party situations, as the latter is associated with a higher cognitive load than the former. Furthermore, previous research suggests an increasing REA, when listening becomes more perceptually challenging. The present study examined the REA by using virtual acoustics to simulate static and dynamic cocktail-party situations, with three spatially separated talkers uttering concurrent matrix sentences. Sentences were presented at low sound pressure levels or processed with a noise vocoder to increase perceptual load. Sixteen young normal-hearing adults participated in the study. The REA was assessed by means of word recognition scores and a detailed error analysis. Word recognition revealed a greater REA for the dynamic than for the static situations, compatible with the view that an increase in cognitive load results in a heightened REA. Also, the REA depended on the type of perceptual load, as indicated by a higher REA associated with vocoded compared to low-level stimuli. The results of the error analysis support both structural and attentional models of the REA.

Effects of presentation level on speech-on-speech masking by voice-gender difference and spatial separation between talkers.

Many previous studies have reported that speech segregation performance in multi-talker environments can be enhanced by two major acoustic cues: (1) voice-characteristic differences between talkers; (2) spatial separation between talkers. Here, the improvement they can provide for speech segregation is referred to as "release from masking." The goal of this study was to investigate how masking release performance with two cues is affected by various target presentation levels. Sixteen normal-hearing listeners participated in the speech recognition in noise experiment. Speech-on-speech masking performance was measured as the threshold target-to-masker ratio needed to understand a target talker in the presence of either same- or different-gender masker talkers to manipulate the voice-gender difference cue. These target-masker gender combinations were tested with five spatial configurations (maskers co-located or 15°, 30°, 45°, and 60° symmetrically spatially separated from the target) to manipulate the spatial separation cue. In addition, those conditions were repeated at three target presentation levels (30, 40, and 50 dB sensation levels). Results revealed that the amount of masking release by either voice-gender difference or spatial separation cues was significantly affected by the target level, especially at the small target-masker spatial separation (±15°). Further, the results showed that the intersection points between two masking release types (equal perceptual weighting) could be varied by the target levels. These findings suggest that the perceptual weighting of masking release from two cues is non-linearly related to the target levels. The target presentation level could be one major factor associated with masking release performance in normal-hearing listeners.

Simultaneous relative cue reliance in speech-on-speech masking.

Modern hearing research has identified the ability of listeners to segregate simultaneous speech streams with a reliance on three major voice cues, fundamental frequency, level, and location. Few of these studies evaluated reliance for these cues presented simultaneously as occurs in nature, and fewer still considered the listeners' relative reliance on these cues owing to the cues' different units of measure. In the present study trial-by-trial analyses were used to isolate the listener's simultaneous reliance on the three voice cues, with the behavior of an ideal observer [Green and Swets (1966). (Wiley, New York), pp.151-178] serving as a comparison standard for evaluating relative reliance. Listeners heard on each trial a pair of randomly selected, simultaneous recordings of naturally spoken sentences. One of the recordings was always from the same talker, a distracter, and the other, with equal probability, was from one of two target talkers differing in the three voice cues. The listener's task was to identify the target talker. Among 33 clinically normal-hearing adults only one relied predominantly on voice level, the remaining were split between voice fundamental frequency and/or location. The results are discussed regarding their implications for the common practice in studies of using target-distracter level as a dependent measure of speech-on-speech masking.

An Ideal-observer standard for evaluating relative cue reliance on multi-talker speech segregation

Modern hearing research identifies the ability of listeners to segregate simultaneous speech streams with a reliance on three major voice cues, fundamental frequency, level, and location. Few of these studies, however, present cues simultaneously as in natural listening, and fewer still consider the relative reliance listeners placed on these cues owing to the cues’ different units of measure. In the present study, trial-by-trial analyses were used to isolate listener reliance on the three voice cues presented simultaneously, with the behavior of an ideal observer (Green and Swets, 1966, pp. 151–178) serving as standard for determining their relative reliance. Listeners heard on each trial a pair of randomly selected, simultaneously recordings of naturally spoken sentences. One of the recordings was always from the same talker, a distracter, the other, with equal probability, was from one of two target talkers differing in the three voice cues. The listener’s task was to identify the target talker. Among 33 clinically normal-hearing adults only one relied predominantly on voice level, the remaining were split between voice fundamental frequency and location. The results are discussed regarding their implications for the common practice of using target-distracter level as a dependent measure of multi-talker speech segregation.

The effect of rhythm on selective listening in multiple-source environments for young and older adults

Understanding continuous speech with competing background sounds is challenging, particularly for older adults. One stimulus property that may aid listeners understanding of to-be-attended (target) material is temporal regularity (rhythm). In the context of speech-in-noise understanding, McAuley and colleagues recently showed a target rhythm effect whereby recognition of target speech was better when natural speech rhythm of a target talker was intact than when it was temporally altered. The current study replicates the target rhythm effect using a synthetic vowel sequence paradigm in young adults (Experiment 1) and then uses this paradigm to investigate potential age-related changes in the effect of rhythm on recognition (Experiment 2). Listeners identified the last three vowels of temporally regular (isochronous) and irregular (anisochronous) synthetic vowel sequences in quiet and with a competing background sequence of vowel-like harmonic tone complexes presented at various tempos. The results replicated the target rhythm effect whereby temporal regularity in the vowel sequences improved identification accuracy of young listeners compared to irregular vowel sequences. The magnitude of the effect was not found to be influenced by background tempo, but faster background tempos led to greater vowel identification accuracy independent of regularity. Older listeners also demonstrated a target rhythm effect but received less benefit from the temporal regularity of the target sequences than did young listeners. This study highlights the importance of rhythm for understanding age-related differences in selective listening in complex environments and provides a novel paradigm for investigating effects of rhythm on perception.

Band importance for speech-in-speech recognition in the presence of extended high-frequency cues

Extended high-frequency (EHF; 8–20 kHz) cues support speech recognition in noisy backgrounds, particularly when the masker has reduced EHF levels relative to the target. This scenario can occur in natural auditory scenes when the target talker is facing the listener, but the masker talkers are not. The EHF benefit stands in contrast to past studies that have focused on lower frequencies and presumed that EHFs play no role in speech intelligibility. Although EHF cues improve speech recognition, it is unclear how the magnitude of benefit compares to that of other portions of the speech spectrum. In this ongoing study, we measure band importance functions (BIFs) for a female target and two-talker masker by notch filtering individual contiguous bands from 40 to 20 000 Hz. With the target facing the listener, two masking conditions were tested: (1) masker facing the listener; (2) masker facing 56.25°. Preliminary data indicate an interaction between the filtered band and masker head orientation. For the facing condition, the BIF shows a peak between 0.4 and 3 kHz and drops sharply at higher frequencies, resembling previous data. When the masker faces away, however, the benefit of EHFs increases relative to the lower bands, somewhat flattening the BIF. [Work supported by NIH grant R01-DC019745].

Relating spatial release from masking to lateralization using interaural differences

Spatial acoustic cues such as interaural differences in time (ITD) and level (ILD) contribute to the benefit of reduced interference (masking) that occurs when competing sounds (maskers) differ in spatial location from the target talker. This benefit (spatial release from masking or SRM) is well studied but the contributions of ITD and ILD to SRM during speech-on-speech masking remain poorly understood. One hypothesis regarding mixed results in the literature is that SRM primarily reflects differences in overall perceived spatial location rather than independent contributions of ITD and ILD. We evaluated this hypothesis by relating SRM directly to the perceived spatial locations of speech stimuli that were processed in five conditions: ITD only, ILD only, consistent (ITD + ILD), and opposing (ITD – ILD or ILD – ITD). We measured lateralization for speech stimuli in each condition. Lateralization was consistently strongest for consistent ITD + ILD cues, but weaker and more variable across other conditions. Finally, we evaluate a model relating individual differences in SRM to ITD/ILD-based lateralization of target and masker sounds. [Work supported by NIH R01-DC016643]

Localization and externalization of speech through hearing aids with adaptive features

Hearing aids with advanced digital signal processing features that analyze acoustic scenes can amplify wanted sounds (e.g., speech) while utilizing complex noise reduction features to attenuate unwanted sounds (e.g., noise). These adaptive features can be programmed to be more or less aggressive in order to prioritize listening comfort or speech detail. Importantly, these adaptive features may alter the interaural cues of relevant target talkers located in front of the listener, causing unwanted effects on spatial perception. Here, we explored these effects. Spondees were presented from 11 different loudspeakers (located between ±75° azimuth separated by 15°) in an anechoic chamber. Binaural recordings were obtained from the ear canals of a Bruel & Kjaer Head and Torso Simulator, which was fitted with Oticon More hearing aids. The recordings were then presented over headphones to listeners, who indicated the perceived direction and degree of externalization of the spondees. Several conditions were tested, which differed in the acoustics of the room (anechoic and simulated reverberation), the background (quiet and noise), and the hearing-aid coupling (unaided, open-fit, and occluded). Behavioral data will be compared to measured interaural cues in the corresponding binaural recordings.

Speech recognition in the presence of speech maskers in children

Listening to a target talker in the presence of other talkers follows a protracted developmental period. Such listening skills are critical for speech-language development and have clinical value. The purpose of the present study was to examine speech-in-speech recognition ability using digits in children. The major advantages of using digits are that it overcomes the biggest challenge of test administration, and digits are among the few first words children learn. Data were collected from 32 normal-hearing children (4–12 years) using full bandwidth speech materials. The test had four conditions like the Listening in Spatial Noise test. (1) Low cue: The target (male talker) and two male maskers were presented from the front. (2) Talker advantage: The target and two female maskers were presented from the front. (3) Spatial advantage: The target was presented from the front, and two male maskers were presented, one each from ±90°. (4) The target was presented from the front, and two female maskers were presented, one from ±90°. Results demonstrate developmental effects, good test-retest reliability, and feasibility of the computerized version of the test. Furthermore, results will be discussed in the context of various conditions and potential clinical applications.

Audio-visual scene analysis in conditions with head- and eye-steered beamformers in virtual reality

In crowded social settings, listeners often face the challenge of following a conversation in the presence of other conversations. Several factors influence the difficulty of this task, including the number of talkers, the amount of reverberation, and the hearing status. Beamformers in hearing aids have the potential to mitigate these factors by improving the signal-to-noise ratio, but their effectiveness in real-world settings has not yet been clearly demonstrated. Here, we used virtual reality to investigate the effect of head- and eye-steered beamformers on the ability of participants to analyze complex audio-visual scenes. The participants’ task was to find and locate an ongoing story in a mixture of other stories in scenes differing in terms of the number of concurrent talkers and the amount of reverberation. The talkers were distributed in the frontal hemisphere between ±105°. The primary outcome measure was the time taken to identify the location of the target talker. Preliminary results show shorter response times with beamforming (in comparison to an omnidirectional setting), especially when more talkers were present. This framework provides a new means for examining the effects of hearing technologies on behavior in complex audio-visual scenes.

Selective spatial attention in lateralized multi-talker speech perception: EEG correlates and the role of age

Speech comprehension under dynamic cocktail party conditions requires auditory search for relevant speech content and focusing spatial attention on the target talker. Here, we investigated the development of these cognitive processes in a population of 329 participants aged 20–70 years. We used a multi-talker speech detection and perception task in which pairs of words (each consisting of a cue and a target word) were simultaneously presented from lateralized positions. Participants attended to predefined cue words and responded to the corresponding target. Task difficulty was varied by presenting cue and target stimuli at different intensity levels. Decline in performance was observed only in the oldest group (age range 53–70 years) and only in the most difficult condition. The EEG analysis of neurocognitive correlates of lateralized auditory attention and stimulus evaluation (N2ac, LPCpc, alpha power lateralization) revealed age-associated changes in focussing on and processing of task-relevant information, while no such deficits were found on early auditory search and target segregation. Irrespective of age, more challenging listening conditions were associated with an increased allocation of attentional resources.