Cocktail-party Listening Research Articles

Speech comprehension in the context of speaker changes: The importance of voice-feature continuity at the cocktail party

In this talk, I will show how different groups of listeners use voice acoustics to enhance speech comprehension under adverse listening conditions, specifically when the auditory scene comprises a multitude of sounds heard at once. These “cocktail-party”-like situations pose a difficult conceptual problem: To comprehend target speech, listeners need to attend to the target voice while at the same time ignoring other irrelevant sounds. The cocktail-party problem is made considerably easier when all target sounds are spoken by the same talker. Previous work suggests that such benefits from voice continuity can be—in large part—attributed to two prominent voice features: Glottal-pulse rate (GPR) and vocal-tract length (VTL). GPR determines the fundamental frequency of a speech sound and is perceived as vocal pitch; VTL determines the spectral envelope of a speech sound and is perceived as an aspect of vocal timbre. Apart from being important voice identity cues, GPR and VTL have been shown to play a crucial role in cocktail-party listening. Here, I will present data from a series of experiments highlighting the importance of voice-feature continuity for speech comprehension at the cocktail party.

Cocktail-party listening and cognitive abilities show strong pleiotropy.

The cocktail-party problem refers to the difficulty listeners face when trying to attend to relevant sounds that are mixed with irrelevant ones. Previous studies have shown that solving these problems relies on perceptual as well as cognitive processes. Previously, we showed that speech-reception thresholds (SRTs) on a cocktail-party listening task were influenced by genetic factors. Here, we estimated the degree to which these genetic factors overlapped with those influencing cognitive abilities. We measured SRTs and hearing thresholds (HTs) in 493 listeners, who ranged in age from 18 to 91 years old. The same individuals completed a cognitive test battery comprising 18 measures of various cognitive domains. Individuals belonged to large extended pedigrees, which allowed us to use variance component models to estimate the narrow-sense heritability of each trait, followed by phenotypic and genetic correlations between pairs of traits. All traits were heritable. The phenotypic and genetic correlations between SRTs and HTs were modest, and only the phenotypic correlation was significant. By contrast, all genetic SRT-cognition correlations were strong and significantly different from 0. For some of these genetic correlations, the hypothesis of complete pleiotropy could not be rejected. Overall, the results suggest that there was substantial genetic overlap between SRTs and a wide range of cognitive abilities, including abilities without a major auditory or verbal component. The findings highlight the important, yet sometimes overlooked, contribution of higher-order processes to solving the cocktail-party problem, raising an important caveat for future studies aiming to identify specific genetic factors that influence cocktail-party listening.

Selective auditory attention modulates cortical responses to sound location change for speech in quiet and in babble.

Listeners use the spatial location or change in spatial location of coherent acoustic cues to aid in auditory object formation. From stimulus-evoked onset responses in normal-hearing listeners using electroencephalography (EEG), we have previously shown measurable tuning to stimuli changing location in quiet, revealing a potential window into the cortical representations of auditory scene analysis. These earlier studies used non-fluctuating, spectrally narrow stimuli, so it was still unknown whether previous observations would translate to speech stimuli, and whether responses would be preserved for stimuli in the presence of background maskers. To examine the effects that selective auditory attention and interferers have on object formation, we measured cortical responses to speech changing location in the free field with and without background babble (+6 dB SNR) during both passive and active conditions. Active conditions required listeners to respond to the onset of the speech stream when it occurred at a new location, explicitly indicating 'yes' or 'no' to whether the stimulus occurred at a block-specific location either 30 degrees to the left or right of midline. In the aggregate, results show similar evoked responses to speech stimuli changing location in quiet compared to babble background. However, the effect of the two background environments diverges somewhat when considering the magnitude and direction of the location change and where the subject was attending. In quiet, attention to the right hemifield appeared to evoke a stronger response than attention to the left hemifield when speech shifted in the rightward direction. No such difference was found in babble conditions. Therefore, consistent with challenges associated with cocktail party listening, directed spatial attention could be compromised in the presence of stimulus noise and likely leads to poorer use of spatial cues in auditory streaming.

An Effect of Gaze Direction in Cocktail Party Listening.

It is well established that gaze direction can influence auditory spatial perception, but the implications of this interaction for performance in complex listening tasks is unclear. In the current study, we investigated whether there is a measurable effect of gaze direction on speech intelligibility in a "cocktail party" listening situation. We presented sequences of digits from five loudspeakers positioned at 0°, ± 15°, and ± 30° azimuth, and asked participants to repeat back the digits presented from a designated target loudspeaker. In different blocks of trials, the participant visually fixated on a cue presented at the target location or at a nontarget location. Eye position was tracked continuously to monitor compliance. Performance was best when fixation was on-target (vs. off-target) and the size of this effect depended on the specific configuration. This result demonstrates an influence of gaze direction in multitalker mixtures, even in the absence of visual speech information.

The Genetic contribution to solving the cocktail-party problem

SummaryCommunicating in everyday situations requires solving the cocktail-party problem, or segregating the acoustic mixture into its constituent sounds and attending to those of most interest. Humans show dramatic variation in this ability, leading some to experience real-world problems irrespective of whether they meet criteria for clinical hearing loss. Here, we estimated the genetic contribution to cocktail-party listening by measuring speech-reception thresholds (SRTs) in 425 people from large families and ranging in age from 18 to 91 years. Roughly half the variance of SRTs was explained by genes (h2 = 0.567). The genetic correlation between SRTs and hearing thresholds (HTs) was medium (ρG = 0.392), suggesting that the genetic factors influencing cocktail-party listening were partially distinct from those influencing sound sensitivity. Aging and socioeconomic status also strongly influenced SRTs. These findings may represent a first step toward identifying genes for “hidden hearing loss,” or hearing problems in people with normal HTs.

Revealing Perceptional and Cognitive Mechanisms in Static and Dynamic Cocktail Party Listening by Means of Error Analyses.

In cocktail party situations multiple talkers speak simultaneously, which causes listening to be perceptually and cognitively challenging. Such situations can either be static (fixed target talker) or dynamic, meaning the target talker switches occasionally and in a potentially unpredictable way. To shed light on the perceptional and cognitive mechanisms in static and dynamic cocktail party situations, we conducted an analysis of error types that occur during a multi-talker speech recognition test. The error analysis distinguished between misunderstood or omitted words (random errors) and target-masker confusions. To investigate the effects of aging and hearing impairment, we compared data from three listener groups, comprised of young as well as older adults with and without hearing loss. In the static condition, error rates were generally very low, except for the older hearing-impaired listeners. Consistent with the assumption of decreased audibility, they showed a notable amount of random errors. In the dynamic condition, errors increased compared to the static condition, especially immediately following a target talker switch. Those increases were similar for random and confusion errors. The older hearing-impaired listeners showed greater difficulties than the younger adults in trials not preceded by a switch. These results suggest that the load associated with dynamic cocktail party listening affects the ability to focus attention on the talker of interest and the retrieval of words from short-term memory, as indicated by the increased amount of confusion and random errors. This was most pronounced in the older hearing-impaired listeners proposing an interplay of perceptual and cognitive mechanisms.

Speech envelope enhancement to improve cocktail-party listening

A novel speech enhancement scheme is being developed in our lab with the goal of improving speech intelligibility in “cocktail party” listening environments. By manipulating the temporal envelope to increase the salience of acoustic onsets, the algorithm improves access to binaural cues sampled at these onsets. The hope is that this will lead to a more robust spatial perception and improved streaming of competing talkers that could impact a variety of applications including hearing aids and cochlear implants. The algorithm is designed to run in real time, and in order to test its efficacy in real-world environments, it must be implemented on a portable device. For this project, the algorithm is being written in the Teensyduino software and uploaded to a Tympan Rev E DSP unit fitted with a Tympan AIC CODEC daughterboard. External speech is both captured by and played back through a pair of Tympan BTE earpieces, and all processing is performed independently for each ear by the Tympan’s Teensy 4.1 microcontroller. Assuming that the algorithm is successfully implemented, speech signals delivered to the listener will have perceptibly higher crest factors (i.e., higher transient energy) than at input. The portable, bilateral, real-time implementation developed here will allow us to evaluate any associated binaural effects and their consequences in complex real-world listening tasks.

Costs of dynamic cocktail party listening: Investigating the effects of cognitive abilities and hearing impairment.

Situations with multiple competing talkers are especially challenging for listeners with hearing impairment. These "cocktail party" situations can either be static (fixed target talker) or dynamic (changing target talker). Relative to static situations, dynamic listening is typically associated with increased cognitive load and decreased speech recognition ("costs"). This study addressed the role of hearing impairment and cognition in two groups of older listeners with and without hearing loss. In most of the dynamic situations, the costs did not differ between the listener groups. There was no clear evidence that overall costs show an association with the individuals' cognitive abilities.

Additivity of segregation cues in simulated cocktail-party listening.

An approach is borrowed from Measurement Theory [Krantz et al. (1971). Foundations of Measurement (Academic, New York), Vol. 1] to evaluate the interaction of voice fundamental frequency and spatial cues in the segregation of talkers in simulated cocktail-party listening. The goal is to find a mathematical expression whereby the combined effect of cues can be simply related to their individual effects. On each trial, the listener judged whether an interleaved sequence of four vowel triplets (heard over headphones) were spoken by the same (MMM) or different (FMF) talkers. The talkers had nominally different fundamental frequencies and spoke from nominally different locations (simulated using head-related transfer functions). Natural variation in these cues was simulated by adding a small, random perturbation to the nominal values independently for each vowel on each trial. Psychometric functions (PFs) relating d' performance to the difference in nominal values were obtained for the cues presented individually and in combination. The results revealed a synergistic interaction of cues wherein the PFs for cues presented in combination exceeded the simple vector sum of the PFs for the cues presented individually. The results are discussed in terms of their implications for possible emergent properties of cues affecting performance in simulated cocktail-party listening.

Static and dynamic cocktail party listening in younger and older adults

Verbal communication often takes place in situations with several simultaneous speakers (“cocktail party listening”). These situations can be static (only one listening target) or dynamic (with alternating targets). In particular, dynamic cocktail party listening is believed to generate extra cognitive load and appears to be particularly demanding for older listeners.Two groups of younger and older listeners with good hearing and normal cognition participated in the present study. Three different, spatially separated talker voices uttering matrix sentences were presented to each listener with varying types and probabilities of target switches. Moreover, several neuropsychological tests were conducted to investigate general cognitive characteristics that may be important for speech understanding in these situations.In a static condition with a priori knowledge of the target talker, both age groups revealed very high speech recognition performance. In comparison, dynamic conditions caused extra costs associated with the need to monitor different potential sound sources and to refocus attention when the target changed. The amount of costs depended on the probability and type of target talker alterations. Again, no significant age-group differences were found. No significant associations of cognitive characteristics and costs could be shown. However, a more fine-grained analysis based on the calculation of general and specific switch costs showed different mechanisms in older and younger listeners.This study confirms that dynamic cocktail party listening is associated with costs that depend on the type and probability of target switches. It extends previous research by showing that the effects of switching type and probability are similar for younger and older listeners with good hearing and good cognitive abilities. It further shows that, despite comparable costs of dynamic listening, mechanisms are different for the two age groups, as switching auditory attention may be preserved with aging, but monitoring different sound sources appears to be more difficult for older adults.

Suprathreshold Processing and Cocktail Party Listening in Younger and Older Adults with Normal Hearing

Present study aimed to investigate the effect of age and suprathreshold processing on cocktail party listening in individuals with normal hearing sensitivity. A total of 92 participants with normal hearing sensitivity were included in the study. They were divided into two groups based on their age. Fifty two young normal hearing adults in the age range of 20–40 years and 40 older normal hearing adults in the age range of 60-80 years. Tests administered included speech perception in noise test, spatial selective attention, gap detection thresholds, temporal modulation transfer function, inter-aural time difference, differential limen of frequency and ripple noise discrimination. Results showed that older adults performed poorer than younger adults in all the tests. Also, temporal cues showed a better relation with speech perception in noise compared to the spectral cues. This can be attributed to the disrupted neural synchrony which is due to poor frequency selectivity as observed through ripple noise discrimination. Individuals rely more on temporal cues due to poorer frequency resolution and phase locking mechanism and also on top down processes such as selective attention too. A degraded speech input would lead them to rely more on their higher cognition.

A detection-theoretic approach to cocktail-party listening

Cocktail-party listening (CPL) is a key-term referring to difficult listening situations wherein one must "h"" and follow the speech of individual talkers in a crowd. Such situations are commonplace in everyday listening but can be a challenge to study. Important factors affecting performance, such as voice similarity among talkers and uncertainty associated with the dynamic variation of speech, can be difficult to quantify. Spectral and spatial acoustic cues distinguishing talkers are of critical importance but are expressed in different physical units making their relative role difficult to evaluate beyond the particulars of a study. There are also often huge individual differences in listener performance that can complicate the interpretation of results. This paper reviews recent applications of detection theory designed to address these challenges. The distinguishing feature of the approach is that key factors are evaluated in terms of their contribution to the information divergence of talkers, DKL; a single statistic that dictates optimal performance for each task [Lutfi et al., J. Acoust. Soc. Am. 134, 2160–2170 (2013)]. Both published and unpublished studies are reviewed demonstrating the application of the approach to each of the major challenges described above. [Work supported by NIDCD R01-DC001262].

Synergy of spectral and spatial segregation cues in simulated cocktail party listening

An approach is borrowed from Measurement Theory [Krantz et al. (1971). Foundations of Measurement, Vol.1] to evaluate the interaction of spectral and spatial cues in the segregation of talkers in simulated cocktail-party listening. The goal is to determine whether mathematical transformations exist whereby the combined effect of cues can be additively related to their individual effects. On each trial, the listener judged whether an interleaved sequence of 4 vowel triplets (heard over headphones) were spoken by the same BBB BBB... or different ABA ABA... talkers. The talkers had nominally different fundamental frequencies and spoke from nominally different locations (simulated using Kemar HRTFs). Natural variation in these cues was simulated by adding a small, random perturbation to nominal values independently for each vowel on each trial. Psychometric functions (PFs) relating d’ performance to the difference in nominal values were obtained for the cues presented individually and in combination. The results revealed a synergistic interaction of cues wherein the PFs for cues presented in combination exceeded the simple vector sum of the PFs for the cues presented individually. The results are discussed in terms of their implications for possible emergent properties of cues affecting performance in simulated cocktail-party listening.

Influence of subcortical auditory processing and cognitive measures on cocktail party listening in younger and older adults

Objective: The study aimed to investigate the influence of subcortical auditory processing and cognitive measures on cocktail party listening in younger and older adults with normal hearing sensitivity.Design: Tests administered included quick speech perception in noise test to assess cocktail party listening, speech auditory brainstem response to assess subcortical auditory processing and digit span, digit sequencing and spatial selective attention test to assess cognitive processing.Study sample: A total of 92 participants with normal hearing sensitivity participated in the study. They were divided into two groups: 52 young adults (20–40 years) and 40 older adults (60–80 years).Results: The older adults performed significantly poorer than, the younger adults on the quick speech perception in noise test and various cognitive measures. Further, cognitive measures correlated with speech perception in noise in younger and older adults. The results of this study also showed that there was a significant deterioration in brainstem encoding of speech with ageing. Further, it was also noted that the fundamental frequency of the speech auditory brainstem response correlated with speech perception in noise.Conclusions: It can be concluded from this study that subcortical auditory processing and cognitive measures play a role in cocktail party listening.

Individual differences in cocktail party listening: The relative role of decision weights and internal noise

A simulated “cocktail-party” listening experiment was conducted to determine the relative role of decision weights and internal noise in accounting for the large individual differences in performance typically observed in these experiments. The listener heard over headphones interleaved sequences of random vowels and were asked to judge on each trial whether the vowels were spoken by the same AAA or different ABA talkers. The A and B vowels had nominally different Fo and spatial position (simulated using Kemar HRTFs), but were randomly perturbed around these values on each presentation. Decision weights for each dimension, internal noise, and efficiency measures were estimated using COSS analysis [Berg (1990). J. Acoust. Soc. Am. 88, 149-158]. Decision weights were nonoptimal and differed across listeners, but weighting efficiency across individuals was quite similar. Individual differences in performance accuracy ranging over 40 percentage points were largely related to differences in internal noise. The...

Rapid learning of sound schema for the cocktail party problem

Auditory scene analysis depends on knowledge of natural sound structure, but little is known about how source-specific structures might be learned and applied. We explored whether listeners internalize “schemas”—the abstract structure shared by different occurrences of the same type of sound source—during cocktail-party listening. We measured the ability to detect one of two concurrent “melodies” that did not differ in mean pitch (nor in timbre), ensuring that only the structure of these melodies over time could be used to distinguish them. Target melodies were cued by presenting them in isolation before each mixture, transposed to avoid exact repetition. The task was to determine if the cued melody was present in the subsequent mixture. Listeners performed above chance despite transposition between cue and target. Particular melodic schemas could recur across a subset of trials within a block, as well as across blocks separated by epochs in which the schema was absent. Recurrence across trials within a block facilitated target detection, and the advantage grew over the experiment despite intervening blocks, suggestive of learning. The results indicate that rapid and persistent internalization of source schemas can promote accurate perceptual organization of sound sources that recur intermittently in the auditory environment.

Role of Binaural Temporal Fine Structure and Envelope Cues in Cocktail-Party Listening.

While conversing in a crowded social setting, a listener is often required to follow a target speech signal amid multiple competing speech signals (the so-called "cocktail party" problem). In such situations, separation of the target speech signal in azimuth from the interfering masker signals can lead to an improvement in target intelligibility, an effect known as spatial release from masking (SRM). This study assessed the contributions of two stimulus properties that vary with separation of sound sources, binaural envelope (ENV) and temporal fine structure (TFS), to SRM in normal-hearing (NH) human listeners. Target speech was presented from the front and speech maskers were either colocated with or symmetrically separated from the target in azimuth. The target and maskers were presented either as natural speech or as "noise-vocoded" speech in which the intelligibility was conveyed only by the speech ENVs from several frequency bands; the speech TFS within each band was replaced with noise carriers. The experiments were designed to preserve the spatial cues in the speech ENVs while retaining/eliminating them from the TFS. This was achieved by using the same/different noise carriers in the two ears. A phenomenological auditory-nerve model was used to verify that the interaural correlations in TFS differed across conditions, whereas the ENVs retained a high degree of correlation, as intended. Overall, the results from this study revealed that binaural TFS cues, especially for frequency regions below 1500 Hz, are critical for achieving SRM in NH listeners. Potential implications for studying SRM in hearing-impaired listeners are discussed. Acoustic signals received by the auditory system pass first through an array of physiologically based band-pass filters. Conceptually, at the output of each filter, there are two principal forms of temporal information: slowly varying fluctuations in the envelope (ENV) and rapidly varying fluctuations in the temporal fine structure (TFS). The importance of these two types of information in everyday listening (e.g., conversing in a noisy social situation; the "cocktail-party" problem) has not been established. This study assessed the contributions of binaural ENV and TFS cues for understanding speech in multiple-talker situations. Results suggest that, whereas the ENV cues are important for speech intelligibility, binaural TFS cues are critical for perceptually segregating the different talkers and thus for solving the cocktail party problem.

Electrophysiological correlates of cocktail-party listening

Detecting, localizing, and selectively attending to a particular sound source of interest in complex auditory scenes composed of multiple competing sources is a remarkable capacity of the human auditory system. The neural basis of this so-called “cocktail-party effect” has remained largely unknown. Here, we studied the cortical network engaged in solving the “cocktail-party” problem, using event-related potentials (ERPs) in combination with two tasks demanding horizontal localization of a naturalistic target sound presented either in silence or in the presence of multiple competing sound sources. Presentation of multiple sound sources, as compared to single sources, induced an increased P1 amplitude, a reduction in N1, and a strong N2 component, resulting in a pronounced negativity in the ERP difference waveform (N2d) around 260ms after stimulus onset. About 100ms later, the anterior contralateral N2 subcomponent (N2ac) occurred in the multiple-sources condition, as computed from the amplitude difference for targets in the left minus right hemispaces. Cortical source analyses of the ERP modulation, resulting from the contrast of multiple vs. single sources, generally revealed an initial enhancement of electrical activity in right temporo–parietal areas, including auditory cortex, by multiple sources (at P1) that is followed by a reduction, with the primary sources shifting from right inferior parietal lobule (at N1) to left dorso–frontal cortex (at N2d). Thus, cocktail-party listening, as compared to single-source localization, appears to be based on a complex chronology of successive electrical activities within a specific cortical network involved in spatial hearing in complex situations.

The mutual roles of temporal glimpsing and vocal characteristics in cocktail-party listening

At a cocktail party, listeners must attend selectively to a target speaker and segregate their speech from distracting speech sounds uttered by other speakers. To solve this task, listeners can draw on a variety of vocal, spatial, and temporal cues. Recently, Vestergaard et al. [J. Acoust. Soc. Am. 125, 1114-1124 (2009)] developed a concurrent-syllable task to control temporal glimpsing within segments of concurrent speech, and this allowed them to measure the interaction of glottal pulse rate and vocal tract length and reveal how the auditory system integrates information from independent acoustic modalities to enhance recognition. The current paper shows how the interaction of these acoustic cues evolves as the temporal overlap of syllables is varied. Temporal glimpses as short as 25 ms are observed to improve syllable recognition substantially when the target and distracter have similar vocal characteristics, but not when they are dissimilar. The effect of temporal glimpsing on recognition performance is strongly affected by the form of the syllable (consonant-vowel versus vowel-consonant), but it is independent of other phonetic features such as place and manner of articulation.

Effect of target-masker similarity on across-ear interference in a dichotic cocktail-party listening task

Similarity between the target and masking voices is known to have a strong influence on performance in monaural and binaural selective attention tasks, but little is known about the role it might play in dichotic listening tasks with a target signal and one masking voice in the one ear and a second independent masking voice in the opposite ear. This experiment examined performance in a dichotic listening task with a target talker in one ear and same-talker, same-sex, or different-sex maskers in both the target and the unattended ears. The results indicate that listeners were most susceptible to across-ear interference with a different-sex within-ear masker and least susceptible with a same-talker within-ear masker, suggesting that the amount of across-ear interference cannot be predicted from the difficulty of selectively attending to the within-ear masking voice. The results also show that the amount of across-ear interference consistently increases when the across-ear masking voice is more similar to the target speech than the within-ear masking voice is, but that no corresponding decline in across-ear interference occurs when the across-ear voice is less similar to the target than the within-ear voice. These results are consistent with an "integrated strategy" model of speech perception where the listener chooses a segregation strategy based on the characteristics of the masker present in the target ear and the amount of across-ear interference is determined by the extent to which this strategy can also effectively be used to suppress the masker in the unattended ear.