Multi-talker Environments Research Articles

A Method for Assessing Auditory Spatial Analysis in Reverberant Multitalker Environments.

Deficits in spatial hearing can have a negative impact on listeners' ability to orient in their environment and follow conversations in noisy backgrounds and may exacerbate the experience of hearing loss as a handicap. However, there are no good tools available for reliably capturing the spatial hearing abilities of listeners in complex acoustic environments containing multiple sounds of interest. The purpose of this study was to explore a new method to measure auditory spatial analysis in a reverberant multitalker scenario. This study was a descriptive case control study. Ten listeners with normal hearing (NH) aged 20-31 yr and 16 listeners with hearing impairment (HI) aged 52-85 yr participated in the study. The latter group had symmetrical sensorineural hearing losses with a four-frequency average hearing loss of 29.7 dB HL. A large reverberant room was simulated using a loudspeaker array in an anechoic chamber. In this simulated room, 96 scenes comprising between one and six concurrent talkers at different locations were generated. Listeners were presented with 45-sec samples of each scene, and were required to count, locate, and identify the gender of all talkers, using a graphical user interface on an iPad. Performance was evaluated in terms of correctly counting the sources and accuracy in localizing their direction. Listeners with NH were able to reliably analyze scenes with up to four simultaneous talkers, while most listeners with hearing loss demonstrated errors even with two talkers at a time. Localization performance decreased in both groups with increasing number of talkers and was significantly poorer in listeners with HI. Overall performance was significantly correlated with hearing loss. This new method appears to be useful for estimating spatial abilities in realistic multitalker scenes. The method is sensitive to the number of sources in the scene, and to effects of sensorineural hearing loss. Further work will be needed to compare this method to more traditional single-source localization tests.

Effect of motion on speech recognition

The benefit of spatial separation for talkers in a multi-talker environment is well documented. However, few studies have examined the effect of talker motion on speech recognition. In the current study, we evaluated the effects of (1) motion of the target or distracters, (2) a priori information about the target and distracter spatial configurations, and (3) target and distracter location. In total, seventeen young adults with normal hearing were tested in a large anechoic chamber in two experiments. In Experiment 1, seven stimulus conditions were tested using the Coordinate Response Measure (Bolia et al., 2000) speech corpus, in which subjects were required to report the key words in a target sentence presented simultaneously with two distracter sentences. As in previous studies, there was a significant improvement in key word identification for conditions in which the target and distracters were spatially separated as compared to the co-located conditions. In addition, 1) motion of either talker or distracter resulted in improved performance compared to stationary presentation (talker motion yielded significantly better performance than distracter motion) 2) a priori information regarding stimulus configuration was not beneficial, and 3) performance was significantly better with key words at 0° azimuth as compared to −60° (on the listener's left). Experiment 2 included two additional conditions designed to assess whether the benefit of motion observed in Experiment 1 was due to the motion itself or to the fact that the motion conditions introduced small spatial separations in the target and distracter key words. Results showed that small spatial separations (on the order of 5-8°) resulted in improved performance (relative to co-located key words) whether the sentences were moving or stationary. These results suggest that in the presence of distracting messages, motion of either target or distracters and/or small spatial separation of the key words may be beneficial for sound source segregation and thus for improved speech recognition.

Identification and localization of concurrent speech signals in an auditory dual task

In Air Force operations, there is often a competing demand for auditory attention: listeners must identify and repeat a critical message, while also detecting and localizing the presence of a threat. Previous research (Iyer et al., 2014) has shown significant dual task interference, so that the ability of listeners to maintain a high level of performance on a primary task (keyword identification) was offset by decreased performance in a secondary task (critical call sign detection), relative to the respective single-task performances. The current experiment was designed to examine dual-task performance when the secondary task was changed from detection to localization. Listeners identified color-number keywords originating at 0o azimuth (primary task), while they localized the presence of a critical call sign originating from one of thirty-two locations surrounding them in 360° on the horizontal plane. The difficulty of the primary task was varied by employing an N-back task (0, 1, or 2) to increase memory load. Results indicated that the cost of dividing attention between the two tasks increased as the memory load in the primary task increased from 1-back to 2-back. More interestingly, while localization accuracy was unaffected, listeners’ ability to recognize the critical call sign degraded significantly as the difficulty of the primary task increased. The results have important implications for spatial auditory displays in multitalker environments.

Effects of a consistent target or masker voice on target speech intelligibility in two- and three-talker mixtures.

When the spatial location or identity of a sound is held constant, it is not masked as effectively by competing sounds. This suggests that experience with a particular voice over time might facilitate perceptual organization in multitalker environments. The current study examines whether listeners benefit from experience with a voice only when it is the target, or also when it is a masker, using diotic presentation and a closed-set task (coordinate response measure). A reliable interaction was observed such that, in two-talker mixtures, consistency of masker or target voice over 3-7 trials significantly benefited target recognition performance, whereas in three-talker mixtures, target, but not masker, consistency was beneficial. Overall, this work suggests that voice consistency improves intelligibility, although somewhat differently when two talkers, compared to three talkers, are present, suggesting that consistent-voice information facilitates intelligibility in at least two different ways. Listeners can use a template-matching strategy to extract a known voice from a mixture when it is the target. However, consistent-voice information facilitates segregation only when two, but not three, talkers are present.

Auditory masking of speech in reverberant multi-talker environments.

Auditory localization research needs to be performed in more realistic testing environments to better capture the real-world abilities of listeners and their hearing devices. However, there are significant challenges involved in controlling the audibility of relevant target signals in realistic environments. To understand the important aspects influencing target detection in more complex environments, a reverberant room with a multi-talker background was simulated and presented to the listener in a loudspeaker-based virtual sound environment. Masked thresholds of a short speech stimulus were measured adaptively for multiple target source locations in this scenario. It was found that both distance and azimuth of the target source have a strong influence on the masked threshold. Subsequently, a functional model was applied to analyze the factors influencing target detectability. The model is comprised of an auditory front-end that generates an internal representation of the stimuli in both ears, followed by a decision device combining d' information across time, frequency and both ears. The model predictions of the masked thresholds were overall in very good agreement with the experimental results. An analysis of the model processes showed that head shadow effects, signal spectrum, and reverberation have a strong impact on target audibility in the given scenario.

A Binaural Grouping Model for Predicting Speech Intelligibility in Multitalker Environments.

Spatially separating speech maskers from target speech often leads to a large intelligibility improvement. Modeling this phenomenon has long been of interest to binaural-hearing researchers for uncovering brain mechanisms and for improving signal-processing algorithms in hearing-assistive devices. Much of the previous binaural modeling work focused on the unmasking enabled by binaural cues at the periphery, and little quantitative modeling has been directed toward the grouping or source-separation benefits of binaural processing. In this article, we propose a binaural model that focuses on grouping, specifically on the selection of time-frequency units that are dominated by signals from the direction of the target. The proposed model uses Equalization-Cancellation (EC) processing with a binary decision rule to estimate a time-frequency binary mask. EC processing is carried out to cancel the target signal and the energy change between the EC input and output is used as a feature that reflects target dominance in each time-frequency unit. The processing in the proposed model requires little computational resources and is straightforward to implement. In combination with the Coherence-based Speech Intelligibility Index, the model is applied to predict the speech intelligibility data measured by Marrone et al. The predicted speech reception threshold matches the pattern of the measured data well, even though the predicted intelligibility improvements relative to the colocated condition are larger than some of the measured data, which may reflect the lack of internal noise in this initial version of the model.

The mismatch negativity as a measure of auditory stream segregation in a simulated “cocktail-party” scenario: effect of age

With age the ability to understand speech in multitalker environments usually deteriorates. The central auditory system has to perceptually segregate and group the acoustic input into sequences of distinct auditory objects. The present study used electrophysiological measures to study effects of age on auditory stream segregation in a multitalker scenario. Younger and older adults were presented with streams of short speech stimuli. When a single target stream was presented, the occurrence of a rare (deviant) syllable among a frequent (standard) syllable elicited the mismatch negativity (MMN), an electrophysiological correlate of automatic deviance detection. The presence of a second, concurrent stream consisting of the deviant syllable of the target stream reduced the MMN amplitude, especially when located nearby the target stream. The decrease in MMN amplitude indicates that the rare syllable of the target stream was less perceived as deviant, suggesting reduced stream segregation with decreasing stream distance. Moreover, the presence of a concurrent stream increased the MMN peak latency of the older group but not that of the younger group. The results provide neurophysiological evidence for the effects of concurrent speech on auditory processing in older adults, suggesting that older adults need more time for stream segregation in the presence of concurrent speech.

An evaluation of the performance of two binaural beamformers in complex and dynamic multitalker environments

Objective: Binaural beamformers are super-directional hearing aids created by combining microphone outputs from each side of the head. While they offer substantial improvements in SNR over conventional directional hearing aids, the benefits (and possible limitations) of these devices in realistic, complex listening situations have not yet been fully explored. In this study we evaluated the performance of two experimental binaural beamformers. Design: Testing was carried out using a horizontal loudspeaker array. Background noise was created using recorded conversations. Performance measures included speech intelligibility, localization in noise, acceptable noise level, subjective ratings, and a novel dynamic speech intelligibility measure. Study sample: Participants were 27 listeners with bilateral hearing loss, fitted with BTE prototypes that could be switched between conventional directional or binaural beamformer microphone modes. Results: Relative to the conventional directional microphones, both binaural beamformer modes were generally superior for tasks involving fixed frontal targets, but not always for situations involving dynamic target locations. Conclusions: Binaural beamformers show promise for enhancing listening in complex situations when the location of the source of interest is predictable.

Spatial release of cognitive load measured in a dual-task paradigm in normal-hearing and hearing-impaired listeners.

This study investigated whether spatial separation between talkers helps reduce cognitive processing load, and how hearing impairment interacts with the cognitive load of individuals listening in multi-talker environments. A dual-task paradigm was used in which performance on a secondary task (visual tracking) served as a measure of the cognitive load imposed by a speech recognition task. Visual tracking performance was measured under four conditions in which the target and the interferers were distinguished by (1) gender and spatial location, (2) gender only, (3) spatial location only, and (4) neither gender nor spatial location. Results showed that when gender cues were available, a 15° spatial separation between talkers reduced the cognitive load of listening even though it did not provide further improvement in speech recognition (Experiment I). Compared to normal-hearing listeners, large individual variability in spatial release of cognitive load was observed among hearing-impaired listeners. Cognitive load was lower when talkers were spatially separated by 60° than when talkers were of different genders, even though speech recognition was comparable in these two conditions (Experiment II). These results suggest that a measure of cognitive load might provide valuable insight into the benefit of spatial cues in multi-talker environments.

The focal index as a singular metric for beamforming effectiveness.

Hearing-impaired individuals frequently cite intelligibility problems in multi-talker environments. Microphone arrays performing time-delay beamforming address conditions of poor signal-to-noise ratio by spatially filtering incoming sound. Existing beam pattern metrics including peak side lobe level, integrated side lobe level, beamwidth, and planar directivity index fail to quantitatively capture all elements essential for improving speech intelligibility in multi-talker situations. The focal index (FI) was developed to address these deficiencies. Simulations were performed to exemplify the robust nature of the FI and to demonstrate the utility of this metric for driving array parameter selection. Beam patterns were generated and the metrics were calculated and evaluated against the strict unidirectional requirements for the array. Array performance was assessed by human subjects in a speech recognition task that incorporated competing speech from multiple locations. Simulations of array output were presented under conditions differing in array sparsity. The resulting human subject data were used to demonstrate the linear relationship (R(2) > 0.975) between speech-intelligibility-weighted FI (SII-FI) and the signal-to-noise ratio thresholds for 20% and 80% correct responses. Data indicate that the FI and SII-FI are robust singular metrics for determining the effectiveness of the array.

How difficult is difficult? Speech perception in noise in the elderly hearing impaired.

Perception of speech in the presence of competing multitalker noise is difficult for most individuals with sensory hearing loss, and in particular, for the elderly hearing impaired. Elderly people frequently report that these difficulties are poorly compensated for by hearing aids, albeit the algorithms and technologies aiming to improve speech perception in noise. The aim of the current study was therefore to assess competing speech signals processing by measuring the amount of signal to noise ratio (SNR) loss experienced by elderly hearing impaired individuals and their performance in dichotic listening tests. Speech in multitalker babble noise and dichotic listening were assessed in older hearing impaired individuals and in young normal-hearing adults. The average dichotic scores in the elderly group were substantially and significantly lower compared with the scores of the younger group with a significant right ear advantage (higher accuracy in reporting words arriving at the right ear than words arriving at the left ear) in the elderly group. Speech identification in noise in the older group was significantly poorer than in the younger group (SNR loss: 10 dB). The poor performance in the elderly hearing impaired group highlights the difficulties faced by hearing impaired older adults in demanding listening environments. Our results suggest that it may be unreasonable to expect that elderly hearing impaired individuals may fully understand speech in multitalker environments even when hearing aids are used.

Spatial attention in an auditory dual task

Research on divided auditory attention has focused on the ability of listeners to report keywords from two spatially separated simultaneous talkers (Best et al., 2006); however, the information that listeners extract from each talker is the same (i.e., keyword identification). In realistic listening environments, there is often a competing demand for auditory attention; listeners might be required to monitor critical auditory events while also responding to an ongoing auditory signal. It is not clear how spatial separation affects performance in auditory dual-tasks under varying levels of task difficulty. Listeners identified an ongoing stream of color/number keywords originating at 0° azimuth (primary task), while detecting the presence of a critical call sign originating from locations ranging from −45° to + 45° (secondary task). The difficulty of the primary task was varied by introducing noise or by requiring stimuli identification in an auditory one- or two-back memory recall task. The difficulty of the secondary task was varied by increasing the set-size of critical call signs that listeners had to monitor and by changing the SNRs. Preliminary results indicate that a listener's ability to detect the presence of a critical call sign increased as its spatial location moved further away from 0° azimuth and performance was modulated bythe difficulty of the primary task. The result has important implications for spatial attention as a function of task difficulty in multitalker environments.

Spatial and temporal modifications of multitalker speech can improve speech perception in older adults

Speech perception in multitalker environments often requires listeners to divide attention among several concurrent talkers before focusing on one talker with pertinent information. Such attentionally demanding tasks are particularly difficult for older adults due both to age-related hearing loss (presbacusis) and general declines in attentional processing and associated cognitive abilities. This study investigated two signal-processing techniques that have been suggested as a means of improving speech perception accuracy of older adults: time stretching and spatial separation of target talkers. Stimuli in each experiment comprised 2–4 fixed-form utterances in which listeners were asked to consecutively 1) detect concurrently spoken keywords in the beginning of the utterance (divided attention); and, 2) identify additional keywords from only one talker at the end of the utterance (selective attention). In Experiment 1, the overall tempo of each utterance was unaltered or slowed down by 25%; in Experiment 2 the concurrent utterances were spatially coincident or separated across a 180-degree hemifield. Both manipulations improved performance for elderly adults with age-appropriate hearing on both tasks. Increasing the divided attention load by attending to more concurrent keywords had a marked negative effect on performance of the selective attention task only when the target talker was identified by a keyword, but not by spatial location. These findings suggest that the temporal and spatial modifications of multitalker speech improved perception of multitalker speech primarily by reducing competition among cognitive resources required to perform attentionally demanding tasks.

The effects of dialect variation on speech intelligibility in a multitalker background

ABSTRACTSpeech intelligibility in a multitalker background can be affected by the language of both the talker and the interfering speech. This study investigated whether this interaction is modulated by dialect variations of the same language. American English listeners were presented with target sentences in either their own General American English (GAE) or a different accent (Southern American English [SAE]) masked by either GAE or SAE two-talker babble at three sound to noise ratios (SNRs): +3, 0, and –3 dB. All speech materials were produced by male talkers. Across all conditions, SAE target was more intelligible than GAE. Intelligibility of either target decreased as the level of the interfering babble noise increased. Target accent interacted with masking accent: at +3 dB SNR, GAE (and not SAE) was the more effective masker. The target-masker interaction was different as listening conditions deteriorated: at 0 and –3 dB SNR, masking accent did not affect GAE target, but when the target was SAE, the SAE masker (and not GAE) was more effective. Thus, at increased noise levels, listeners benefited from the mismatch between the target and masking accents only when the target was in a nonnative accent. These results demonstrate that dialect variation can influence listeners’ performance in a multitalker environment. The apparent asymmetry in intelligibility of accents may be in part related to dialect-specific prosodic and phonetic features.

Spatial release of the cognitive effort of understanding speech in multi-talker environments

Ervin Hafter was prescient in recognizing the need for taking into account top-down cognitive processing for understanding the interaction between auditory perception and cognition. His insight, that signal processing such as noise reduction may modify cognitive demands without changing auditory performance, led to a seminal study in collaboration with the Starkey Hearing Research Center which showed that hearing aid technology can reduce the cognitive effort of understanding noisy speech. Inspired by Erv’s insight, we are studying if increasing the spatial separation between competing talkers reduces the cognitive effort needed to listen in multi-talker environments. Following the lead of Erv’s seminal study, performance on a simultaneous secondary task, in our case visual tracking, is a measure of the cognitive effort consumed by the primary task of understanding target speech. Our results show that spatial separation can reduce cognitive effort even when it does not give further improvement in speech intelligibility over existing segregation cues. These results suggest that a measure of cognitive effort is useful for assessing the benefit of hearing technology that improves spatial segregation. This is an important finding because the measure addresses benefit along a dimension that is not captured in standard assessment of speech reception performance.

A Free-Source Method (FrSM) for Calibrating a Large-Aperture Microphone Array

Large-aperture microphone arrays can be used to capture and enhance speech from individual talkers in noisy, multi-talker, and reverberant environments. However, they must be calibrated, often more than once, to obtain accurate 3-dimensional coordinates for all microphones. Direct-measurement techniques, such as using a measuring tape or a laser-based tool are cumbersome and time-consuming. Some previous methods that used acoustic signals for array calibration required bulky hardware and/or fixed, known source locations. Others, which allowed more flexible source placement, often have issues with real data, have reported results for 2D only, or work only for small arrays. This paper describes a complete and robust method for automatic calibration using acoustic signals which is simple, repeatable, accurate, and has been shown to work for a real system. The method requires only a single transducer (speaker) with a microphone attached above its center. The unit is freely moved around the focal volume of the microphone array generating a single long recording from all the microphones. After that, the system is completely automatic. We describe the free source method (FrSM), validate its effectiveness and present accuracy results against measured ground truth. The performance of FrSM is compared to that from several other methods for a real 128-microphone array.

Attentional switching when listeners respond to semantic meaning expressed by multiple talkers

The party problem asks how we know what one person says when others are speaking at the same time. In our simulated cocktail party, a subject sits among multiple talkers, each telling a different story. A sequence of questions are drawn from the various stories and presented visually for subjects to answer with manual responses. One talker is identified (visually) as the primary, the one from whom a majority of questions are taken. Pay is based on correct answers, and attention is assessed by comparing response accuracy on questions from the primary and from the other talkers. With interest in the difficulty faced by older and often hearing-impaired listeners in multi-talker environments, the present experiment looks at the speed of shifting attention. All questions are from a primary, chosen at random from question to question. The speed of shifting is measured by varying the time from when the new primary is identified to the moment when the relevant information appears in that story. In a re...

Performance of a highly directional microphone array in a multi-talker reverberant environment

A visual guided hearing aid (VGHA) has recently been developed, which uses an eye tracker to steer the “acoustic look direction” (ALD) of a beamforming microphone array. The current study evaluates the performance of this highly directional microphone in providing spatial release from masking (SRM) under acoustically dry and reverberant conditions. Four normal-hearing subjects participated in a speech intelligibility test with collocated and spatially separated speech maskers when listening either through the microphone array or through KEMAR to simulate “natural” binaural conditions. The results indicated that near-normal SRM was achieved by listening through the VGHA in both environments. In the acoustically dry condition, SRM was similar to the measured signal-to-noise ratio (SNR) gain from the microphone array. However, in the reverberant condition, subjects showed significantly greater SRM than predicted from the measured SNR gain from the array. This is consistent with the measured improvement in SNR for the early part of the room impulse response for the target but not for the spatially separated maskers. This indicates that in some reverberant conditions the microphone array provides substantial source selection benefits.

A convex model and L1 minimization for musical noise reduction in blind source separation

Blind source separation (BSS) methods are useful tools to recover or enhance in- dividual speech sources from their mixtures in a multi-talker environment. A class of efficient BSS methods are based on the mutual exclusion hypothesis of the source signal Fourier spectra on the time- frequency (TF) domain, and subsequent data clustering and classification. Though such methodology is simple, the discontinuous decisions in the TF domain for classification often cause defects in the recovered signals in the time domain. The defects are perceived as unpleasant ringing sounds, the so called musical noise. Post-processing is desired for further quality enhancement. In this paper, an efficient musical noise reduction method is presented based on a convex model of time-domain sparse filters. The sparse filters are intended to cancel out the interference due to major sparse peaks in the mixing coefficients or physically the early arrival and high energy portion of the room impulse responses. This strategy is efficiently carried out by l1 regularization and the split Bregman method. Evaluations by both synthetic and room recorded speech and music data show that our method outperforms existing musical noise reduction methods in terms of objective and subjective measures. Our method can be used as a post-processing tool for more general and recent versions of TF domain BSS methods as well.

The effects of reverberation on a listener's ability to recognize target sentences in the presence of up to three synchronized masking sentences

Objective: To determine the effects of room reverberation on target sentence recognition in the presence of 0-to-3 synchronous masking sentences. Design: Target and masker sentences were presented through four loudspeakers (±90° and ±45° azimuth; 1m from the listener) in rooms having reverberation times (RT) of 0.2, 0.4, 0.6, and 1.1 s. Study Sample: Four groups of 13 listeners each participated in the study (N = 52). Results: In rooms with RTs of 0.2, 0.4, and 0.6 s, mean speech recognition scores (SRSs) were similar, with scores ranging from 96–100%, 90–95%, 75–80%, and 53–60%, when 0, 1, 2, and 3 competing sentences were present, respectively. However, in the room with a RT = 1.1 s, SRSs deteriorated significantly faster as the number of competing sentences increased; mean scores were 93%, 73%, 26%, and 10%, in the 0, 1, 2, 3, competing sentence condition, respectively. The majority of errors in SRSs (98%) resulted from listeners reporting words presented in masking sentences along with those in target sentences (mixing errors). Conclusions: Results indicate that reverberation has a similar influence on SRSs measured in multi-talker environments, when room reverberation is ≤ 0.6 s. However, SRSs are dramatically reduced in the room with a RT = 1.1 s, even when only one competing talker is present.SumarioObjetivo: Determinar los efectos de la reverberación de un cuarto para el reconocimiento de oraciones blanco, en presencia sincrónica de 0–3 oraciones enmascarantes. Diseño: Las oraciones blanco y las enmascarantes se presentaron por medio de cuatro altoparlantes (+90° y ±45° azimuth; a 1m del oyente) en cuartos que tenían tiempos de reverberación (RT) de 0.2, 0.4, 0.6 y 1.1 seg. Muestra de estudio: Participaron en este estudio, cuatro grupos de 13 oyentes cada uno (N=52). Resultados: En cuartos con RT de 0.2, 0.4 y 0.6 seg, las puntuaciones medias de reconocimiento del habla (SRS) fueron similares, con puntuaciones que variaron de 96–100%, 90–95%, 75–80%, y 53–60%, cuando se presentaron, respectivamente, 0, 1, 2, y 3 oraciones competitivas. Sin embargo, en el cuarto con RT=1.1 seg, las SRS se deterioraron significativamente más rápido, conforme el número de frases competitivas se incrementó; las puntuaciones medias fueron de 93%, 73%, 26%, y 10%, en las condiciones de 0, 1, 2 y 3 oraciones competitivas, respectivamente. La mayoría de los errores en las SRS (98%) resultaron de oyentes que reportaron palabras presentes en las oraciones enmascarantes junto con aquellas de las oraciones blanco. (mezcla de errores). Conclusiones: Los resultados indican que la reverberación tiene una influencia similar en las SRS, medidas en ambientes de hablantes múltiples, cuando la reverberación es ≦0.6 sec. Sin embargo, las SRS se reducen dramáticamente en un cuarto con un RT=1.1 seg, incluso cuando solamente está presente un mensaje competitivo.