Unattended Speech Research Articles

Semantic processing of unattended speech in dichotic listening.

This study investigated whether unattended speech is processed at a semantic level in dichotic listening using a semantic priming paradigm. A lexical decision task was administered in which target words were presented in the attended auditory channel, preceded by two prime words presented simultaneously in the attended and unattended channels, respectively. Both attended and unattended primes were either semantically related or unrelated to the attended targets. Attended prime-target pairs were presented in isolation, whereas unattended primes were presented in the context of a series of rapidly presented words. The fundamental frequency of the attended stimuli was increased by 40 Hz relative to the unattended stimuli, and the unattended stimuli were attenuated by 12 dB [+12 dB signal-to-noise ratio (SNR)] or presented at the same intensity level as the attended stimuli (0 dB SNR). The results revealed robust semantic priming of attended targets by attended primes at both the +12 and 0 dB SNRs. However, semantic priming by unattended primes emerged only at the 0 dB SNR. These findings suggest that the semantic processing of unattended speech in dichotic listening depends critically on the relative intensities of the attended and competing signals.

Neural Systems Involved When Attending to a Speaker.

Remembering what a speaker said depends on attention. During conversational speech, the emphasis is on working memory, but listening to a lecture encourages episodic memory encoding. With simultaneous interference from background speech, the need for auditory vigilance increases. We recreated these context-dependent demands on auditory attention in 2 ways. The first was to require participants to attend to one speaker in either the absence or presence of a distracting background speaker. The second was to alter the task demand, requiring either an immediate or delayed recall of the content of the attended speech. Across 2 fMRI studies, common activated regions associated with segregating attended from unattended speech were the right anterior insula and adjacent frontal operculum (aI/FOp), the left planum temporale, and the precuneus. In contrast, activity in a ventral right frontoparietal system was dependent on both the task demand and the presence of a competing speaker. Additional multivariate analyses identified other domain-general frontoparietal systems, where activity increased during attentive listening but was modulated little by the need for speech stream segregation in the presence of 2 speakers. These results make predictions about impairments in attentive listening in different communicative contexts following focal or diffuse brain pathology.

Differential modulation of auditory responses to attended and unattended speech in different listening conditions

This study investigates how top-down attention modulates neural tracking of the speech envelope in different listening conditions. In the quiet conditions, a single speech stream was presented and the subjects paid attention to the speech stream (active listening) or watched a silent movie instead (passive listening). In the competing speaker (CS) conditions, two speakers of opposite genders were presented diotically. Ongoing electroencephalographic (EEG) responses were measured in each condition and cross-correlated with the speech envelope of each speaker at different time lags. In quiet, active and passive listening resulted in similar neural responses to the speech envelope. In the CS conditions, however, the shape of the cross-correlation function was remarkably different between the attended and unattended speech. The cross-correlation with the attended speech showed stronger N1 and P2 responses but a weaker P1 response compared to the cross-correlation with the unattended speech. Furthermore, the N1 response to the attended speech in the CS condition was enhanced and delayed compared with the active listening condition in quiet, while the P2 response to the unattended speaker in the CS condition was attenuated compared with the passive listening in quiet. Taken together, these results demonstrate that top-down attention differentially modulates envelope-tracking neural activity at different time lags and suggest that top-down attention can both enhance the neural responses to the attended sound stream and suppress the responses to the unattended sound stream.

Spiking neural network model of cortical auditory source segregation

Humans have the remarkable ability to tune into a particular voice even in loud noisy environments. The neural underpinnings of this amazing perceptual phenomenon are not yet fully understood. Recent EcoG [1] and MEG studies [2] have established that the neural representation of the attended speaker’s speech is much stronger than the unattended (distractor) speech when human subjects are asked to pay attention to a target speaker in a mixture of speech. How the brain sieves through the mixture waveform to enhance the target speaker’s speech and attenuate the background acoustic scene is still being investigated. In this work, we propose a spiking neural network architecture based on the theory of temporal coherence [3] to achieve auditory source segregation. Our model does not require training on the background noise or prior exposure to the target speech. Along with using bottom-up spectro-temporal features and pitch features, the model can also accommodate top-down attentional mechanisms to generate segregated phase locked neural representations to target speaker’s speech envelope. The model comprises of a feature extraction stage followed by clustering stage. The feature extraction stage mimics the auditory pathway starting from a cochlear representation followed by a multi-resolution analysis of the cochlear output using a bank of band-pass filters (cortical stage), to provide a rich timbre representation. Dominant pitch tracks are extracted from the sound mixture and processed through the same set of band-pass filters as the timbre channels. The output of the feature extraction stage comprising of the pitch and timbre channels are transduced into a spike-based representation using leaky integrate and fire neurons with time constants tuned to the bandwidth of the multi-resolution band-pass filters. The clustering stage comprises of a bank of coincidence detector neurons. Using the pitch signals as anchors the coincidence detector neurons can segregate the two sources from the mixture timbre representation. Thus, the output of the coincidence detector neurons comprises only of responses phase locked to the envelope of a single source. This model does not require any weight learning, is unsupervised and can segregate sources online. Previous studies on correlation based sound segregation employed network of neurons with intrinsic oscillator dynamics [4]. In this work, clustering of features belonging to a single source is driven only by the temporal coherence of spectro-temporal features of the given source. This spike-based representation provides an easy mechanism to group coherent features, which otherwise would require computationally expensive numerical routines for online, adaptive principal components analysis. Future work is aimed at reconstructing the speech waveform from the segregated spike trains.

Envelope responses in single-trial EEG indicate attended speaker in a ‘cocktail party’

Objective. Recent studies have shown that auditory cortex better encodes the envelope of attended speech than that of unattended speech during multi-speaker (‘cocktail party’) situations. We investigated whether these differences were sufficiently robust within single-trial electroencephalographic (EEG) data to accurately determine where subjects attended. Additionally, we compared this measure to other established EEG markers of attention. Approach. High-resolution EEG was recorded while subjects engaged in a two-speaker ‘cocktail party’ task. Cortical responses to speech envelopes were extracted by cross-correlating the envelopes with each EEG channel. We also measured steady-state responses (elicited via high-frequency amplitude modulation of the speech) and alpha-band power, both of which have been sensitive to attention in previous studies. Using linear classifiers, we then examined how well each of these features could be used to predict the subjects’ side of attention at various epoch lengths. Main results. We found that the attended speaker could be determined reliably from the envelope responses calculated from short periods of EEG, with accuracy improving as a function of sample length. Furthermore, envelope responses were far better indicators of attention than changes in either alpha power or steady-state responses. Significance. These results suggest that envelope-related signals recorded in EEG data can be used to form robust auditory BCI’s that do not require artificial manipulation (e.g., amplitude modulation) of stimuli to function.

Effects of unattended speech on performance and subjective distraction: The role of acoustic design in open-plan offices

Unattended background speech is a known source of cognitive and subjective distraction in open-plan offices. This study investigated whether the deleterious effects of background speech can be affected by room acoustic design that decreases speech intelligibility, as measured by the Speech Transmission Index (STI). The experiment was conducted in an open-plan office laboratory (84m2) in which four acoustic conditions were physically built. Three conditions contained background speech. A quiet condition was included for comparison. The speech conditions differed in terms of the degree of absorption, screen height, desk isolation, and the level of masking sound. The speech sounds simulated an environment where phone conversations are heard from different locations varying in distance. Ninety-eight volunteers were tested. The presence of background speech had detrimental effects on the subjective perceptions of noise effects and on cognitive performance in short-term memory and working memory tasks. These effects were not attenuated nor amplified within a three-hour working period. The reduction of the STI by room acoustic means decreased subjective disturbance, whereas the effects on cognitive performance were somewhat smaller than expected. The effects of room acoustic design on subjective distraction were stronger among noise-sensitive subjects, suggesting that they benefited more from acoustic improvements than non-sensitive subjects. The results imply that reducing the STI is beneficial for performance and acoustic satisfaction especially regarding speech coming from more distant desks. However, acoustic design does not sufficiently decrease the distraction caused by speech from adjacent desks.

Effects of spectral degradation on attentional modulation of auditory responses to continuous speech

Auditory attention enhances phase-locked responses to the temporal envelope of the attended speech stream in a competing background. This study investigates how cortical responses are affected by spectral degradation and the extent to which low-frequency fine-structure cues facilitate sound segregation. Two competing speech streams were presented diotically to normal-hearing subjects. Each subject was tested with unprocessed speech, vocoder speech (8 to 64 channels), and six-channel high-frequency vocoder + lowpass filtered speech simulating electric-acoustic stimulation (EAS). Ongoing EEG responses were measured in each condition. Cross-correlation between speech envelope and EEG responses was calculated at different time lags. For unprocessed speech, the cross-correlation function showed opposite signs of correlations between the attended and unattended speech, supporting the neural mechanism of suppression of the competing speech stream. This suppression mechanism, however, was only evident for the 32- and 64-channel vocoder conditions, but not for the 8- and 16-channel conditions. This indicates that greater frequency resolution is required for sound segregation. For EAS, the pattern of cortical responses was similar to that of the eight-channel vocoder condition, although speech intelligibility was higher in EAS than vocoder speech. Consistent with previous psychoacoustic evidence, these neural results further argue against the segregation mechanism for EAS benefit.

Visual short-term store can compensate for a defective phonological short-term store in patients with apraxia of speech.

The aim of the study was to investigate whether, within the phonological short-term memory (STM) system, speech articulation disorders primarily due to cortical damage are associated with deficits in the phonological recoding of visual-verbal material, and whether the visual short-term store (STS) can compensate for inefficient access to the phonological STS, as suggested in previous reports. Two patients (AE and TM) with apraxia of speech due to atrophic cerebral damage were administered a battery of tasks devised to explore the organization of the phonological STM. AE's span was normal and TM's span was markedly reduced. Phonological similarity and word-length effects: Both patients showed the effects in visual presentation; the effect was less evident in verbal presentation. This suggests a phonological STS disorder and preserved rehearsal/phonological recoding, consistently with involvement of the left parietal regions, which was documented by the MRI in both patients. Unexpectedly, TM had a longer span for similar than for dissimilar words in visually presented stimuli. Silent phonology (on written material): At variance with controls, stress assignment improved during unattended speech both in AE and TM, while in both patients and controls a detrimental effect of unattended speech was documented in the initial sound task. To account for this unusual pattern of results, we hypothesized that whenever possible, AE and TM adopt the strategy that takes advantage of the visual STM store to compensate for a defective phonological STS.

Disruptive effect of unattended noise-vocoded speech on recall of visually presented digits: Interaction between the number of frequency bands and languages

To assess the effects of degraded irrelevant speech on the serial recall of visually presented digits, noise-vocoded speech was generated in Japanese and German. Effects of the participants' native language were also examined by studying 40 Japanese and 40 German listeners. The number of frequency bands used in vocoding and the language (native or not) the irrelevant sound was derived from affected performance significantly. The participants' native language had a greater disruptive effect than the non-native language, particularly in conditions in which intelligibility was moderate. Speech sounds appear to have been processed automatically although the participant was instructed to neglect them. This must have required some amount of cognitive resources, which could have been used for the recall task otherwise. This automatic interference was stronger when the native language was used, probably because it contained perceptual cues that were more difficult to degrade.

Suppression of competing speech through entrainment of cortical oscillations

People are highly skilled at attending to one speaker in the presence of competitors, but the neural mechanisms supporting this remain unclear. Recent studies have argued that the auditory system enhances the gain of a speech stream relative to competitors by entraining (or "phase-locking") to the rhythmic structure in its acoustic envelope, thus ensuring that syllables arrive during periods of high neuronal excitability. We hypothesized that such a mechanism could also suppress a competing speech stream by ensuring that syllables arrive during periods of low neuronal excitability. To test this, we analyzed high-density EEG recorded from human adults while they attended to one of two competing, naturalistic speech streams. By calculating the cross-correlation between the EEG channels and the speech envelopes, we found evidence of entrainment to the attended speech's acoustic envelope as well as weaker yet significant entrainment to the unattended speech's envelope. An independent component analysis (ICA) decomposition of the data revealed sources in the posterior temporal cortices that displayed robust correlations to both the attended and unattended envelopes. Critically, in these components the signs of the correlations when attended were opposite those when unattended, consistent with the hypothesized entrainment-based suppressive mechanism.

Priming motivation through unattended speech

This study examines whether motivation can be primed through unattended speech. Study 1 used a dichotic-listening paradigm and repeated strength measures. In comparison to the baseline condition, in which the unattended channel was only composed by neutral words, the presence of words related to high (low) intensity of motivation led participants to exert more (less) strength when squeezing a hand dynamometer. In a second study, a barely audible conversation was played while participants' attention was mobilized on a demanding task. Participants who were exposed to a conversation depicting intrinsic motivation performed better and persevered longer in a subsequent word-fragment completion task than those exposed to the same conversation made unintelligible. These findings suggest that motivation can be primed without attention.

Attention effects on the processing of task-relevant and task-irrelevant speech sounds and letters

We used event-related brain potentials (ERPs) to study effects of selective attention on the processing of attended and unattended spoken syllables and letters. Participants were presented with syllables randomly occurring in the left or right ear and spoken by different voices and with a concurrent foveal stream of consonant letters written in darker or lighter fonts. During auditory phonological (AP) and non-phonological tasks, they responded to syllables in a designated ear starting with a vowel and spoken by female voices, respectively. These syllables occurred infrequently among standard syllables starting with a consonant and spoken by male voices. During visual phonological and non-phonological tasks, they responded to consonant letters with names starting with a vowel and to letters written in dark fonts, respectively. These letters occurred infrequently among standard letters with names starting with a consonant and written in light fonts. To examine genuine effects of attention and task on ERPs not overlapped by ERPs associated with target processing or deviance detection, these effects were studied only in ERPs to auditory and visual standards. During selective listening to syllables in a designated ear, ERPs to the attended syllables were negatively displaced during both phonological and non-phonological auditory tasks. Selective attention to letters elicited an early negative displacement and a subsequent positive displacement (Pd) of ERPs to attended letters being larger during the visual phonological than non-phonological task suggesting a higher demand for attention during the visual phonological task. Active suppression of unattended speech during the AP and non-phonological tasks and during the visual phonological tasks was suggested by a rejection positivity (RP) to unattended syllables. We also found evidence for suppression of the processing of task-irrelevant visual stimuli in visual ERPs during auditory tasks involving left-ear syllables.

Brain activity during auditory and visual phonological, spatial and simple discrimination tasks

We used functional magnetic resonance imaging to measure human brain activity during tasks demanding selective attention to auditory or visual stimuli delivered in concurrent streams. Auditory stimuli were syllables spoken by different voices and occurring in central or peripheral space. Visual stimuli were centrally or more peripherally presented letters in darker or lighter fonts. The participants performed a phonological, spatial or “simple” (speaker-gender or font-shade) discrimination task in either modality. Within each modality, we expected a clear distinction between brain activations related to nonspatial and spatial processing, as reported in previous studies. However, within each modality, different tasks activated largely overlapping areas in modality-specific (auditory and visual) cortices, as well as in the parietal and frontal brain regions. These overlaps may be due to effects of attention common for all three tasks within each modality or interaction of processing task-relevant features and varying task-irrelevant features in the attended-modality stimuli. Nevertheless, brain activations caused by auditory and visual phonological tasks overlapped in the left mid-lateral prefrontal cortex, while those caused by the auditory and visual spatial tasks overlapped in the inferior parietal cortex. These overlapping activations reveal areas of multimodal phonological and spatial processing. There was also some evidence for intermodal attention-related interaction. Most importantly, activity in the superior temporal sulcus elicited by unattended speech sounds was attenuated during the visual phonological task in comparison with the other visual tasks. This effect might be related to suppression of processing irrelevant speech presumably distracting the phonological task involving the letters.

The Impact of Synthetic and Accented Speech on Unattended Recall in a Dichotic Listening Task

Synthetic speech is a commonly used form of computer-generated speech. Synthetic speech is different than natural speech as it has a different pacing and lacks intonation. English natural speech with a foreign accent also has a different pacing and pronunciation than an American’s own natural accented speech. A study was performed to determine if unattended synthetic speech was more difficult to process than unattended natural speech when one is engaged in an activity that demands a great deal of attention. Participants engaged in a dichotic listening task where they wore headphones and repeated information from one ear (attended), while ignoring information from their other ear (unattended). It was found that the presence of natural speech in the unattended ear resulted in participants reporting more low-threshold information from the task, and that accent did not impact the amount reported. It was concluded that synthetic speech is inherently different than natural speech, as degrading the natural speech by including a foreign accent did not result in the same deficits as synthetic speech.

Discrepant Visual Speech Facilitates Covert Selective Listening in “Cocktail Party” Conditions

The presence of congruent visual speech information facilitates the identification of auditory speech, while the addition of incongruent visual speech information often impairs accuracy. This latter arrangement occurs naturally when one is being directly addressed in conversation but listens to a different speaker. Under these conditions, performance may diminish since: (a) one is bereft of the facilitative effects of the corresponding lip motion and (b) one becomes subject to visual distortion by incongruent visual speech; by contrast, speech intelligibility may be improved due to (c) bimodal localization of the central unattended stimulus. Participants were exposed to centrally presented visual and auditory speech while attending to a peripheral speech stream. In some trials, the lip movements of the central visual stimulus matched the unattended speech stream; in others, the lip movements matched the attended peripheral speech. Accuracy for the peripheral stimulus was nearly one standard deviation greater with incongruent visual information, compared to the congruent condition which provided bimodal pattern recognition cues. Likely, the bimodal localization of the central stimulus further differentiated the stimuli and thus facilitated intelligibility. Results are discussed with regard to similar findings in an investigation of the ventriloquist effect, and the relative strength of localization and speech cues in covert listening.

The Sound of Feelings: Electrophysiological Responses to Emotional Speech in Alexithymia

BackgroundAlexithymia is a personality trait characterized by difficulties in the cognitive processing of emotions (cognitive dimension) and in the experience of emotions (affective dimension). Previous research focused mainly on visual emotional processing in the cognitive alexithymia dimension. We investigated the impact of both alexithymia dimensions on electrophysiological responses to emotional speech in 60 female subjects.MethodologyDuring unattended processing, subjects watched a movie while an emotional prosody oddball paradigm was presented in the background. During attended processing, subjects detected deviants in emotional prosody. The cognitive alexithymia dimension was associated with a left-hemisphere bias during early stages of unattended emotional speech processing, and with generally reduced amplitudes of the late P3 component during attended processing. In contrast, the affective dimension did not modulate unattended emotional prosody perception, but was associated with reduced P3 amplitudes during attended processing particularly to emotional prosody spoken in high intensity.ConclusionsOur results provide evidence for a dissociable impact of the two alexithymia dimensions on electrophysiological responses during the attended and unattended processing of emotional prosody. The observed electrophysiological modulations are indicative of a reduced sensitivity to the emotional qualities of speech, which may be a contributing factor to problems in interpersonal communication associated with alexithymia.

Divided Listening in Noise in a Mock-up of a Military Command Post

This study investigated divided listening in noise in a mock-up of a vehicular command post. The effects of background noise from the vehicle, unattended speech of coworkers on speech understanding, and a visual cue that directed attention to the message source were examined. Sixteen normal-hearing males participated in sixteen listening conditions, defined by combinations of the absence/presence of vehicle and speech babble noises, availability of a vision cue, and number of channels (2 or 3, diotic or dichotic, and loudspeakers) over which concurrent series of call sign, color, and number phrases were presented. All wore a communications headset with integrated hearing protection. A computer keyboard was used to encode phrases beginning with an assigned call sign. Subjects achieved close to 100% correct phrase identification when presented over the headset (with or without vehicle noise) or over the loudspeakers, without vehicle noise. In contrast, the percentage correct phrase identification was significantly less by 30 to 35% when presented over loudspeakers with vehicle noise. Vehicle noise combined with babble noise decreased the accuracy by an additional 12% for dichotic listening. Vision cues increased phrase identification accuracy by 7% for diotic listening. Outcomes could be explained by the at-ear energy spectra of the speech and noise.

Neural coding of continuous speech in auditory cortex during monaural and dichotic listening

The cortical representation of the acoustic features of continuous speech is the foundation of speech perception. In this study, noninvasive magnetoencephalography (MEG) recordings are obtained from human subjects actively listening to spoken narratives, in both simple and cocktail party-like auditory scenes. By modeling how acoustic features of speech are encoded in ongoing MEG activity as a spectrotemporal response function, we demonstrate that the slow temporal modulations of speech in a broad spectral region are represented bilaterally in auditory cortex by a phase-locked temporal code. For speech presented monaurally to either ear, this phase-locked response is always more faithful in the right hemisphere, but with a shorter latency in the hemisphere contralateral to the stimulated ear. When different spoken narratives are presented to each ear simultaneously (dichotic listening), the resulting cortical neural activity precisely encodes the acoustic features of both of the spoken narratives, but slightly weakened and delayed compared with the monaural response. Critically, the early sensory response to the attended speech is considerably stronger than that to the unattended speech, demonstrating top-down attentional gain control. This attentional gain is substantial even during the subjects' very first exposure to the speech mixture and therefore largely independent of knowledge of the speech content. Together, these findings characterize how the spectrotemporal features of speech are encoded in human auditory cortex and establish a single-trial-based paradigm to study the neural basis underlying the cocktail party phenomenon.

Cortical neural coding of speech in simple and complex auditory scenes.

The neural representation of continuous speech in human auditory cortex was obtained noninvasively via magnetoencephalography. The neural response was recorded from human subjects listening to a spoken narrative, either in clean or in the presence of interfering speech. The cortical neural response to clean speech is demonstrated to precisely track the slow temporal modulations (<10 Hz) of speech in a broad spectral region between 400 Hz and 2 kHz. The neural code is sufficiently faithful to decode acoustic features of speech. To examine the robustness of, and the role of attention in, this neural code, another spoken narrative was presented simultaneously, either to a different ear (dichotically) or to the same ear (diotically), instructing the subjects to focus on only one of the two speech signals. The cortical representation of the attended speech is found to be substantially stronger than that of the unattended speech. This attentional effect is significant during the subjects’ first exposure to the spoken narratives. These results demonstrate that auditory cortex precisely represents the slow temporal modulations of speech and maintains separate neural representations for concurrent speech signals that can be individually and strongly modulated by attention. [Work supported by the NIDCD Grant No. R01‐DC‐008342.]

The focus of attention at the virtual cocktail party—Electrophysiological evidence

The width of the attentional focus during the selection of one of two concurrent normal human participants was investigated using event-related potentials. Two stories were presented from virtual locations located 15° to the left and right azimuth by convolving the speech message by the appropriate head-related transfer function determined for each individual participant. Task irrelevant probe stimuli (phoneme/da/uttered by the same speaker as the story) were presented in rapid sequence from the same virtual locations. Occasionally, probes were presented at locations 15 or 30° lateral of the standard probes. Probes coinciding with the attended message gave rise to a fronto-central negativity relative to the phoneme probes coinciding with the unattended speech message. This was similar to the typical ERP attention effect. On the attended side probes deviating from the standard location by 30° elicited a different type of negative response, tentatively identified as a reorienting negativity, whereas probes deviating by 15° did not. These results are taken to suggest that spatial information is used for message selection in a cocktail-party situation but that the focus of spatial attention is relatively wide.