Auditory Scene Analysis Research Articles

Auditory attention: listening to the soundscape of Between the Acts as a literary cocktail party

ABSTRACT The cocktail party problem refers to people’s auditory scene analysis in the complex auditory scene of everyday life in acoustics, which has also been addressed in modern literature. In this article, I define the literary cocktail party as a literary soundscape characterised by noise, with the dynamics of attention motivating its sound events and overall material relationality. Taking Virginia Woolf’s Between the Acts as an example, this article analyses how the interaction of words and sounds works with the dynamics of passive and active attention – two types of attention working in the bottom-up and top-down neural processes of the brain respectively. In different stages of the literary cocktail party, the development of sound events incorporates sonic materiality into the understanding of literary form, promotes the literary effect of defamiliarisation and creates relations in the soundscape. In the end, the article proposes ‘dip listening’ as a productive listening method for modern literary soundscape that is capable of not only generating an embodied sense of epiphany in daily experience but also creates the betweenness among various agencies in a shared tiredness.

Preliminary Evidence for Global Properties in Human Listeners During Natural Auditory Scene Perception.

Theories of auditory and visual scene analysis suggest the perception of scenes relies on the identification and segregation of objects within it, resembling a detail-oriented processing style. However, a more global process may occur while analyzing scenes, which has been evidenced in the visual domain. It is our understanding that a similar line of research has not been explored in the auditory domain; therefore, we evaluated the contributions of high-level global and low-level acoustic information to auditory scene perception. An additional aim was to increase the field's ecological validity by using and making available a new collection of high-quality auditory scenes. Participants rated scenes on 8 global properties (e.g., open vs. enclosed) and an acoustic analysis evaluated which low-level features predicted the ratings. We submitted the acoustic measures and average ratings of the global properties to separate exploratory factor analyses (EFAs). The EFA of the acoustic measures revealed a seven-factor structure explaining 57% of the variance in the data, while the EFA of the global property measures revealed a two-factor structure explaining 64% of the variance in the data. Regression analyses revealed each global property was predicted by at least one acoustic variable (R2 = 0.33-0.87). These findings were extended using deep neural network models where we examined correlations between human ratings of global properties and deep embeddings of two computational models: an object-based model and a scene-based model. The results support that participants' ratings are more strongly explained by a global analysis of the scene setting, though the relationship between scene perception and auditory perception is multifaceted, with differing correlation patterns evident between the two models. Taken together, our results provide evidence for the ability to perceive auditory scenes from a global perspective. Some of the acoustic measures predicted ratings of global scene perception, suggesting representations of auditory objects may be transformed through many stages of processing in the ventral auditory stream, similar to what has been proposed in the ventral visual stream. These findings and the open availability of our scene collection will make future studies on perception, attention, and memory for natural auditory scenes possible.

Development of auditory scene analysis: a mini-review.

Most auditory environments contain multiple sound waves that are mixed before reaching the ears. In such situations, listeners must disentangle individual sounds from the mixture, performing the auditory scene analysis. Analyzing complex auditory scenes relies on listeners ability to segregate acoustic events into different streams, and to selectively attend to the stream of interest. Both segregation and selective attention are known to be challenging for adults with normal hearing, and seem to be even more difficult for children. Here, we review the recent literature on the development of auditory scene analysis, presenting behavioral and neurophysiological results. In short, cognitive and neural mechanisms supporting stream segregation are functional from birth but keep developing until adolescence. Similarly, from 6 months of age, infants can orient their attention toward a target in the presence of distractors. However, selective auditory attention in the presence of interfering streams only reaches maturity in late childhood at the earliest. Methodological limitations are discussed, and a new paradigm is proposed to clarify the relationship between auditory scene analysis and speech perception in noise throughout development.

Acute Aromatase Inhibition Impairs Neural and Behavioral Auditory Scene Analysis in Zebra Finches.

Auditory perception can be significantly disrupted by noise. To discriminate sounds from noise, auditory scene analysis (ASA) extracts the functionally relevant sounds from acoustic input. The zebra finch communicates in noisy environments. Neurons in their secondary auditory pallial cortex (caudomedial nidopallium, NCM) can encode song from background chorus, or scenes, and this capacity may aid behavioral ASA. Furthermore, song processing is modulated by the rapid synthesis of neuroestrogens when hearing conspecific song. To examine whether neuroestrogens support neural and behavioral ASA in both sexes, we retrodialyzed fadrozole (aromatase inhibitor, FAD) and recorded in vivo awake extracellular NCM responses to songs and scenes. We found that FAD affected neural encoding of songs by decreasing responsiveness and timing reliability in inhibitory (narrow-spiking), but not in excitatory (broad-spiking) neurons. Congruently, FAD decreased neural encoding of songs in scenes for both cell types, particularly in females. Behaviorally, we trained birds using operant conditioning and tested their ability to detect songs in scenes after administering FAD orally or injected bilaterally into NCM. Oral FAD increased response bias and decreased correct rejections in females, but not in males. FAD in NCM did not affect performance. Thus, FAD in the NCM impaired neuronal ASA but that did not lead to behavioral disruption suggesting the existence of resilience or compensatory responses. Moreover, impaired performance after systemic FAD suggests involvement of other aromatase-rich networks outside the auditory pathway in ASA. This work highlights how transient estrogen synthesis disruption can modulate higher-order processing in an animal model of vocal communication.

Soundscape, attention and cognitive load

The cocktail party problem was first coined by E Colin Cherry in 1953 and it describes the archetypal challenge of listening in a complex soundscape (e.g., multiple talkers conversing vying for your attention in a crowded restaurant). In the past two decades, the field of psychoacoustics has steadily marched towards understanding how we listen in these naturalistic environments. Many studies have focused on studying the psychological and physiological aspects of auditory scene analysis and object-based attention as well as how these processes differ in typical listeners from others with listening difficulties. In recent years, there has also been a burgeoning interest in understanding how cognitive load and listening effort affect our ease of listening in different situational contexts (e.g., talking while driving). In this talk, a brief survey of modern psychoacoustic experimental approaches will be presented in the hope to spur new collaborative research ideas with those who study soundscape.

Generalization of auditory expertise in audio engineers and instrumental musicians

From auditory perception to general cognition, the ability to play a musical instrument has been associated with skills both related and unrelated to music. However, it is unclear if these effects are bound to the specific characteristics of musical instrument training, as little attention has been paid to other populations such as audio engineers and designers whose auditory expertise may match or surpass that of musicians in specific auditory tasks or more naturalistic acoustic scenarios. We explored this possibility by comparing students of audio engineering (n = 20) to matched conservatory-trained instrumentalists (n = 24) and to naive controls (n = 20) on measures of auditory discrimination, auditory scene analysis, and speech in noise perception. We found that audio engineers and performing musicians had generally lower psychophysical thresholds than controls, with pitch perception showing the largest effect size. Compared to controls, audio engineers could better memorise and recall auditory scenes composed of non-musical sounds, whereas instrumental musicians performed best in a sustained selective attention task with two competing streams of tones. Finally, in a diotic speech-in-babble task, musicians showed lower signal-to-noise-ratio thresholds than both controls and engineers; however, a follow-up online study did not replicate this musician advantage. We also observed differences in personality that might account for group-based self-selection biases. Overall, we showed that investigating a wider range of forms of auditory expertise can help us corroborate (or challenge) the specificity of the advantages previously associated with musical instrument training.

Dementia is an auditory brain disorder

AbstractBackgroundWhile impaired hearing has emerged as a major potentiating factor for dementia, the basis for this relationship has not been defined. Most studies have emphasised the role of peripheral hearing loss. However, we hear with our brains: natural listening places immense computational demands on neural circuitry, and the neural networks pre‐eminently targeted by major dementias are essential for decoding and responding appropriately to the auditory environment. Auditory brain dysfunction is therefore predicted both to contribute importantly to hearing impairment in Alzheimer’s and other dementias and to be integral to the expression of neurodegenerative pathologies. Here I present an overview of evidence that speaks to this hypothesis.MethodsIn a series of linked experiments, we have assessed aspects of auditory cognition – the processes whereby the brain perceives, interprets and acts upon sounds ‐ in patient cohorts with Alzheimer’s disease and major syndromes of frontotemporal dementia, relative to healthy older individuals. We designed novel psychoacoustic tests to assess aspects of auditory scene analysis (the ‘cocktail party effect’ and localisation of sounds in space), comprehension of acoustically degraded speech and the processing of semantic and emotional information in sound in cognitively impaired people. Neuroanatomical mechanisms were assessed using voxel‐based morphometry and functional MRI.ResultsAcross experiments, after taking peripheral hearing function and general cognitive performance into account, patients with Alzheimer’s disease and frontotemporal dementias showed characteristic ‘auditory phenotypes’ or profiles of auditory cognitive impairment. Alzheimer’s disease is associated with impaired auditory scene analysis, linked to atrophy and dysfunction of temporo‐parietal cortical regions in the core ‘default mode network’ targeted by Alzheimer pathology. Frontotemporal dementia syndromes produce distinct impairments of auditory feature analysis, sound recognition and emotional valuation.ConclusionsAuditory brain dysfunction plays a key role in everyday hearing impairment and reflects core pathophysiology in major dementias. These findings raise the exciting prospect that auditory cognitive measures might be used in the early diagnosis of dementia, in guiding novel hearing interventions and as dynamic markers of therapeutic response in the dawning era of disease modification.

Spectral Grouping of Electrically Encoded Sound Predicts Speech-in-Noise Performance in Cochlear Implantees

ObjectivesCochlear implant (CI) users exhibit large variability in understanding speech in noise. Past work in CI users found that spectral and temporal resolution correlates with speech-in-noise ability, but a large portion of variance remains unexplained. Recent work on normal-hearing listeners showed that the ability to group temporally and spectrally coherent tones in a complex auditory scene predicts speech-in-noise ability independently of the audiogram, highlighting a central mechanism for auditory scene analysis that contributes to speech-in-noise. The current study examined whether the auditory grouping ability also contributes to speech-in-noise understanding in CI users.DesignForty-seven post-lingually deafened CI users were tested with psychophysical measures of spectral and temporal resolution, a stochastic figure-ground task that depends on the detection of a figure by grouping multiple fixed frequency elements against a random background, and a sentence-in-noise measure. Multiple linear regression was used to predict sentence-in-noise performance from the other tasks.ResultsNo co-linearity was found between any predictor variables. All three predictors (spectral and temporal resolution plus the figure-ground task) exhibited significant contribution in the multiple linear regression model, indicating that the auditory grouping ability in a complex auditory scene explains a further proportion of variance in CI users’ speech-in-noise performance that was not explained by spectral and temporal resolution.ConclusionMeasures of cross-frequency grouping reflect an auditory cognitive mechanism that determines speech-in-noise understanding independently of cochlear function. Such measures are easily implemented clinically as predictors of CI success and suggest potential strategies for rehabilitation based on training with non-speech stimuli.

Sound Event Detection utilizing Spectro-Temporal Receptive Field

Sound event detection (SED) is a task of recognizing the target sound events with their respective onset and offset time in a recorded audio clip. SED is closely related to auditory scene analysis, the process by which human auditory systems perceive sound into perceptually meaningful components. While human auditory system is trained to perform SED well enough, the way sound is transformed into neural response within human auditory system still remain as the subject of ongoing research. One of efforts to describe the relationship between sound and neural response is the spectro-temporal receptive field (STRF). STRF acts as a linear function between sound time-frequency representation and primary auditory cortex (A1) cell response, so that neural response can be predicted by convolution of time-frequency representation of sound and STRF. In addition, STRF is designed to capture spectro-temporal modulation in that A1 cells are much reactive to spectro-temporally modulated ripple. In this work, we utilized STRF as a kernel of the first layer in the convolutional neural network (CNN) to extract neural response from input audio clip to make SED model similar to human auditory system. Then, we constructed two-branched SED model named as Two Branch STRF-Net (TB-STRFNet) composed of STRF branch and basic branch. The STRF branch extracts spectro-temporal modulation information and the basic branch extracts detailed and complex time-frequency information. TB-STRFNet outperformed the baseline by 4.3% in terms of main metric in DCASE 2023 Task 4 subtask B.

Development of an adaptive test of musical scene analysis abilities for normal-hearing and hearing-impaired listeners

Auditory scene analysis (ASA) is the process through which the auditory system makes sense of complex acoustic environments by organising sound mixtures into meaningful events and streams. Although music psychology has acknowledged the fundamental role of ASA in shaping music perception, no efficient test to quantify listeners’ ASA abilities in realistic musical scenarios has yet been published. This study presents a new tool for testing ASA abilities in the context of music, suitable for both normal-hearing (NH) and hearing-impaired (HI) individuals: the adaptive Musical Scene Analysis (MSA) test. The test uses a simple ‘yes–no’ task paradigm to determine whether the sound from a single target instrument is heard in a mixture of popular music. During the online calibration phase, 525 NH and 131 HI listeners were recruited. The level ratio between the target instrument and the mixture, choice of target instrument, and number of instruments in the mixture were found to be important factors affecting item difficulty, whereas the influence of the stereo width (induced by inter-aural level differences) only had a minor effect. Based on a Bayesian logistic mixed-effects model, an adaptive version of the MSA test was developed. In a subsequent validation experiment with 74 listeners (20 HI), MSA scores showed acceptable test–retest reliability and moderate correlations with other music-related tests, pure-tone-average audiograms, age, musical sophistication, and working memory capacities. The MSA test is a user-friendly and efficient open-source tool for evaluating musical ASA abilities and is suitable for profiling the effects of hearing impairment on music perception.

Interactions between attention and auditory scene analysis

Do auditory objects form automatically, or only when a listener focuses attention on a sound? This review argues that the question is ill posed—that both are simultaneously correct. Intuition suggests that the processes of source segregation and attention happen sequentially: first segregation parses a complex scene into constituent objects, and then selection pulls out an important sound to be analyzed in detail. But, rather than a processing hierarchy in which object formation occurs first, followed by selection, auditory scene analysis and attention interact in a heterarchy: formation and selection influence one another, feed back upon each other, and are not easily separable in terms of either how they are implemented in the brain or how their effects can be measured. There is not one particular site in the auditory pathway where objects “first appear;” instead, an object-based representation emerges gradually and imperfectly from the auditory nerve through the brainstem and midbrain to the various divisions of the cortex. Similarly, attentional selection accrues at every stage as one moves from the ear to the cortex. This talk will review examples from perceptual and physiological experiments supporting the idea that attention and object formation interact heterarchical.

The effect of angular disparity between sound source sequences on the perceptual organization of auditory objects

Within auditory scene analysis is a well-known streaming phenomenon which is observed when listeners group several sounds into “auditory objects” in complex auditory scenes. When two auditory sequences consist of similar sounds, these sequences tend to be perceived as a single stream, but can form segregated streams when the sounds are presented from disparate angles. This study investigated the tolerance thresholds for the angular disparity at which the sound sources would segregate rather than group together into single auditory objects, when 50-ms one-third octave bands of noise were presented in the horizontal plane with changing azimuth angle. Listeners were asked to judge whether the presented noise sequences were perceived as a single stream or not. The results revealed that listeners tend to perceive the presented sequences as a single stream when the angular disparity between noises was small; however, these tolerance thresholds depended on the spatial region within which the sequences were presented. Tolerance thresholds became large when the noise sequences were presented from the side, in comparison to the situation when sequences were presented from in front of the listeners. Results were consistent with the minimum audible angle on the horizontal plane.

Effects of age-related cochlear deafferentation and central gain on auditory scene analysis

Animal models show that cochlear afferent nerve endings are more vulnerable than sensory hair cells to age-related damage. Because such cochlear deafferentation is not apparent in standard audiometry, the extent to which it contributes to deficits in human hearing is debated, and the intervening neural processes are poorly characterized. This presentation will describe our efforts to address these gaps through co-ordinated experiments in at-risk humans and a chinchilla model. Our results suggest that cochlear deafferentation is widespread in middle age despite clinically normal audiometric sensitivity. Furthermore, the resulting reduction in peripheral input appears to trigger compensatory central “gain” at the cortical level, likely through altered local balance of excitation and inhibition. Consistent with the important role of inhibition in parsing temporal regularities across different frequency components of sound, central gain is also associated with reduced ability to perceptually segregate acoustic scenes with multiple sources into individual perceptual streams. Taken together, our results suggest that age-related cochlear deafferentation may affect hearing not only by reducing the fidelity of input encoding but also by interfering with the central auditory system’s ability to extract targets in noisy environments.

Towards audiovisual scene analysis and object formation

The neural mechanisms underpinning auditory scene analysis and object formation have been of intense research interest in the past two decades. Fundamentally, however, we live in a multisensory environment. Even Cherry in his original paper posited that “lip reading” as a way for us to solve the cocktail party problem. Yet, how different aspects of visual cues (e.g., timing, linguistic information) help listeners follow conversation in a complex acoustic scene is still not well understood. In this talk, we present a theoretical framework to study audiovisual scene analysis that has been extrapolated from the unisensory object-based attention literature and posit the following questions: How do we define a multi-modal object? What are the predictions from unisensory object-based attention theory when we apply to the audiovisual domain? What are the conceptual models to test the different neural mechanisms that underpin audiovisual scene analysis? Answering these questions would move us closer to addressing the cocktail party problem in the real-world setting as well as help us create, de novo, audiovisual scenes that are more engaging in the augmented/virtual reality world.

Development of speech perception in noise: Neural correlates of stream segregation in children and adolescents

In noisy backgrounds, listeners perform the auditory scene analysis: they parse the different auditory streams (“stream segregation”), and selectively focus on one stream as it unfolds over time (“selective attention”). Auditory scene analysis is thought to mature slowly. Here, we sought to investigate developmental changes in the neural signature of auditory stream segregation. Children (n = 17, aged 8 to 17) and young adults (n = 10) were presented with sequences of sounds consisting in stochastic variations of figures and backgrounds. These figure-ground sequences have been shown to elicit distinct “signature” EEG responses, including the object-related negativity (ORN) and P400, which reflect the processing concurrent auditory objects. Participants were also presented with a consonant identification task, where the consonant was presented in quiet, in the presence of one interfering talker, and in the presence of speech-shaped-noise. Preliminary results indicate a developmental effect on figure-ground segregation. Interestingly, we observe a significant correlation (r = 0.45) between stream segregation and speech perception in noise. However, we do not observe a developmental effect on the ORN/P400. Results will be discussed in light with the literature on the topic of auditory scene analysis and the development of the central auditory pathways.

Perception of global properties, objects, and settings in natural auditory scenes

Theories of auditory scene analysis suggest our perception of scenes relies on identifying and segregating objects within it. However, a more global process may occur while analyzing scenes, which has been evidenced in the visual domain. In our first experiment, we studied perception of eight global properties (e.g., openness), using a collection of 200 high-quality auditory scenes. Participants showed high agreement on their ratings of global properties. The global properties were explained by a two-factor model. Acoustic features of scenes were explained by a seven-factor model, and linearly predicted the global ratings by different amounts (R-squared = 0.33–0.87), although we also observed nonlinear relationships between acoustical and global variables. A multi-layer neural network trained to recognize auditory objects in everyday soundscapes from YouTube shows high-level embeddings of our 200 scenes are correlated with some global variables at earlier stages of processing than others. In a second experiment, we evaluated participants’ accuracy in identifying the setting of and objects within scenes across three durations (1, 2, and 4 s). Overall, participants performed better on the object identification task, but needed longer duration stimuli to do so. These results suggest object identification may require more processing time and/or attention switching than setting identification.

Latencies of click-evoked auditory responses in a harbor porpoise exceed the time interval between subsequent echolocation clicks.

Most auditory evoked potential (AEP) studies in echolocating toothed whales measure neural responses to outgoing clicks and returning echoes using short-latency auditory brainstem responses (ABRs) arising a few ms after acoustic stimuli. However, little is known about longer-latency cortical AEPs despite their relevance for understanding echo processing and auditory stream segregation. Here, we used a non-invasive AEP setup with low click repetition rates on a trained harbor porpoise to test the long-standing hypothesis that echo information from distant targets is completely processed before the next click is emitted. We reject this hypothesis by finding reliable click-related AEP peaks with latencies of 90 and 160 ms, which are longer than 99% of click intervals used by echolocating porpoises, demonstrating that some higher-order echo processing continues well after the next click emission even during slow clicking. We propose that some of the echo information, such as range to evasive prey, is used to guide vocal-motor responses within 50-100 ms, but that information used for discrimination and auditory scene analysis is processed more slowly, integrating information over many click-echo pairs. We conclude by showing theoretically that the identified long-latency AEPs may enable hearing sensitivity measurements at frequencies ten times lower than current ABR methods.

Selective auditory attention modulates cortical responses to sound location change for speech in quiet and in babble.

Listeners use the spatial location or change in spatial location of coherent acoustic cues to aid in auditory object formation. From stimulus-evoked onset responses in normal-hearing listeners using electroencephalography (EEG), we have previously shown measurable tuning to stimuli changing location in quiet, revealing a potential window into the cortical representations of auditory scene analysis. These earlier studies used non-fluctuating, spectrally narrow stimuli, so it was still unknown whether previous observations would translate to speech stimuli, and whether responses would be preserved for stimuli in the presence of background maskers. To examine the effects that selective auditory attention and interferers have on object formation, we measured cortical responses to speech changing location in the free field with and without background babble (+6 dB SNR) during both passive and active conditions. Active conditions required listeners to respond to the onset of the speech stream when it occurred at a new location, explicitly indicating 'yes' or 'no' to whether the stimulus occurred at a block-specific location either 30 degrees to the left or right of midline. In the aggregate, results show similar evoked responses to speech stimuli changing location in quiet compared to babble background. However, the effect of the two background environments diverges somewhat when considering the magnitude and direction of the location change and where the subject was attending. In quiet, attention to the right hemifield appeared to evoke a stronger response than attention to the left hemifield when speech shifted in the rightward direction. No such difference was found in babble conditions. Therefore, consistent with challenges associated with cocktail party listening, directed spatial attention could be compromised in the presence of stimulus noise and likely leads to poorer use of spatial cues in auditory streaming.

PITCH ESTIMATION FRAMEWORK FOR SPEECH SEGREGATION USING COCHLEAGRAM MORPHING

Computational auditory scene analysis (CASA) has significant role in speech segregation from monaural audio mixtures and generally a measure for performance of speech recognition systems. Pitch estimation has a substantial role in performance of CASA systems. This study presents a novel pitch estimation framework for speech segregation from monaural audio mixtures using cochleagram morphing. The proposed framework takes the rough estimation of target pitch from given audio mixtures containing speech and background interferences. Discrete set consisting morphed versions of cochleagram is obtained using k-Means clustering. The estimated pitch values are improved by validating and smoothing them to morphed cochleagram. Measure of refined estimated pitch contours along with harmonicity and temporal continuity are used to segregate target speech. The proposed framework produced 83.13% accuracy for MIR-1k dataset which is considerably higher than the existing methods.

Neuroimaging evidence for the direct role of auditory scene analysis in object perception.

Auditory Scene Analysis (ASA) refers to the grouping of acoustic signals into auditory objects. Previously, we have shown that perceived musicality of auditory sequences varies with high-level organizational features. Here, we explore the neural mechanisms mediating ASA and auditory object perception. Participants performed musicality judgments on randomly generated pure-tone sequences and manipulated versions of each sequence containing low-level changes (amplitude; timbre). Low-level manipulations affected auditory object perception as evidenced by changes in musicality ratings. fMRI was used to measure neural activation to sequences rated most and least musical, and the altered versions of each sequence. Next, we generated two partially overlapping networks: (i) a music processing network (music localizer) and (ii) an ASA network (base sequences vs. ASA manipulated sequences). Using Representational Similarity Analysis, we correlated the functional profiles of each ROI to a model generated from behavioral musicality ratings as well as models corresponding to low-level feature processing and music perception. Within overlapping regions, areas near primary auditory cortex correlated with low-level ASA models, whereas right IPS was correlated with musicality ratings. Shared neural mechanisms that correlate with behavior and underlie both ASA and music perception suggests that low-level features of auditory stimuli play a role in auditory object perception.