Model Of Sound Localization Research Articles

Deep neural network models of sound localization reveal how perception is adapted to real-world environments

Mammals localize sounds using information from their two ears. Localization in real-world conditions is challenging, as echoes provide erroneous information, and noises mask parts of target sounds. To better understand real-world localization we equipped a deep neural network with human ears and trained it to localize sounds in a virtual environment. The resulting model localized accurately in realistic conditions with noise and reverberation. In simulated experiments, the model exhibited many features of human spatial hearing: sensitivity to monaural spectral cues and interaural time and level differences, integration across frequency, biases for sound onsets, and limits on localization of concurrent sources. But when trained in unnatural environments without either reverberation, noise, or natural sounds, these performance characteristics deviated from those of humans. The results show how biological hearing is adapted to the challenges of real-world environments and illustrate how artificial neural networks can reveal the real-world constraints that shape perception.

Measurements and models for tone localization by moving listeners

Listeners were given the challenging task of localizing a steady-state sine tone in a moderately reverberant room. Listeners attempted to optimize performance by moving during the nine-second tone interval. To discover listener localization strategies, the experiment acquired complete records of listener head motion (Polhemus head tracker) and the signals in the ear canals (Etymotic probe tubes). Experiment data were used to test two models of sound localization by moving listeners: According to the “Relative Null" model, listeners search for nulls in relevant interaural differences and use the momentary nose orientations where these nulls occur as indicators of source azimuth. According to the “Inferred Azimuth" model, listeners acquire a continuous, long-term record of inferred source location based on momentary head orientation combined with corresponding interaural differences. Models include different cue weighting assumptions and are evaluated by their ability to predict listener responses. The Relative Null model proves to be more accurate, but sometimes fails to make a prediction. The Inferred Azimuth model always makes multiple predictions, and often leads to further insights. These models are the most straightforward of numerous possibilities. Alternatives that weight the time course of listener cues, should be considered.

Physiological Diversity Influences Detection of Stimulus Envelope and Fine Structure in Neurons of the Medial Superior Olive.

The neurons of the medial superior olive (MSO) of mammals extract azimuthal information from the delays between sounds reaching the two ears [interaural time differences (ITDs)]. Traditionally, all models of sound localization have assumed that MSO neurons represent a single population of cells with specialized and homogeneous intrinsic and synaptic properties that enable the detection of synaptic coincidence on a timescale of tens to hundreds of microseconds. Here, using patch-clamp recordings from large populations of anatomically labeled neurons in brainstem slices from male and female Mongolian gerbils (Meriones unguiculatus), we show that MSO neurons are far more physiologically diverse than previously appreciated, with properties that depend regionally on cell position along the topographic map of frequency. Despite exhibiting a similar morphology, neurons in the MSO exhibit subthreshold oscillations of differing magnitudes that drive action potentials at rates between 100 and 800 Hz. These oscillations are driven primarily by voltage-gated sodium channels and are distinct from resonant properties derived from other active membrane properties. We show that graded differences in these and other physiological properties across the MSO neuron population enable the MSO to duplex the encoding of ITD information in both fast, submillisecond time-varying signals as well as in slower envelopes.SIGNIFICANCE STATEMENT Neurons in the medial superior olive (MSO) encode sound localization cues by detecting microsecond differences in the arrival times of inputs from the left and right ears, and it has been assumed that this computation is made possible by highly stereotyped structural and physiological specializations. Here we report using a large (>400) sample size in which MSO neurons show a strikingly large continuum of functional properties despite exhibiting similar morphologies. We demonstrate that subthreshold oscillations mediated by voltage-gated Na+ channels play a key role in conferring graded differences in firing properties. This functional diversity likely confers capabilities of processing both fast, submillisecond-scale synaptic activity (acoustic "fine structure"), and slow-rising envelope information that is found in amplitude-modulated sounds and speech patterns.

Population rate-coding predicts correctly that human sound localization depends on sound intensity.

Human sound localization is an important computation performed by the brain. Models of sound localization commonly assume that sound lateralization from interaural time differences is level invariant. Here we observe that two prevalent theories of sound localization make opposing predictions. The labelled-line model encodes location through tuned representations of spatial location and predicts that perceived direction is level invariant. In contrast, the hemispheric-difference model encodes location through spike-rate and predicts that perceived direction becomes medially biased at low sound levels. Here, behavioral experiments find that softer sounds are perceived closer to midline than louder sounds, favoring rate-coding models of human sound localization. Analogously, visual depth perception, which is based on interocular disparity, depends on the contrast of the target. The similar results in hearing and vision suggest that the brain may use a canonical computation of location: encoding perceived location through population spike rate relative to baseline.

A biomimetic 2D transistor for audiomorphic computing

In this article, we introduce a biomimetic audiomorphic device that captures the neurobiological architecture and computational map inside the auditory cortex of barn owl known for its exceptional hunting ability in complete darkness using auditory cues. The device consists of multiple split-gates with nanogaps on a semiconducting MoS2 channel connected to the source/drain contacts for imitating the spatial map of coincidence detector neurons and tunable RC circuits for imitating the interaural time delay neurons following the Jeffress model of sound localization. Furthermore, we use global back-gating capability to demonstrate neuroplasticity to capture behavioral and/or adaptation related changes in the barn owl. Finally, the virtual source model for current transport is combined with finite element COMSOL multiphysics simulations to explain and project the performance of the biomimetic audiomorphic device. We find that the precision of the biomimetic device can supersede the barn owl by orders of magnitude.

A Recommender System for Improving Median Plane Sound Localization Performance Based on a Nonlinear Representation of HRTFs

We propose a new method to improve median plane sound localization performance using a nonlinear representation of head-related transfer functions (HRTFs) and a recommender system. First, we reduce the dimensionality of an HRTF data set with multiple subjects using manifold learning in conjunction with a customized intersubject graph which takes into account relevant prior knowledge of HRTFs. Then, we use a sound localization model to estimate a subject’s localization performance in terms of polar error and quadrant error rate. These metrics are merged to form a single rating per HRTF pair that we feed into a recommender system. Finally, the recommender system takes the low-dimensional HRTF representation as well as the ratings obtained from the localization model to predict the best HRTF set, possibly constructed by mixing HRTFs from different individuals, that minimizes a subject’s localization error. The simulation results show that our method is capable of choosing a set of HRTFs that improves the median plane localization performance with respect to the mean localization performance using non-individualized HRTFs. Moreover, the localization performance achieved by our HRTF recommender system shows no significant difference to the localization performance observed with the best matching non-individualized HRTFs but with the advantage of not having to perform listening tests with all individuals’ HRTFs from the database.

A Bayesian computational basis for auditory selective attention using head rotation and the interaural time-difference cue.

The process of resolving mixtures of several sounds into their separate individual streams is known as auditory scene analysis and it remains a challenging task for computational systems. It is well-known that animals use binaural differences in arrival time and intensity at the two ears to find the arrival angle of sounds in the azimuthal plane, and this localization function has sometimes been considered sufficient to enable the un-mixing of complex scenes. However, the ability of such systems to resolve distinct sound sources in both space and frequency remains limited. The neural computations for detecting interaural time difference (ITD) have been well studied and have served as the inspiration for computational auditory scene analysis systems, however a crucial limitation of ITD models is that they produce ambiguous or “phantom” images in the scene. This has been thought to limit their usefulness at frequencies above about 1khz in humans. We present a simple Bayesian model and an implementation on a robot that uses ITD information recursively. The model makes use of head rotations to show that ITD information is sufficient to unambiguously resolve sound sources in both space and frequency. Contrary to commonly held assumptions about sound localization, we show that the ITD cue used with high-frequency sound can provide accurate and unambiguous localization and resolution of competing sounds. Our findings suggest that an “active hearing” approach could be useful in robotic systems that operate in natural, noisy settings. We also suggest that neurophysiological models of sound localization in animals could benefit from revision to include the influence of top-down memory and sensorimotor integration across head rotations.

Erratum: Modeling sound-source localization in sagittal planes for human listeners [J. Acoust. Soc. Am. 136, 791-802 (2014)

This erratum concerns Eq. (4) of the original article, which defines the distance metric of the comparison process of the sagittal-plane sound localization model. The distance metric was actually implemented as a mean absolute difference but was erroneously described as a L1-norm difference.

Interaction of Number Magnitude and Auditory Localization.

The interplay of perception and memory is very evident when we perceive and then recognize familiar stimuli. Conversely, information in long-term memory may also influence how a stimulus is perceived. Prior work on number cognition in the visual modality has shown that in Western number systems long-term memory for the magnitude of smaller numbers can influence performance involving the left side of space, while larger numbers have an influence toward the right. Here, we investigated in the auditory modality whether a related effect may bias the perception of sound location. Subjects (n = 28) used a swivel pointer to localize noise bursts presented from various azimuth positions. The noise bursts were preceded by a spoken number (1-9) or, as a nonsemantic control condition, numbers that were played in reverse. The relative constant error in noise localization (forward minus reversed speech) indicated a systematic shift in localization toward more central locations when the number was smaller and toward more peripheral positions when the preceding number magnitude was larger. These findings do not support the traditional left-right number mapping. Instead, the results may reflect an overlap between codes for number magnitude and codes for sound location as implemented by two channel models of sound localization, or possibly a categorical mapping stage of small versus large magnitudes.

The role of interaural level differences in the localization of low-frequency sine tones

Five human listeners reported the azimuthal locations of low-frequency sine tones presented in free field, either by a 180-degree loudspeaker array or by virtual reality. The virtual sources were synthesized using cross-talk cancellation based on signals continuously monitored in the listener's ear canals. The experiment tested the duplex model of sound localization, especially the role of interaural level differences (fixed ILD = 0, 6, or 12 dB) for frequencies of 750 Hz or less. Trials with real sources, baseline virtual sources, and virtual sources with manipulated ILDs were always combined in all experiments. Experiments showed that the interaural time differences (ITD) dominated most fixed opposing ILDs, as previously reported, only when interaural phase differences (IPD) were less than 90 degrees. When IPDs exceeded 90 degrees, the ITD lost its influence, and localization was dominated by ILDs. Localization judgments never followed IPDs across a 180-degree boundary, except when a zeroed ILD caused judgments to become chaotic. Within the 90-degree IPD limit, judgments for fixed ILD appeared to follow the ITD better than the IPD. Abnormal interaural conditions (e.g., ITDs and ILDs of opposite sign) led to a notable increase in front-back confusions. [Work supported by AFOSR grant FA9550-11-1-0101.]

Decoding Sound Source Location and Separation Using Neural Population Activity Patterns

The strategies by which the central nervous system decodes the properties of sensory stimuli, such as sound source location, from the responses of a population of neurons are a matter of debate. We show, using the average firing rates of neurons in the inferior colliculus (IC) of awake rabbits, that prevailing decoding models of sound localization (summed population activity and the population vector) fail to localize sources accurately due to heterogeneity in azimuth tuning across the population. In contrast, a maximum-likelihood decoder operating on the pattern of activity across the population of neurons in one IC accurately localized sound sources in the contralateral hemifield, consistent with lesion studies, and did so with a precision consistent with rabbit psychophysical performance. The pattern decoder also predicts behavior in response to incongruent localization cues consistent with the long-standing "duplex" theory of sound localization. We further show that the pattern decoder accurately distinguishes two concurrent, spatially separated sources from a single source, consistent with human behavior. Decoder detection of small amounts of source separation directly in front is due to neural sensitivity to the interaural decorrelation of sound, at both low and high frequencies. The distinct patterns of IC activity between single and separated sound sources thereby provide a neural correlate for the ability to segregate and localize sources in everyday, multisource environments.

Side peak suppression in responses of an across-frequency integration model to stimuli of varying bandwidth as demonstrated analytically and by implementation

Multiplication-like sound localization models are subjected to phase ambiguities for high-frequency tonal stimuli as multiplication creates several equivalent response peaks in tuning curves. By increasing the bandwidth of the stimulus, phase ambiguities can be reduced, which is often referred to as side peak suppression. In this study we present a Jeffress-based sound localization model, and determine side peak suppression analytically. The results were verified with an implementation of the same model, and compared to physiological data of barn owls. Three types of stimuli were analyzed: pure-tone stimuli, two-tone complexes with varying frequency distances, and noise signals with variable bandwidths. As an additional parameter we also determined the half-width of the main response peak to examine the scaling of tuning curves in azimuth. Results showed that side peak suppression did not only depend on bandwidth, but also on the center frequency and the distance of the side peak to the main response peak. In particular, the analytical model predicted that side peak suppression is a function of relative bandwidth, whereas half-width is inversely proportional to center frequency, with a proportionality factor depending on relative bandwidth. The analytical approach and the implementation yielded equivalent tuning curves (deviation < 1%). Moreover, the electrophysiological data recorded in barn owls closely matched the predicted tuning curves.

Threshold interaural time differences and the centroid model of sound localization

The centroid display model of sound lateralization hypothesizes an array of brain-stem cells with wide ranges of best frequencies (f) and best interaural time delays (ITD, τ). The cells are distributed according to function p(f,τ), and images are lateralized according to the centroid of an excitation pattern on this array, the rate-ITD function. The centroid display was tested by calculations using model cells for the medial superior olive, as the origin of the rate-ITD function. The cells had excitatory inputs, membrane potential increments, and time constants established by physiological measurements. Cells were driven with realistic frequency-dependent synchrony. The predictions were compared to the measured frequency dependence of ITD thresholds for human listeners. It was found that for high frequencies, 750 Hz and greater, the model could successfully account for the thresholds by making appropriate adjustments to p(f,τ). For lower frequencies, the model greatly underestimated thresholds for any reasonable p(f,τ) because integration over the wide range of τ reduced the variability in the rate-ITD function to unreasonable values. It is concluded that the centroid model fails to account for human thresholds at low frequency. [Work supported by the AFOSR and NIDCD.]

Spiking neural network model of sound localization using the interaural intensity difference.

In this paper, a spiking neural network (SNN) architecture to simulate the sound localization ability of the mammalian auditory pathways using the interaural intensity difference cue is presented. The lateral superior olive was the inspiration for the architecture, which required the integration of an auditory periphery (cochlea) model and a model of the medial nucleus of the trapezoid body. The SNN uses leaky integrate-and-fire excitatory and inhibitory spiking neurons, facilitating synapses and receptive fields. Experimentally derived head-related transfer function (HRTF) acoustical data from adult domestic cats were employed to train and validate the localization ability of the architecture, training used the supervised learning algorithm called the remote supervision method to determine the azimuthal angles. The experimental results demonstrate that the architecture performs best when it is localizing high-frequency sound data in agreement with the biology, and also shows a high degree of robustness when the HRTF acoustical data is corrupted by noise.

Neural Development of Binaural Tuning through Hebbian Learning Predicts Frequency-Dependent Best Delays

Birds use microsecond differences in the arrival times of the sounds at the two ears to infer the location of a sound source in the horizontal plane. These interaural time differences (ITDs) are encoded by binaural neurons which fire more when the ITD matches their "best delay." In the textbook model of sound localization, the best delays of binaural neurons reflect the differences in axonal delays of their monaural inputs, but recent observations have cast doubts on this classical view because best delays were found to depend on preferred frequency. Here, we show that these observations are in fact consistent with the notion that best delays are created by differences in axonal delays, provided ITD tuning is created during development through spike-timing-dependent plasticity: basilar membrane filtering results in correlations between inputs to binaural neurons, which impact the selection of synapses during development, leading to the observed distribution of best delays.

A functional spiking model of the ITD processing pathway of the barn owl

Sound localization in the barn owl relies on the early processing of two binaural cues, ITD (interaural time differences) and ILD (interaural level differences), that takes place in two parallel pathways in the auditory brainstem. Previous modeling studies fell into one of two classes, either they used biophysically plausible neural models on simplified binaural stimuli with artificially induced ITDs, or abstract cross-correlation models on realistic acoustical inputs. Here we present a functional spiking neural model of the ITD processing pathway of the barn owl, with a realistic virtual acoustic environment. The acoustical environment was reproduced by filtering natural sounds through measured Head-Related Transfer Functions (HRTFs) in the barn owl (Tyto Alba). Basilar membrane filtering was modeled with a bank of linear gammachirp filters and compression. Monaural neurons in the Nucleus Magnucellularis (NM) receive inputs from auditory nerve fibers and project axons to binaural neurons in the Nucleus Laminaris (NL), with various conduction delays, so that best delays depend on preferred frequency in accordance with measured distributions [1]. Empirical findings show that the best tuning of NL neurons is invariant to level and ILD changes [3], which implies that the spike timing of their monaural inputs should also be invariant to level. This is a challenging problem, since integration to a fixed threshold inevitably results in earlier spike timing when the input level is increased. We address this issue by using a spiking model with dynamic threshold [2,4]. We derive analytical conditions under which one can achieve this level invariance, and show through numerical simulations that the responses of our model to realistic inputs are indeed robust to level and ILD changes. The azimuth is estimated from the activation pattern of NL neurons. We found that the model could accurately estimate the azimuth of a large set of natural sounds with various levels, and significant background noise. Furthermore, we studied the robustness of ITD estimation in our model to reflective environments. We believe this study is a first step towards the development of neural models of sound localization in ecological environments.

Jitter Effect on the Performance of the Sound Localization Model of Medial Superior Olive Neural Circuit

Objectives: Additional properties of the stochastic neural circuit model suggested in [1] were studied. Methods: The performance of the whole circuit when the system employs a different jitter was studied by extensive simulations. By performance we mean the time needed to obtain a reliable estimate of ITD. Results: It was found that the relation between jitter and performance is nonlinear and we estimated a plausible range of jitter values for the model. Conclusion: To conclude, there exists an upper bound of the timing jitter since the number of neurons needed to compensate the injected noise grows exponentially and above certain jitter values becomes unrealistically high.

Mechanisms for Adjusting Interaural Time Differences to Achieve Binaural Coincidence Detection

Understanding binaural perception requires detailed analyses of the neural circuitry responsible for the computation of interaural time differences (ITDs). In the avian brainstem, this circuit consists of internal axonal delay lines innervating an array of coincidence detector neurons that encode external ITDs. Nucleus magnocellularis (NM) neurons project to the dorsal dendritic field of the ipsilateral nucleus laminaris (NL) and to the ventral field of the contralateral NL. Contralateral-projecting axons form a delay line system along a band of NL neurons. Binaural acoustic signals in the form of phase-locked action potentials from NM cells arrive at NL and establish a topographic map of sound source location along the azimuth. These pathways are assumed to represent a circuit similar to the Jeffress model of sound localization, establishing a place code along an isofrequency contour of NL. Three-dimensional measurements of axon lengths reveal major discrepancies with the current model; the temporal offset based on conduction length alone makes encoding of physiological ITDs impossible. However, axon diameter and distances between Nodes of Ranvier also influence signal propagation times along an axon. Our measurements of these parameters reveal that diameter and internode distance can compensate for the temporal offset inferred from axon lengths alone. Together with other recent studies, these unexpected results should inspire new thinking on the cellular biology, evolution, and plasticity of the circuitry underlying low-frequency sound localization in both birds and mammals.

Spike Transmission Delay at the Calyx of Held In Vivo: Rate Dependence, Phenomenological Modeling, and Relevance for Sound Localization

Transmission at central synapses exhibits rapid changes in response amplitude under different patterns of stimulation. Whether the delay associated with the transmission of action potentials is similarly modifiable is important for temporally precise computations. We address this question at the calyx of Held of the medial nucleus of the trapezoid body (MNTB) in Mongolian gerbils in vivo using extracellular recordings. Here the pre- and postsynaptic activity can be observed simultaneously, allowing the definition of an action potential transmission delay (ATD) from the pre- to the postsynaptic side. We find the ATD to increase as a function of spike rate (10-40%). The temporal dynamics of the ATD increase exhibit an exponential shape with activity-dependent time constants ( approximately 15-25 ms). Recovery dynamics of ATD were mono- (20-70 ms) or biexponential with fast (3-20 ms) and slow time constants (50-500 ms). Using a phenomenological model to capture ATD dynamics, we estimated DeltaATD = 5-30 micros per transmitted action potential. Using vocalizations and cage noise stimuli, we confirm that substantial changes in ATD occur in natural situations. Because the ATD changes cover the behaviorally relevant range of interaural time differences in gerbils, these results could provide constraints for models of sound localization.

A systems and neurobiological model of nucleus angularis and nucleus laminaris in the barn owl

The representation of binaural sound localization cues was investigated in a model of the barn owl’s auditory nerve, nucleus angularis (NA), nucleus magnocellularis (NM), nucleus laminaris (NL), and superior olivary nucleus (SON) using spiking leaky integrate and fire neurons. A phenomenological model of the barn owl’s auditory-nerve spike response was created to reproduce responses to tonal stimuli and exhibit two-tone rate suppression effects. The representation of stimulus level by NA was investigated by decoding the envelope of the input signal from populations of NA model neurons where the envelope was defined as in a previous computational model of sound localization in the barn owl [B.J. Fischer and C.H. Anderson, Adv. NIPS 16 (2004)]. Simulations demonstrated that inclusion of NA neurons with type IV responses increased the accuracy of the representation of the stimulus envelope. The effect of inhibition on the output of the population of NL model neurons was investigated and compared to a cross-correlation model. The output of the model SON provides inhibitory feedback to ipsilateral NA, NM, and NL model neurons. Simulations demonstrated that inclusion of inhibition of NL neurons by SON output increased peak–trough differences in ITD tuning curves.