Human Sound Localization Research Articles

The Effect of Training on Localizing HoloLens-Generated 3D Sound Sources.

Sound localization is a crucial aspect of human auditory perception. VR (virtual reality) technologies provide immersive audio platforms that allow human listeners to experience natural sounds based on their ability to localize sound. However, the simulations of sound generated by these platforms, which are based on the general head-related transfer function (HRTF), often lack accuracy in terms of individual sound perception and localization due to significant individual differences in this function. In this study, we aimed to investigate the disparities between the perceived locations of sound sources by users and the locations generated by the platform. Our goal was to determine if it is possible to train users to adapt to the platform-generated sound sources. We utilized the Microsoft HoloLens 2 virtual platform and collected data from 12 subjects based on six separate training sessions arranged in 2 weeks. We employed three modes of training to assess their effects on sound localization, in particular for studying the impacts of multimodal error, visual, and sound guidance in combination with kinesthetic/postural guidance, on the effectiveness of the training. We analyzed the collected data in terms of the training effect between pre- and post-sessions as well as the retention effect between two separate sessions based on subject-wise paired statistics. Our findings indicate that, as far as the training effect between pre- and post-sessions is concerned, the effect is proven to be statistically significant, in particular in the case wherein kinesthetic/postural guidance is mixed with visual and sound guidance. Conversely, visual error guidance alone was found to be largely ineffective. On the other hand, as far as the retention effect between two separate sessions is concerned, we could not find any meaningful statistical implication on the effect for all three error guidance modes out of the 2-week session of training. These findings can contribute to the improvement of VR technologies by ensuring they are designed to optimize human sound localization abilities.

Dynamic spectral cues do not affect human sound localization during small head movements.

Natural listening involves a constant deployment of small head movement. Spatial listening is facilitated by head movements, especially when resolving front-back confusions, an otherwise common issue during sound localization under head-still conditions. The present study investigated which acoustic cues are utilized by human listeners to localize sounds using small head movements (below ±10° around the center). Seven normal-hearing subjects participated in a sound localization experiment in a virtual reality environment. Four acoustic cue stimulus conditions were presented (full spectrum, flattened spectrum, frozen spectrum, free-field) under three movement conditions (no movement, head rotations over the yaw axis and over the pitch axis). Localization performance was assessed using three metrics: lateral and polar precision error and front-back confusion rate. Analysis through mixed-effects models showed that even small yaw rotations provide a remarkable decrease in front-back confusion rate, whereas pitch rotations did not show much of an effect. Furthermore, MSS cues improved localization performance even in the presence of dITD cues. However, performance was similar between stimuli with and without dMSS cues. This indicates that human listeners utilize the MSS cues before the head moves, but do not rely on dMSS cues to localize sounds when utilizing small head movements.

The pinna enhances angular discrimination in the frontal hemifield.

Human sound localization in the horizontal dimension is thought to be dominated by binaural cues, particularly interaural time delays, because monaural localization in this dimension is relatively poor. Remaining ambiguities of front versus back and up versus down are distinguished by high-frequency spectral cues generated by the pinna. The experiments in this study show that this account is incomplete. Using binaural listening throughout, the pinna substantially enhanced horizontal discrimination in the frontal hemifield, making discrimination in front better than discrimination at the rear, particularly for directions away from the median plane. Eliminating acoustic effects of the pinna by acoustically bypassing them or low-pass filtering abolished the advantage at the front without affecting the rear. Acoustic measurements revealed a pinna-induced spectral prominence that shifts smoothly in frequency as sounds move from 0° to 90° azimuth. The improved performance is discussed in terms of the monaural and binaural changes induced by the pinna.

Deep neural network models of sound localization reveal how perception is adapted to real-world environments

Mammals localize sounds using information from their two ears. Localization in real-world conditions is challenging, as echoes provide erroneous information, and noises mask parts of target sounds. To better understand real-world localization we equipped a deep neural network with human ears and trained it to localize sounds in a virtual environment. The resulting model localized accurately in realistic conditions with noise and reverberation. In simulated experiments, the model exhibited many features of human spatial hearing: sensitivity to monaural spectral cues and interaural time and level differences, integration across frequency, biases for sound onsets, and limits on localization of concurrent sources. But when trained in unnatural environments without either reverberation, noise, or natural sounds, these performance characteristics deviated from those of humans. The results show how biological hearing is adapted to the challenges of real-world environments and illustrate how artificial neural networks can reveal the real-world constraints that shape perception.

Odor-induced sound localization bias under unilateral intranasal trigeminal stimulation.

As a stereo odor cue, internostril odor influx could help us in many spatial tasks, including localization and navigation. Studies have also revealed that this benefit could be modulated by the asymmetric concentrations of both influxes (left nose vs right nose). The interaction between olfaction and vision, such as in object recognition and visual direction judgment, has been documented; however, little has been revealed about the impact of odor cues on sound localization. Here we adopted the ventriloquist paradigm in auditory-odor interactions and investigated sound localization with the concurrent unilateral odor influx. Specifically, we teased apart both the "nature" of the odors (pure olfactory stimulus vs. mixed olfactory/trigeminal stimulus) and the location of influx (left nose vs. right nose) and examined sound localization with the method of constant stimuli. Forty-one participants, who passed the Chinese Smell Identification Test, perceived sounds with different azimuths (0°, 5°, 10°, and 20° unilaterally deflected from the sagittal plane by head-related transfer function) and performed sound localization (leftward or rightward) tasks under concurrent, different unilateral odor influxes (10% v/v phenylethyl alcohol, PEA, as pure olfactory stimulus, 1% m/v menthol as mixed olfactory/trigeminal stimulus, and propylene glycol as the control). Meanwhile, they reported confidence levels of the judgments. Results suggested that unilateral PEA influx did not affect human sound localization judgments. However, unilateral menthol influx systematically biased the perceived sound localization, shifting toward the odor source. Our study provides evidence that unilateral odor influx could bias perceived sound localization only when the odor activates the trigeminal nerves.

Why Did the Earwitnesses to the John F. Kennedy Assassination Not Agree About the Location of the Gunman?

Earwitnesses to the 1963 assassination of President John F. Kennedy (JFK) did not agree about the location of the gunman even though their judgments about the number and timing of the gunshots were reasonably consistent. Even earwitnesses at the same general location disagreed. An examination of the acoustics of supersonic bullets and the characteristics of human sound localization help explain the general disagreement about the origin of the gunshots. The key fact is that a shock wave produced by the supersonic bullet arrived prior to the muzzle blast for many earwitnesses, and the shock wave provides erroneous information about the origin of the gunshot. During the government's official re-enactment of the JFK assassination in 1978, expert observers were highly accurate in localizing the origin of gunshots taken from either of two locations, but their supplementary observations help explain the absence of a consensus among the earwitnesses to the assassination itself.

Novel Position Estimation using Differential Timing Information for Asynchronous LTE/NR Networks

Positioning techniques have been a common objective since the early development of wireless networks. However, current positioning methods in cellular networks, for instance, are still primarily focused on the use of the Global Navigation Satellite System (GNSS), which has several limitations, like high power drainage and failure in indoor scenarios. This study introduces a novel approach employing standard LTE signaling in order to provide high accuracy positioning estimation. The proposed technique is designed in analogy to the human sound localization system, eliminating the need of having information from three spatially diverse Base Stations (BSs). This is inspired by the perfect human 3D sound localization with two ears. A field study is carried out in a dense urban city to verify the accuracy of the proposed technique, with more than 20 thousand measurement samples collected. The achieved positioning accuracy is meeting the latest Federal Communications Commission (FCC) requirements in the planner dimension.

Goal-driven, neurobiological-inspired convolutional neural network models of human spatial hearing

The human brain effortlessly solves the complex computational task of sound localization using a mixture of spatial cues. How the brain performs this task in naturalistic listening environments (e.g. with reverberation) is not well understood. In the present paper, we build on the success of deep neural networks at solving complex and high-dimensional problems [1] to develop goal-driven, neurobiological-inspired convolutional neural network (CNN) models of human spatial hearing. After training, we visualize and quantify feature representations in intermediate layers to gain insights into the representational mechanisms underlying sound location encoding in CNNs. Our results show that neurobiological-inspired CNN models trained on real-life sounds spatialized with human binaural hearing characteristics can accurately predict sound location in the horizontal plane. CNN localization acuity across the azimuth resembles human sound localization acuity, but CNN models outperform human sound localization in the back. Training models with different objective functions - that is, minimizing either Euclidean or angular distance - modulates localization acuity in particular ways. Moreover, different implementations of binaural integration result in unique patterns of localization errors that resemble behavioral observations in humans. Finally, feature representations reveal a gradient of spatial selectivity across network layers, starting with broad spatial representations in early layers and progressing to sparse, highly selective spatial representations in deeper layers. In sum, our results show that neurobiological-inspired CNNs are a valid approach to modeling human spatial hearing. This work paves the way for future studies combining neural network models with empirical measurements of neural activity to unravel the complex computational mechanisms underlying neural sound location encoding in the human auditory pathway.

A framework for designing head-related transfer function distance metrics that capture localization perception.

Linear comparisons can fail to describe perceptual differences between head-related transfer functions (HRTFs), reducing their utility for perceptual tests, HRTF selection methods, and prediction algorithms. This work introduces a machine learning framework for constructing a perceptual error metric that is aligned with performance in human sound localization. A neural network is first trained to predict measurement locations from a large database of HRTFs and then fine-tuned with perceptual data. It demonstrates robust model performance over a standard spectral difference error metric. A statistical test is employed to quantify the information gain from the perceptual observations as a function of space.

Population rate-coding predicts correctly that human sound localization depends on sound intensity.

Human sound localization is an important computation performed by the brain. Models of sound localization commonly assume that sound lateralization from interaural time differences is level invariant. Here we observe that two prevalent theories of sound localization make opposing predictions. The labelled-line model encodes location through tuned representations of spatial location and predicts that perceived direction is level invariant. In contrast, the hemispheric-difference model encodes location through spike-rate and predicts that perceived direction becomes medially biased at low sound levels. Here, behavioral experiments find that softer sounds are perceived closer to midline than louder sounds, favoring rate-coding models of human sound localization. Analogously, visual depth perception, which is based on interocular disparity, depends on the contrast of the target. The similar results in hearing and vision suggest that the brain may use a canonical computation of location: encoding perceived location through population spike rate relative to baseline.

Perceived Target Range Shapes Human Sound-Localization Behavior.

The auditory system relies on binaural differences and spectral pinna cues to localize sounds in azimuth and elevation. However, the acoustic input can be unreliable, due to uncertainty about the environment, and neural noise. A possible strategy to reduce sound-location uncertainty is to integrate the sensory observations with sensorimotor information from previous experience, to infer where sounds are more likely to occur. We investigated whether and how human sound localization performance is affected by the spatial distribution of target sounds, and changes thereof. We tested three different open-loop paradigms, in which we varied the spatial range of sounds in different ways. For the narrowest ranges, target-response gains were highly idiosyncratic and deviated from an optimal gain predicted by error-minimization; in the horizontal plane the deviation typically consisted of a response overshoot. Moreover, participants adjusted their behavior by rapidly adapting their gain to the target range, both in elevation and in azimuth, yielding behavior closer to optimal for larger target ranges. Notably, gain changes occurred without any exogenous feedback about performance. We discuss how the findings can be explained by a sub-optimal model in which the motor-control system reduces its response error across trials to within an acceptable range, rather than strictly minimizing the error.

A demonstration about human sound localization

All people can find from where a sound comes. In fact, the brain interprets the information that comes to the two ears based mainly on intensity and time difference between of the sound reception by the inner ear. Here we present a simple demonstration that shows those two cues. Using a dummy head with two microphones inserted in each ear we can observe the intensity as a function of time for both ears simultaneously.

Identification of perceptually relevant methods of inter-aural time difference estimation.

The inter-aural time difference (ITD) is a fundamental cue for human sound localization. Over the past decades several methods have been proposed for its estimation from measured head-related impulse response (HRIR) data. Nevertheless, inter-method variations in ITD calculation have been found to exceed the known just noticeable differences (JNDs), leading to possible perceptible artifacts in virtual binaural auditory scenes, when personalized HRIRs are being used. In the absence of an objective means for validating ITD estimations, this paper examines which methods lead to the most perceptually relevant results. A subjective lateralization study compared objective ITDs to perceptually evaluated inter-aural pure delay offsets. Results clearly indicate the first-onset threshold detection method, using a low relative threshold of -30 dB, applied on 3 kHz low-pass filtered HRIRs as consistently the most perceptually relevant procedure across various metrics. Several alternative threshold values and methods based on the maximum or centroid of the inter-aural cross correlation of similarly filtered HRIR or HRIR envelopes also provided reasonable results. On the contrary, phase-based methods employing the integrated relative group delay or auditory model were not found to perform as well.

Identifying a perceptually relevant estimation method of the inter-aural time delay

The Inter-aural Time Difference (ITD) is a fundamental cue for human sound localization. Over the past decades, several methods have been proposed for its estimation from measured Head-Related Impulse Response (HRIR) data. Nevertheless, inter-method variations in ITD calculation have been found to exceed the known Just Noticeable Differences (JNDs), hence leading to possible perceptible artifacts in virtual binaural auditory scenes, even for cases when personalized HRIRs are being used. In the absence of an objective means for validating ITD estimations, this paper evaluates which methods lead to the most perceptually relevant results. A subjective lateralization study compared objective ITDs to perceptually driven inter-aural pure delay offsets. Results clearly indicate the first-onset Threshold detection method, using a low relative threshold of -30 dB, applied on 3 kHz low-pass filtered HRIRs as the most perceptually relevant procedure across various metrics. Alternative threshold values and methods based on the maximum or centroid of the Inter-Aural Cross Correlation of similarly filtered HRIRs or HRIR envelopes also provided reasonable results. On the contrary, phase-based methods employing the Integrated Relative Group Delay were not found to perform as well.

Reconstructing spectral cues for sound localization from responses to rippled noise stimuli.

Human sound localization in the mid-saggital plane (elevation) relies on an analysis of the idiosyncratic spectral shape cues provided by the head and pinnae. However, because the actual free-field stimulus spectrum is a-priori unknown to the auditory system, the problem of extracting the elevation angle from the sensory spectrum is ill-posed. Here we test different spectral localization models by eliciting head movements toward broad-band noise stimuli with randomly shaped, rippled amplitude spectra emanating from a speaker at a fixed location, while varying the ripple bandwidth between 1.5 and 5.0 cycles/octave. Six listeners participated in the experiments. From the distributions of localization responses toward the individual stimuli, we estimated the listeners’ spectral-shape cues underlying their elevation percepts, by applying maximum-likelihood estimation. The reconstructed spectral cues resulted to be invariant to the considerable variation in ripple bandwidth, and for each listener they had a remarkable resemblance to the idiosyncratic head-related transfer functions (HRTFs). These results are not in line with models that rely on the detection of a single peak or notch in the amplitude spectrum, nor with a local analysis of first- and second-order spectral derivatives. Instead, our data support a model in which the auditory system performs a cross-correlation between the sensory input at the eardrum-auditory nerve, and stored representations of HRTF spectral shapes, to extract the perceived elevation angle.

Head movement in human sound localization

The aim of the present research is to optimize acoustic guide signals for visually impaired persons. The signal must be localized in the correct direction by the impaired person, since they aid their safe movement. Generally speaking, head movements increase localization accuracy of continuous sound. First, we clarified whether or not the effect of head movement appears in sound localization of intermittent sound in place of continuous one. Localization tests showed that effectiveness of head movements persist even if the sound is intermittent rather than continuous. However, investigations have not yet been carried out to make it clear whether or not spontaneous head movements occur at all during natural sound localization, although the previous studies were premised on the movements. If the movements always occur spontaneously, we could expect greater flexibility on designing acoustic guide signals. Second, we discussed directly whether or not spontaneous head movements occur during natural sound locali...

Integrated processing of spatial cues in human auditory cortex

Human sound source localization relies on acoustical cues, most importantly, the interaural differences in time and level (ITD and ILD). For reaching a unified representation of auditory space the auditory nervous system needs to combine the information provided by these two cues. In search for such a unified representation, we conducted a magnetoencephalography (MEG) experiment that took advantage of the location-specific adaptation of the auditory cortical N1 response. In general, the attenuation caused by a preceding adaptor sound to the response elicited by a probe depends on their spatial arrangement: if the two sounds coincide, adaptation is stronger than when the locations differ. Here, we presented adaptor–probe pairs that contained different localization cues, for instance, adaptors with ITD and probes with ILD. We found that the adaptation of the N1 amplitude was location-specific across localization cues. This result can be explained by the existence of auditory cortical neurons that are sensitive to sound source location independent on which cue, ITD or ILD, provides the location information. Such neurons would form a cue-independent, unified representation of auditory space in human auditory cortex.

Human cortical sensitivity to interaural time difference in high-frequency sounds

Human sound source localization relies on various acoustical cues one of the most important being the interaural time difference (ITD). ITD is best detected in the fine structure of low-frequency sounds but it may also contribute to spatial hearing at higher frequencies if extracted from the sound envelope. The human brain mechanisms related to this envelope ITD cue remain unexplored. Here, we tested the sensitivity of the human auditory cortex to envelope ITD in magnetoencephalography (MEG) recordings. We found two types of sensitivity to envelope ITD. First, the amplitude of the auditory cortical N1m response was smaller for zero envelope ITD than for long envelope ITDs corresponding to the sound being in opposite phase in the two ears. Second, the N1m response amplitude showed ITD-specific adaptation for both fine-structure and for envelope ITD. The auditory cortical sensitivity was weaker for envelope ITD in high-frequency sounds than for fine-structure ITD in low-frequency sounds but occurred within a range of ITDs that are encountered in natural conditions. Finally, the participants were briefly tested for their behavioral ability to detect envelope ITD. Interestingly, we found a correlation between the behavioral performance and the neural sensitivity to envelope ITD. In conclusion, our findings show that the human auditory cortex is sensitive to ITD in the envelope of high-frequency sounds and this sensitivity may have behavioral relevance.

Perceptual factors contribute more than acoustical factors to sound localization abilities with virtual sources.

Human sound localization abilities rely on binaural and spectral cues. Spectral cues arise from interactions between the sound wave and the listener's body (head-related transfer function, HRTF). Large individual differences were reported in localization abilities, even in young normal-hearing adults. Several studies have attempted to determine whether localization abilities depend mostly on acoustical cues or on perceptual processes involved in the analysis of these cues. These studies have yielded inconsistent findings, which could result from methodological issues. In this study, we measured sound localization performance with normal and modified acoustical cues (i.e., with individual and non-individual HRTFs, respectively) in 20 naïve listeners. Test conditions were chosen to address most methodological issues from past studies. Procedural training was provided prior to sound localization tests. The results showed no direct relationship between behavioral results and an acoustical metrics (spectral-shape prominence of individual HRTFs). Despite uncertainties due to technical issues with the normalization of the HRTFs, large acoustical differences between individual and non-individual HRTFs appeared to be needed to produce behavioral effects. A subset of 15 listeners then trained in the sound localization task with individual HRTFs. Training included either visual correct-answer feedback (for the test group) or no feedback (for the control group), and was assumed to elicit perceptual learning for the test group only. Few listeners from the control group, but most listeners from the test group, showed significant training-induced learning. For the test group, learning was related to pre-training performance (i.e., the poorer the pre-training performance, the greater the learning amount) and was retained after 1 month. The results are interpreted as being in favor of a larger contribution of perceptual factors than of acoustical factors to sound localization abilities with virtual sources.

Natural auditory scene statistics shapes human spatial hearing.

Human perception, cognition, and action are laced with seemingly arbitrary mappings. In particular, sound has a strong spatial connotation: Sounds are high and low, melodies rise and fall, and pitch systematically biases perceived sound elevation. The origins of such mappings are unknown. Are they the result of physiological constraints, do they reflect natural environmental statistics, or are they truly arbitrary? We recorded natural sounds from the environment, analyzed the elevation-dependent filtering of the outer ear, and measured frequency-dependent biases in human sound localization. We find that auditory scene statistics reveals a clear mapping between frequency and elevation. Perhaps more interestingly, this natural statistical mapping is tightly mirrored in both ear-filtering properties and in perceived sound location. This suggests that both sound localization behavior and ear anatomy are fine-tuned to the statistics of natural auditory scenes, likely providing the basis for the spatial connotation of human hearing.