Speaker Localization Research Articles

Simultaneous Identification and Localization of Still and Mobile Speakers Based on Binaural Robot Audition

[abstFig src='/00290001/06.jpg' width='300' text='Efficient mobile speaker tracking' ] This paper jointly addresses the tasks of speaker identification and localization with binaural signals. The proposed system operates in noisy and echoic environments and involves limited computations. It demonstrates that a simultaneous identification and localization operation can benefit from a common signal processing front end for feature extraction. Moreover, a joint exploitation of the identity and position estimation outputs allows the outputs to limit each other’s errors. Equivalent rectangular bandwidth frequency cepstral coefficients (ERBFCC) and interaural level differences (ILD) are extracted. These acoustic features are respectively used for speaker identity and azimuth estimation through artificial neural networks (ANNs). The system was evaluated in simulated and real environments, with still and mobile speakers. Results demonstrate its ability to produce accurate estimations in the presence of noises and reflections. Moreover, the advantage of the binaural context over the monaural context for speaker identification is shown.

Toward an audiovisual attention model for multimodal video content

Visual attention modeling is a very active research field and several image and video attention models have been proposed during the last decade. However, despite the conclusions drawn from various studies about the influence of human gazes by the presence of sound, most of the classical video attention models do not account for the multimodal nature of video (visual and auditory cues). In this paper, we propose an audiovisual saliency model with the aim to predict human gaze maps when exploring video content. The model, intended for videoconferencing, is based on the fusion of spatial, temporal and auditory attentional maps. Based on a real-time audiovisual speaker localization approach, the proposed auditory map is modulated depending of the nature of faces in the video, i.e. speaker or auditor. State-of-the-art performance measures have been used to compare the predicted saliency maps with the eye-tracking ground truth. The obtained results show the very good performance of the proposed model and a significant improvement compared to non-audio models.

Speaker Diarization and Linking of Meeting Data

Finding who spoke when in a collection of recordings, with speakers being uniquely identified across the database, is a challenging task. In this scenario, reasonable computing times and acoustic variation across recordings remain two major concerns to address in state-of-the-art speaker diarization systems. This paper extends prior work on diarizing large speech datasets using algorithms that scale well with increasing amounts of data while compensating for across-recording variability. We follow a two-stage approach performing speaker diarization and speaker linking, the former focusing on local within-recording speaker changes and the latter focusing on global speaker changes across the database. In this study, we explore how these two modules interact with each other, while proposing a diarization fusion approach that prevents diarization errors from propagating to the linking stage. We further explore the diarization fusion for speaker linking using different linking strategies and speaker modeling variants. Evaluation is performed on single distant microphone data from the augmented multiparty interaction corpus show the effectiveness of the fusion approach after speaker linking and intersession variability modeling via joint factor analysis.

Horseshoe bat inspired reception dynamics embed dynamic features into speech signals

Horseshoe bats can alter the shape of noseleaves and outer ears (pinnae) at the time of emission and reception of biosonar pulses. The shape changes are a result of specific muscular action and significantly change the emission and reception beams by a spatial redistribution of energy in mainlobe and sidelobes. Here, we explore the potential of embedding dynamic features into the speech signals by means of bat inspired reception dynamics to improve the speech recognition accuracy and speaker localization. For this analysis, a digit dataset with 11 U.S. English speakers was recorded through a robotic setup that mimics pinna dynamics as observed in bats. The utterances were obtained through open-source speech corpora and amplitude modulated (carrier frequency: 55 kHz) to shift to ultrasonic band. These utterances were then re-recorded through the dynamic pinna set up and later demodulated to extract the original signal. It was found that reception dynamics have been able to enrich the speech signals with dynamic, direction-dependent time-frequency patterns. The next step in this research is to assess whether these patterns can be utilized to improve speech recognition and speaker localization.

Human electrophysiology of endogenous auditory spatial attention

Whereas perceiving the location of a stimulus in space relies on unique bottom-up processing, focusing attention to that location requires the availability and use of separate higher-level processes. It is unclear from previous work how selective attention might mediate and possibly sharpen spatial tuning, as behavioral tasks often lack an unattended comparison, and electrophysiological tasks often lack an attended comparison. We measured evoked responses to changes in the location of continuous noise in the free field when young, normal-hearing listeners were either passively listening or explicitly attending to one speaker location. Stimuli were presented from 1 of 5 frontal loudspeaker locations in the horizontal plane for 1.6 s before switching to another loudspeaker without pause. To ensure active attention for certain blocks, listeners were directed to press a button at the onset of a sound presented from the target location. Results show evidence of both facilitation for attended locations and suppression for unattended locations when compared to passive listening, and hemispheric distribution of activity was hemifield specific. Together, the data provide electrophysiological measures of a listener’s left-right orientation and another step toward cortical models of auditory spatial attention.

Semi-Supervised Sound Source Localization Based on Manifold Regularization

Conventional speaker localization algorithms, based merely on the received microphone signals, are often sensitive to adverse conditions, such as: high reverberation or low signal-to-noise ratio (SNR). In some scenarios, e.g., in meeting rooms or cars, it can be assumed that the source position is confined to a predefined area, and the acoustic parameters of the environment are approximately fixed. Such scenarios give rise to the assumption that the acoustic samples from the region of interest have a distinct geometrical structure. In this paper, we show that the high-dimensional acoustic samples indeed lie on a low-dimensional manifold and can be embedded into a low-dimensional space. Motivated by this result, we propose a semi-supervised source localization algorithm based on two-microphone measurements, which recovers the inverse mapping between the acoustic samples and their corresponding locations. The idea is to use an optimization framework based on manifold regularization, that involves smoothness constraints of possible solutions with respect to the manifold. The proposed algorithm, termed manifold regularization for localization, is adapted while new unlabelled measurements (from unknown source locations) are accumulated during runtime. Experimental results show superior localization performance when compared with a recently presented algorithm based on a manifold learning approach and with the generalized cross-correlation algorithm as a baseline. The algorithm achieves $2^\circ$ accuracy in typical noisy and reverberant environments (reverberation time between $200$ and $800$ ms and SNR between $5$ and $20$ dB).

Acoustic Microphone Geometry Calibration: An overview and experimental evaluation of state-of-the-art algorithms

Today, we are often surrounded by devices with one or more microphones, such as smartphones, laptops, and wireless microphones. If they are part of an acoustic sensor network, their distribution in the environment can be beneficially exploited for various speech processing tasks. However, applications like speaker localization, speaker tracking, and speech enhancement by beamforming avail themselves of the geometrical configuration of the sensors. Therefore, acoustic microphone geometry calibration has recently become a very active field of research. This article provides an application-oriented, comprehensive survey of existing methods for microphone position self-calibration, which will be categorized by the measurements they use and the scenarios they can calibrate. Selected methods will be evaluated comparatively with real-world recordings.

Benefit of contralateral routing of signals for unilateral cochlear implant users.

One way to improve speech understanding in noise for HI with a unilateral hearing loss is by contralateral routing of signals (CROS). Such a CROS-system captures sounds with an additional microphone at the worst hearing ear and transmits these to the better one. The better ear is then provided with a mix of signals originating from both ears. The goal of this study is to quantify the effect of a CROS-system on speech reception thresholds (SRTs) with unilaterally implanted CI-users in diffuse and directional noise as a function of speaker location. Listening tests are performed and an accurate directional intelligibly model is proposed used for further analysis. In diffuse noise it is concluded that the use of a CROS system results in a maximum gain in SRT of 7.9 dB when speech comes from the CROS side compared to a maximum loss in SRT of 2.1 dB when speech comes from the implanted side. In the case of directional noise, the effect of the CROS is symmetric and the maximum loss or gain in SRT was around 9 dB. The study therefore shows a clear potential of using the CROS system in diffuse noise.

Brain correlates of the orientation of auditory spatial attention onto speaker location in a "cocktail-party" situation.

Successful speech perception in complex auditory scenes with multiple competing speakers requires spatial segregation of auditory streams into perceptually distinct and coherent auditory objects and focusing of attention toward the speaker of interest. Here, we focused on the neural basis of this remarkable capacity of the human auditory system and investigated the spatiotemporal sequence of neural activity within the cortical network engaged in solving the "cocktail-party" problem. Twenty-eight subjects localized a target word in the presence of three competing sound sources. The analysis of the ERPs revealed an anterior contralateral subcomponent of the N2 (N2ac), computed as the difference waveform for targets to the left minus targets to the right. The N2ac peaked at about 500 ms after stimulus onset, and its amplitude was correlated with better localization performance. Cortical source localization for the contrast of left versus right targets at the time of the N2ac revealed a maximum in the region around left superior frontal sulcus and frontal eye field, both of which are known to be involved in processing of auditory spatial information. In addition, a posterior-contralateral late positive subcomponent (LPCpc) occurred at a latency of about 700 ms. Both these subcomponents are potential correlates of allocation of spatial attention to the target under cocktail-party conditions.

Performance of speaker localization using microphone array

Speaker localization is a technique to locate and track an active speaker from multiple acoustic sources using microphone array. Microphone array is used to improve the speech quality of recorded speech signal in meeting room and other places. In this work, the time delay estimation between source and each microphone is calculated using a localization method called time differences of arrival (TDOA). TDOA localization consists of two steps namely (a) a time delay estimator and (b) a localization estimator. For time delay estimation, the generalized cross-correlation using phase transform, the generalized cross correlation using maximum likelihood, linear prediction (LP) residual and the Hilbert envelope of the LP residual are chosen for estimating the location of a person. A new speaker localization algorithm known as group search optimization (GSO) algorithm is proposed. The performance of this algorithm is analyzed and compared with Gauss–Newton nonlinear least square method and genetic algorithm. Experimental results show that the proposed GSO method outperforms the other methods in terms of mean square error, root mean square error, mean absolute error, mean absolute percentage error, euclidean distance and mean absolute relative error.

Effects of low speed wind on the recognition/identification and pass-through communication tasks of auditory situation awareness afforded by military hearing protection/enhancement devices and tactical communication and protective systems

Objective: To investigate the effect of controlled low-speed wind-noise on the auditory situation awareness performance afforded by military hearing protection/enhancement devices (HPED) and tactical communication and protective systems (TCAPS). Design: Recognition/identification and pass-through communications tasks were separately conducted under three wind conditions (0, 5, and 10 mph). Subjects wore two in-ear-type TCAPS, one earmuff-type TCAPS, a Combat Arms Earplug in its ‘open’ or pass-through setting, and an EB-15LE™ electronic earplug. Devices with electronic gain systems were tested under two gain settings: ‘unity’ and ‘max’. Testing without any device (open ear) was conducted as a control. Study sample: Ten subjects were recruited from the student population at Virginia Tech. Audiometric requirements were 25 dBHL or better at 500, 1000, 2000, 4000, and 8000 Hz in both ears. Results: Performance on the interaction of communication task-by-device was significantly different only in 0 mph wind speed. The between-device performance differences varied with azimuthal speaker locations. Conclusions: It is evident from this study that stable (non-gusting) wind speeds up to 10 mph did not significantly degrade recognition/identification task performance and pass-through communication performance of the group of HPEDs and TCAPS tested. However, the various devices performed differently as the test sound signal speaker location was varied and it appears that physical as well as electronic features may have contributed to this directional result.

Comparison of mouse minimum audible angle determined in prepulse inhibition and operant conditioning procedures

Both reward based operant conditioning (OC) and reflex-based prepulse inhibition (PPI) procedures are used in sound localisation studies in mice. Since the results of both procedures are compared in the literature, it is important to assess whether they provide similar results if the same stimulus paradigm is applied. Here, we compare the sensitivity of C57BL/6 mice in OC and PPI procedures for detecting a switch in speaker location using broadband and narrowband noise stimuli and determined their minimum audible angle (MAA). In the OC procedure, we calculated d′ values from the hit and false alarm rates. In the PPI procedure, we calculated the area under ROC curves from the startle response amplitudes and derived da values to obtain a sensitivity measure that corresponds to d′. For both procedures, the mean sensitivity to the speaker switch increased with an increase in angular separation. For broadband noise stimuli, a d′ of up to 3.3 (OC) and a da of up to 1.1 (PPI) were observed at large speaker separations. Narrowband noise stimuli resulted in lower sensitivities in both procedures, resulting in a maximum d′ of 2.0 (OC) and a maximum da of 0.3 (PPI). Using a sensitivity of 1.0 as the threshold criterion, broadband noise MAAs in the range from 32° to 46° were observed in the OC procedure whereas a broadband noise MAAs of 108° or higher were observed in the PPI procedure. In the OC procedure, narrowband noise MAAs in the range from 37° to 62° were observed. In the PPI procedure, no narrowband noise MAA could be determined since none of the subjects reached the threshold. Thus, OC procedures result in a better performance of the subjects in the sound localization task than PPI procedures, challenging the view that both procedures can be used interchangeably.

Postdeviance distraction in younger and older adults: Neuro-behavioral evidence from speech perception.

Older adults are usually more easily distracted by task-irrelevant stimuli than younger ones. In addition, there is evidence that it takes them more time to overcome a distracting event. Here, the distracting effect of irregular switches in speaker location was studied in 22 younger and 22 older adults in a speech perception task. The participants responded to target words that were presented either from a frequent location (standard trials) or a rare location (deviant trials). Behavioral performance measures, event-related brain potentials (ERPs), and EEG synchronization (intertrial coherence [ITC]) were analyzed. Behavioral performance was not only decreased in deviant trials, but also in subsequent standard trials, indicating an effect of so-called postdeviance distraction. The negative impact of the distracting event (i.e., the switch in speaker location) on the processing of subsequent (standard) trials was more pronounced in the older versus younger group. The EEG analyses revealed a decreased N2 amplitude and increased latencies in P2 and P3a in the older group, suggesting a reduced and delayed attentional control. Moreover, there were age-related changes in late (200-450 ms) theta ITCs: While the ITC of the young remained constant in trials following a deviant, the ITC of the old decreased, suggesting a decline in processing of stimuli following (task-irrelevant) switches in speaker location. This could contribute to age-related deficits in speech comprehension in spatially variable listening situations. (PsycINFO Database Record

Multichannel Double-Talk Detector based on Fundamental Frequency Estimation

Multichannel acoustic echo cancellers aim at reducing the undesired acoustic coupling among microphones and loudspeakers in multichannel audio/video teleconferencing systems. In this scenario, double-talk refers to the situation in which there is an active speaker both in the remote room and in the local room. The detection of double-talk is one of the main issues to be faced in such systems to ensure a correct echo cancellation. Indeed, filters coefficients update has to be suppressed during double-talk in order to prevent the filter from diverging because of the presence of the local speaker. In this paper, a novel approach for double-talk detection is discussed based on the estimation of the fundamental frequency of the residual error signals through adaptive second-order notch filters. More specifically, simultaneous speaker activity in the remote and local rooms is revealed by exploiting the same frequency tracking introduced for multichannel decorrelation. Several experiments have been carried out to prove the effectiveness of the proposed approach also making comparison with other solutions of the state of the art.

Effects of speaker location and orientation on room sound fields and room to room noise reduction

ASTM E336 requires that loudspeakers be placed at least 5 m from a partition under test, and if that is not possible, then in the corners opposite the partition. The standard further advises that if directional loudspeakers are used in the corners, they should be faced into the corners. In practice, speakers are usually placed in the corners of residential and smaller office spaces, but not in the corners of large spaces. Recent experience has uncovered two effects that need further study. First, when loudspeakers are faced into the corners of a large room, the sound spectrum is strongly colored with a large dip typically in the 100 to 500 Hz range. This effect can be seen in smaller rooms, but is much less pronounced there so it is not normally noticed. Second, the noise reduction measured for a partition of a large room appears to vary depending on whether the speakers are in the corners or not, with lower noise reduction when the speakers are in the corners. This effect does not appear to depend on whether the speakers are faced into the corners or not. Data and any further observations and conclusions developed will be presented.

Acoustical deterrence of Silver Carp (Hypophthalmichthys molitrix)

The invasive Silver Carp (Hypophthalmichthys molitrix) dominate large regions of the Mississippi River drainage and continue to expand their range northward threatening the Laurentian Great Lakes. This study found that complex broadband sound (0–10 kHz) is effective in altering the behavior of Silver Carp with implications for deterrent barriers or potential control measures (e.g., herding fish into nets). The phonotaxic response of Silver Carp was investigated using controlled experiments in outdoor concrete ponds (10 × 4.9 × 1.2 m). Pure tones (500–2000 Hz) and complex sound (underwater field recordings of outboard motors) were broadcast using underwater speakers. Silver Carp always reacted to the complex sounds by exhibiting negative phonotaxis to the sound source and by alternating speaker location, Silver Carp could be directed consistently, up to 37 consecutive times, to opposite ends of the large outdoor pond. However, fish habituated quickly to pure tones, reacting to only approximately 5 % of these presentations and never showed more than two consecutive responses. Previous studies have demonstrated the success of sound barriers in preventing Silver Carp movement using pure tones and this research suggests that a complex sound stimulus would be an even more effective deterrent.

The force of V2 revisited

The main claim of this paper is that embedded V2 clauses in Mainland Scandinavian have the same syntactic structure as main clauses. This means that embedded V2 clauses contain a Force head, and as a consequence, embedded declarative V2 clauses will normally be asserted. Embedded V2 clauses are also similar to main clauses in that they contain linkers which encode the local speaker and addressee. Hence, indexical shift is possible in embedded V2 clauses, but not in non-V2 clauses, which do not contain linkers. In shifted contexts, the assertion that the embedded clause represents will be attributed to the local, implicit speaker, while in other cases, the actual speaker will be responsible for the assertion. The Force head is also the source of the V2 order, since it attracts the finite verb in addition to attracting a phrase from its complement to its Spec. In this way, the connection between V2 order and illocutionary force gets a straightforward explanation.

A Novel Nested Circular Microphone Array and Subband Processing-Based System for Counting and DOA Estimation of Multiple Simultaneous Speakers

This paper addresses the topic of simultaneous speaker localization. The work is related to the generalized cross-correlation (GCC)-based methods for estimating the direction of multiple speakers. Considering the defects of GCC-based direction of arrival (DOA) estimation methods, we have applied several modifications to improve our previous subband processing-based system for the localization of simultaneous speakers. Three modifications have been presented in this paper. In the first step, the DOA estimation method is equipped with a front-end block that determines the number of speakers based on K-means clustering and silhouette criterion. This block provides the true number of speakers for the DOA estimator. Secondly, in order to eliminate the spatial aliasing, we propose a novel nested circular microphone array. In the proposed array design, each microphone pair is only used in appropriate subband according to its inter-microphone distance. In the third step, to overcome the weakness of GCC-phase transform (GCC-PHAT) in noisy and noisy-reverberant conditions, we propose a SNR estimation block. So, we can separate noisy and reverberant conditions and use PHAT filter for reverberant conditions and maximum likelihood filter for noisy situations. The proposed method has been evaluated on both simulated and real multi-speaker speech data in various environmental conditions and different number of speakers. Our evaluations in terms of DOA accuracy demonstrate the superiority of the proposed method compared to the fullband and baseline subband methods.

Robust speaker localization for real-world robots

Autonomous human–robot interaction ultimately requires an artificial audition module that allows the robot to process and interpret a combination of verbal and non-verbal auditory inputs. A key component of such a module is the acoustic localization. The acoustic localization not only enables the robot to simultaneously localize multiple persons and auditory events of interest in the environment, but also provides input to auditory tasks such as speech enhancement and speech recognition. The use of microphone arrays in robots is an efficient and commonly applied approach to the localization problem. In this paper, moving away from simulated environments, we look at the acoustic localization under real-world conditions and limitations. Our approach proposes a series of enhancements, taking into account the imperfect frequency response of the array microphones and addressing the influence of the robot's shape and surface material. Motivated by the importance of the signal's phase information, we introduce a novel pre-processing step for enhancing the acoustic localization. Results show that the proposed approach improves the localization performance in joint noisy and reverberant conditions and allows a humanoid robot to locate multiple speakers in a real-world environment.

Speaker localization in noisy environments using steered response voice power

Many devices, including smart TVs and humanoid robots, can be operated through speech interface. Since a user can interact with such a device at a distance, speech-operated devices must be able to process speech signals from a distance. Although many methods exist to localize speakers via sound source localization, it is very difficult to reliably find the location of a speaker in a noisy environment. In particular, conventional sound source localization methods only find the loudest sound source within a given area, and such a sound source may not necessarily be related to human speech. This can be problematic in real environments where loud noises frequently occur, and the performance of speech-based interfaces for a variety of devices could be negatively impacted as a result. In this paper, a new speaker localization method is proposed. It identifies the location associated with the maximum voice power from all candidate locations. The proposed method is tested under a variety of conditions using both simulation data and real data, and the results indicate that the performance of the proposed method is superior to that of a conventional algorithm for various types of noises1.