Talker Localization Research Articles

Impact of room acoustics and visual cues on speech perception and talker localization by children with mild bilateral or unilateral hearing loss

Impact of room acoustics and visual cues on speech perception and talker localization by children with mild bilateral or unilateral hearing loss

Project starline

We present a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were copresent. It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (e.g., presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (e.g., head nods, eyebrow movements). This milestone is reached by maximizing audiovisual fidelity and the sense of copresence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones. The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space dereverberation, and talker localization.

Swarm Intelligence Based Particle Filter for Alternating Talker Localization and Tracking Using Microphone Arrays

We address the problem of localizing and tracking alternating moving or stationary talkers using microphone arrays in a room environment. One of the main challenges is the frequent and possibly abrupt change of talker positions, which requires the algorithm to capture the active talker rapidly. In addition, the presence of interference, background noise, and room reverberation degrades the tracking performance. We propose a new algorithm that jointly exploits the advantages of the particle filter PF and particle swarm intelligence. The PF is used as a general tracking framework, which incorporates a proposed alternating source-dynamic model for recursive estimation of talker position. Unlike the conventional PF, where particles operate independently in the particle sampling stage, the use of swarm intelligence allows particles to interact with each other, thereby improving convergence toward the active talker location. In addition, the memory mechanism in swarm intelligence allows particles to remain at their previous best-fit state estimate when signals are corrupted by interference, noise, and/or reverberation. Simulations and experiments were conducted to demonstrate the effectiveness of the proposed algorithm.

Single-channel talker localization based on separation of the acoustic transfer function using hidden Markov model and its classification

Single-channel talker localization based on separation of the acoustic transfer function using hidden Markov model and its classification

Talker Localization Based on Interference between Transmitted and Reflected Audible Sound

In many engineering fields, distance to targets is very important. General distance measurement method uses a time delay between transmitted and reflected waves, but it is difficult to estimate the short distance. On the other hand, the method using phase interference to measure the short distance has been known in the field of microwave radar. Therefore, we have proposed the distance estimation method based on interference between transmitted and reflected audible sound, which can measure the distance between microphone and target with one microphone and one loudspeaker. In this paper, we propose talker localization method based on distance estimation using phase interference. We expand the distance estimation method using phase interference into two microphones (microphone array) in order to estimate talker position. The proposed method can estimate talker position by measuring the distance and direction between target and microphone array. In addition, talker's speech is regarded as a noise in the proposed method. Therefore, we also propose combination of the proposed method and CSP (Cross-power Spectrum Phase analysis) method which is one of the DOA (Direction Of Arrival) estimation methods. We evaluated the performance of talker localization in real environments. The experimental result shows the effectiveness of the proposed method.

Single-Channel Talker Localization Based on Discrimination of Acoustic Transfer Functions

This paper presents a sound source (talker) localization method using only a single microphone, where a Gaussian Mixture Model (GMM) of clean speech is introduced to estimate the acoustic transfer function from a user's position. The new method is able to carry out this estimation without measuring impulse responses. The frame sequence of the acoustic transfer function is estimated by maximizing the likelihood of training data uttered from a given position, where the cepstral parameters are used to effectively represent useful clean speech. Using the estimated frame sequence data, the GMM of the acoustic transfer function is created to deal with the influence of a room impulse response. Then, for each test dataset, we find a maximum-likelihood (ML) GMM from among the estimated GMMs corresponding to each position. The effectiveness of this method has been confirmed by talker localization experiments performed in a room environment.

Omnidirectional Audio-Visual Talker Localization Based on Dynamic Fusion of Audio-Visual Features Using Validity and Reliability Criteria

This paper proposes a robust omnidirectional audio-visual (AV) talker localizer for AV applications. The proposed localizer consists of two innovations. One of them is robust omnidirectional audio and visual features. The direction of arrival (DOA) estimation using an equilateral triangular microphone array, and human position estimation using an omnidirectional video camera extract the AV features. The other is a dynamic fusion of the AV features. The validity criterion, called the audio-or visual-localization counter, validates each audio-or visual-feature. The reliability criterion, called the speech arriving evaluator, acts as a dynamic weight to eliminate any prior statistical properties from its fusion procedure. The proposed localizer can compatibly achieve talker localization in a speech activity and user localization in a non-speech activity under the identical fusion rule. Talker localization experiments were conducted in an actual room to evaluate the effectiveness of the proposed localizer. The results confirmed that the talker localization performance of the proposed AV localizer using the validity and reliability criteria is superior to that of conventional localizers.

Acoustic talker localization

An apparatus for locating a talker, comprising a microphone array for receiving multiple audio signals, wherein the microphone array is characterized by a predetermined beampattern, a spectral conditioner for filtering the audio signals to optimize the beampattern for talker localization, a localization estimator for calculating a localization estimate based on the filtered audio signals, an activity detector for detecting periods of speech activity, and decision logic for verifying the localization estimate during the periods of speech activity.

Voice activity detection

A method for detecting voice activity comprises receiving audio signals on a plurality of channels and processing the audio signals on the channels e.g. by beamformers 200 to improve the signal-to-noise ratio thereof. The processed audio signals on each channel are then fed to associated voice activity detection algorithms 202 and further processed. A voice or silence determination is then rendered by decision logic 204 based on at least the output of the voice activity detection algorithms. The method is useful in talker localization systems e.g. for teleconferencing.

A study of robust omnidirectional audio‐visual talker localization algorithm with microphone array and omnidirectional image

In video conferencing environments, it is very important to localize the talker. However, conventional audio signal‐based algorithms often suffer from audio interference, and conventional visual signal‐based algorithms fail in the presence of visual interference. To deal with these problems, this paper proposes a robust omnidirectional audio‐visual talker localization algorithm that not only exploits audio feature parameters, but also subordinately uses visual feature parameters. To achieve omnidirectional audio‐visual talker localization, paired‐omnidirectional microphones are employed as an audio sensor, and an omnidirectional camera is employed as a visual sensor. For robust talker localization, audio feature parameters are extracted using weighted cross‐power spectrum phase (CSP) analysis and CSP coefficient subtraction, and visual feature parameters are extracted using background subtraction and skin‐color detection. The talker is finally located by the fusing of weighted audio/visual feature parameters, and the weight of this feature parameter fusion is automatically controlled based on the reliable criterion of audio feature parameters. The results of localization experiments in an actual room revealed that the proposed audio‐visual talker localization algorithm is superior to that of conventional localizers using only audio or visual feature parameters, but not both. [Work supported by MEXT of Japan.]

Robust Joint Audio–Video Talker Localization in Video Conferencing Using Reliability Information—II: Bayesian Network Fusion

This paper builds on our August 2004 paper where data fusion is used to combine results from multiple audio and video localizers in order to locate a talker in video conferencing applications. The two studies differ in the type of data fusion engine used. The former study explored using a summing voter for data fusion, whereas this study employs a Bayesian network. The novelty of both papers is the use of reliability estimates to improve the overall localization performance and robustness. Reliability estimates, which are derived based on known physical properties of each individual localizer, were used to gauge the trustworthiness of the current localization results. These reliability estimates were introduced into the fusion engines to achieve better performance. Although the summing voter fusion engine used in improves the overall localization performance, it does not take into account the unique characteristics of each localizer. The Bayesian network allows the inclusion of these unique characteristics as part of the fusion process and therefore further improves the overall localization performance. When one or more localizers fails, the persistent erroneous data streams from the failed localizers can negatively affect a statistically based data fusion method, like the Bayesian network, resulting in poor localization accuracy . The study described in this paper explores the use of the reliability estimates to automatically stop the failed devices from contributing in the data fusion process, hence improving the overall robustness of the system. In this paper, we investigate the impact of 1) using a Bayesian network as the data fusion engine, 2) adding reliability estimates into the fusion engine, and 3) using reliability estimates to stop the failed localizers from contributing in the fusion process.

Robust Joint Audio-Video Localization in Video Conferencing Using Reliability Information

This paper proposes a new method for performing joint audio-video talker localization that explores the reliability of the individual localization estimates such as audio, motion detection, and skin-color detection. The reliability information is estimated from the audio and video data separately. The proposed method then uses this reliability information in conjunction with a simple summing voter to dynamically discriminate erroneous outputs from the localizers while performing fusion on the localization results. Based on the voter output, a majority rule is then used to make the final decision of the active talker's current location. The results show that adding the reliability information during fusion improves localization performance when compared to audio only, motion detection only, skin-color detection only, and joint audio-video using straight summing fusion localization methods. The computational complexity of the proposed method is comparable to the existing ones.

An evaluation of talker localization based on direction of arrival estimation and statistical sound source identification

It is very important to capture distant-talking speech for a hands-free speech interface with high quality. A microphone array is an ideal candidate for this purpose. However, this approach requires localizing the target talker. Conventional talker localization algorithms in multiple sound source environments not only have difficulty localizing the multiple sound sources accurately, but also have difficulty localizing the target talker among known multiple sound source positions. To cope with these problems, we propose a new talker localization algorithm consisting of two algorithms. One is DOA (direction of arrival) estimation algorithm for multiple sound source localization based on CSP (cross-power spectrum phase) coefficient addition method. The other is statistical sound source identification algorithm based on GMM (Gaussian mixture model) for localizing the target talker position among localized multiple sound sources. In this paper, we particularly focus on the talker localization performance based on the combination of these two algorithms with a microphone array. We conducted evaluation experiments in real noisy reverberant environments. As a result, we confirmed that multiple sound signals can be identified accurately between ‘‘speech’’ or ‘‘non-speech’’ by the proposed algorithm. [Work supported by ATR, and MEXT of Japan.]

Distant-talking speech recognition based on a 3-D Viterbi search using a microphone array

This paper focuses on microphone arrays to realize distant-talking speech recognition in real environments. In distant-talking situations, users can speak at arbitrary positions while moving. Therefore, it,is very important for high quality speech acquisition using microphone arrays to localize a talker accurately. However, it is very difficult to localize a moving talker in noisy and reverberant environments. The talker localization errors result in performance degradation of speech recognition. One way to solve this problem is to integrate the speech recognition process and the talker localization into a unified framework. This paper proposes a new speech recognition algorithm based on a three-dimensional (3-D) Viterbi search. The 3-D Viterbi method extracts a direction-time sequence of parameter vectors by steering a beam to every direction in every frame, then finds the most likely path in a 3-D trellis space composed of talker directions, input frames and HMM states. This means that speech recognition and talker localization are performed simultaneously within a statistical framework. To evaluate the performance of the 3-D Viterbi method, recognition experiments for real environment data were carried out. The results confirmed that the 3-D Viterbi method drastically improves the recognition performance for the moving talker case as well as for the fixed-position talker case.

Time-delay estimation of reverberated speech exploiting harmonic structure.

The relative time delay associated with a speech signal received at a pair of spatially separated microphones is a key component in talker localization and microphone array beamforming procedures. The traditional method for estimating this parameter utilizes the generalized cross correlation (GCC), the performance of which is compromised by the presence of room reverberations and background noise. Typically, the GCC filtering criteria used are either focused on the signal degradations due to additive noise or those due exclusively to multipath channel effects. There has been relatively little success at applying GCC weighting schemes which are robust to both of these conditions. This paper details an alternative approach which attempts to employ a signal-dependent criterion, namely, the estimated periodicity of the speech signal, to design a GCC filter appropriate for the combination of noise and multipath distortions. Simulations are performed across a range of room conditions to illustrate the utility of the proposed time-delay estimation method relative to conventional GCC filtering approaches.