Hands-free Speech Recognition Research Articles

Establishing document creation support for people with upper limb disabilities by hands-free speech recognition and gaze detection

Everyday life is now inundated with the need to send, receive and process information through the written word. Many workplaces rely heavily on email for communication between colleagues, both local and across the world. News sites, social media and even remaining in contact with friends all primarily employ text as the language medium of choice. Naturally, there are many reasons for this, ranging from the convenience of giving someone time to think and reply to keeping information more private. For the able-bodied, typing on a computer or smartphone is quick and easy. However, these typing-based modes of interaction are much more difficult for those who have upper limb disabilities. If you cannot use your arms or hands in the same manner as the able-bodied, you cannot easily participate in this world of text-based communication. Better solutions are required. In recent years, technology aimed at converting speech into text has been growing. Such functions now exist on all smartphones and many computers, as well as in other mediums like digital personal assistants. This sort of technology can be very convenient if you have a lot on your hands, but it can be transformative for those with upper limb disabilities. The problem, however, is that speech recognition technology still struggles in a few key areas. Firstly, when there is background noise for a microphone to detect, this can confuse the speech recognition software. Background noise, of course, is nearly always present through everyday occurrences such as traffic, other people's conversations and reverberations. The second major problem is the recognition by the software of punctuation. Both cases lead to errors in transcription that are usually corrected manually by the user. Naturally, this is not possible for the upper limb disabled. Finding better solutions to the current shortcomings of speech-to-text software is a focus for Assistant Professor Ryoichi Miyazaki of the Department of Computer Science and Electronic Engineering at the National Institute of Technology, Tokuyama College, Japan.

Strategies for distant speech recognitionin reverberant environments

Reverberation and noise are known to severely affect the automatic speech recognition (ASR) performance of speech recorded by distant microphones. Therefore, we must deal with reverberation if we are to realize high-performance hands-free speech recognition. In this paper, we review a recognition system that we developed at our laboratory to deal with reverberant speech. The system consists of a speech enhancement (SE) front-end that employs long-term linear prediction-based dereverberation followed by noise reduction. We combine our SE front-end with an ASR back-end that uses neural networks for acoustic and language modeling. The proposed system achieved top scores on the ASR task of the REVERB challenge. This paper describes the different technologies used in our system and presents detailed experimental results that justify our implementation choices and may provide hints for designing distant ASR systems.

Efficient Blind Dereverberation and Echo Cancellation Based on Independent Component Analysis for Actual Acoustic Signals

This letter presents a new algorithm for blind dereverberation and echo cancellation based on independent component analysis (ICA) for actual acoustic signals. We focus on frequency domain ICA (FD-ICA) because its computational cost and speed of learning convergence are sufficiently reasonable for practical applications such as hands-free speech recognition. In applying conventional FD-ICA as a preprocessing of automatic speech recognition in noisy environments, one of the most critical problems is how to cope with reverberations. To extract a clean signal from the reverberant observation, we model the separation process in the short-time Fourier transform domain and apply the multiple input/output inverse-filtering theorem (MINT) to the FD-ICA separation model. A naive implementation of this method is computationally expensive, because its time complexity is the second order of reverberation time. Therefore, the main issue in dereverberation is to reduce the high computational cost of ICA. In this letter, we reduce the computational complexity to the linear order of the reverberation time by using two techniques: (1) a separation model based on the independence of delayed observed signals with MINT and (2) spatial sphering for preprocessing. Experiments show that the computational cost grows in proportion to the linear order of the reverberation time and that our method improves the word correctness of automatic speech recognition by 10 to 20 points in a RT₂₀= 670 ms reverberant environment.

A New Evidence Model for Missing Data Speech Recognition With Applications in Reverberant Multi-Source Environments

Conventional hidden Markov model (HMM) decoders often experience severe performance degradations in practice due to their inability to cope with uncertain data in time-varying environments. In order to address this issue, we propose the bounded-Gauss-Uniform mixture probability density function (pdf) as a new class of evidence model for missing data speech recognition. Exemplary for a hands-free speech recognition scenario, we illustrate how the parameters of the new mixture pdf can be estimated with the help of a multi-channel source separation front-end. In comparison with other models the new evidence pdf retains a fuller description of the available data and provides a more effective link between source separation and recognition. The superiority of the bounded-Gauss-Uniform mixture pdf over conventional approaches is demonstrated for a connected digits recognition task under varying test conditions.

Speech Dereverberation Based on Maximum-Likelihood Estimation With Time-Varying Gaussian Source Model

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> Distant acquisition of acoustic signals in an enclosed space often produces reverberant components due to acoustic reflections in the room. Speech dereverberation is in general desirable when the signal is acquired through distant microphones in such applications as hands-free speech recognition, teleconferencing, and meeting recording. This paper proposes a new speech dereverberation approach based on a statistical speech model. A time-varying Gaussian source model (TVGSM) is introduced as a model that represents the dynamic short time characteristics of nonreverberant speech segments, including the time and frequency structures of the speech spectrum. With this model, dereverberation of the speech signal is formulated as a maximum-likelihood (ML) problem based on multichannel linear prediction, in which the speech signal is recovered by transforming the observed signal into one that is probabilistically more like nonreverberant speech. We first present a general ML solution based on TVGSM, and derive several dereverberation algorithms based on various source models. Specifically, we present a source model consisting of a finite number of states, each of which is manifested by a short time speech spectrum, defined by a corresponding autocorrelation (AC) vector. The dereverberation algorithm based on this model involves a finite collection of spectral patterns that form a codebook. We confirm experimentally that both the time and frequency characteristics represented in the source models are very important for speech dereverberation, and that the prior knowledge represented by the codebook allows us to further improve the dereverberated speech quality. We also confirm that the quality of reverberant speech signals can be greatly improved in terms of the spectral shape and energy time-pattern distortions from simply a short speech signal using a speaker-independent codebook. </para>

Harmonicity-Based Blind Dereverberation for Single-Channel Speech Signals

The distant acquisition of acoustic signals in an enclosed space often produces reverberant artifacts due to the room impulse response. Speech dereverberation is desirable in situations where the distant acquisition of acoustic signals is involved. These situations include hands-free speech recognition, teleconferencing, and meeting recording, to name a few. This paper proposes a processing method, named Harmonicity-based dEReverBeration (HERB), to reduce the amount of reverberation in the signal picked up by a single microphone. The method makes extensive use of harmonicity, a unique characteristic of speech, in the design of a dereverberation filter. In particular, harmonicity enhancement is proposed and demonstrated as an effective way of estimating a filter that approximates an inverse filter corresponding to the room impulse response. Two specific harmonicity enhancement techniques are presented and compared; one based on an average transfer function and the other on the minimization of a mean squared error function. Prototype HERB systems are implemented by introducing several techniques to improve the accuracy of dereverberation filter estimation, including time warping analysis. Experimental results show that the proposed methods can achieve high-quality speech dereverberation, when the reverberation time is between 0.1 and 1.0 s, in terms of reverberation energy decay curves and automatic speech recognition accuracy

Robust In-Car Speech Recognition Based on Nonlinear Multiple Regressions

We address issues for improving handsfree speech recognition performance in different car environments using a single distant microphone. In this paper, we propose a nonlinear multiple-regression-based enhancement method for in-car speech recognition. In order to develop a data-driven in-car recognition system, we develop an effective algorithm for adapting the regression parameters to different driving conditions. We also devise the model compensation scheme by synthesizing the training data using the optimal regression parameters and by selecting the optimal HMM for the test speech. Based on isolated word recognition experiments conducted in 15 real car environments, the proposed adaptive regression approach shows an advantage in average relative word error rate (WER) reductions of 52.5% and 14.8%, compared to original noisy speech and ETSI advanced front end, respectively.

Blind spatial subtraction array based on independent component analysis for speech enhancement and recognition

We propose a new blind spatial subtraction array (BSSA) that contains an accurate noise estimator based on independent component analysis (ICA) for the realization of noise-robust hands-free speech recognition. Many previous studies on ICA-based blind source separation often dealt with the special case of speech-speech mixing. However, such a sound mixing is not realistic under common acoustic conditions; the target speech can be approximated to a point source but real noises are often not point sources. Under the condition, our preliminary experiment suggests that the conventional ICA is proficient in the noise estimation rather than the direct speech estimation. Based on the above-mentioned findings, we propose a new noise reduction method that is implemented in subtracting the power spectrum of the estimated noise by ICA from the power spectrum of noise-contaminated observations. This architecture provides us a noise-estimation-error robust speech enhancement rather than a simple linear-filtering-based enhancement. Although nonlinear processing often generates an artificial distortion, the so-called musical noise, it is still applicable to the speech recognition system because the speech decoder is not so sensitive to such a distortion. Experimental results reveal that the proposed BSSA can improve the speech recognition rate by 20% compared with the conventional ICA.

Acoustic Model Adaptation Using First-Order Linear Prediction for Reverberant Speech

This paper describes a hands-free speech recognition technique based on acoustic model adaptation to reverberant speech. In hands-free speech recognition, the recognition accuracy is degraded by reverberation, since each segment of speech is affected by the reflection energy of the preceding segment. To compensate for the reflection signal we introduce a frame-by-frame adaptation method adding the reflection signal to the means of the acoustic model. The reflection signal is approximated by a first-order linear prediction from the observation signal at the preceding frame, and the linear prediction coefficient is estimated with a maximum likelihood method by using the EM algorithm, which maximizes the likelihood of the adaptation data. Its effectiveness is confirmed by word recognition experiments on reverberant speech.

Adaptive Nonlinear Regression Using Multiple Distributed Microphones for In-Car Speech Recognition

In this paper, we address issues in improving hands-free speech recognition performance in different car environments using multiple spatially distributed microphones. In the previous work, we proposed the multiple linear regression of the log spectra (MRLS) for estimating the log spectra of speech at a close-talking microphone. In this paper, the concept is extended to nonlinear regressions. Regressions in the cepstrum domain are also investigated. An effective algorithm is developed to adapt the regression weights automatically to different noise environments. Compared to the nearest distant microphone and adaptive beamformer (Generalized Sidelobe Canceller), the proposed adaptive nonlinear regression approach shows an advantage in the average relative word error rate (WER) reductions of 58.5% and 10.3%, respectively, for isolated word recognition under 15 real car environments.

Adaptive log-spectral regression for in-car speech recognition using multiple distributed microphones

This letter addresses issues in improving hands-free speech recognition performance in different car environments. We propose a new speech-enhancement approach based on optimizing regression of the log-spectra, which is used to estimate the log-spectra of speech at a close-talking microphone by using multiple spatially distributed microphones. The regression weights can be adapted automatically for different noise environments. Compared to the nearest distant microphone and adaptive beamformer generalized sidelobe canceller (GSC), the proposed approach shows an advantage in the average relative word error rate (WER) reduction of 58.5 and 10.3%, respectively, for isolated word recognition under 15 real-car environments.

Use of Microphone Array and Model Adaptation for Hands-Free Speech Acquisition and Recognition

This paper presents a combined microphone array and model adaptation algorithm for hands-free speech recognition. Our purpose is to remove the inconvenience of using head-mounted/hand-holding microphone in conventional speech recognizer. To improve the speech quality with car noise interference, a linear microphone array is applied and acted as robust acquisition system. A time-domain coherence measure (TDCM) is applied to reliably estimate the time delay for speech signals collected by different microphones. The estimated delay is adopted in a delay-and-sum beamformer for speech enhancement. Further, we adapt the speech hidden Markov models to get close to the acoustic conditions of the enhanced test speech for robust speech recognition. In acquisition and recognition experiments using connected Chinese digits, we found that TDCM can effectively estimate the time delay. The increase in the speech sampling rate is helpful to determine the time delay. Incorporating the model adaptation scheme significantly reduces the recognition errors with moderate computation overhead.

Simultaneous Recognition of Distant-Talking Speech of Multiple Talkers Based on the 3-D N-Best Search Method

This paper describes a novel method for hands-free speech recognition and in particular for simultaneous recognition of distant-talking speech of multiple sound sources (talkers or noise sources). Our method is based on the 3-D Viterbi search extended to a 3-D N-best search method to allow simultaneous speech recognition of multiple talkers. The baseline system integrates two existing technologies—3-D Viterbi search and conventional N-best search—into a complete system. However, initial evaluation of the 3-D N-best search-based system showed that new ideas were needed in order to build a system to simultaneously recognize multiple sound sources. Two factors were found to have an important role in system performance. Those two factors are the different likelihood ranges of the talkers and the direction-based separation of the hypotheses. More specifically, since we have to compare hypotheses originating from different talkers, an accurate comparison of these hypotheses cannot be made due to the different likelihood dynamic range of the talkers. Moreover, the hypotheses originated from talkers are located in different directions and therefore separating them based on their direction provides an efficient method for accurate recognition. To solve these problems, we implemented a likelihood normalization technique and a path distance-based clustering technique into the baseline 3-D N-best search-based system. The performance of our system was evaluated by experiments for recognizing the distant-talking speech of two talkers. The experiments were carried out on simulated (with only time delay) data and on reverberated (simulated and real) data. In this paper, we evaluated the proposed method in reverberant environments, and we introduced results obtained by experiments at several reverberation times and results obtained in a real environment. The experiments showed that implementing the two techniques described above produced significant improvements. Best results for simulated data were obtained by implementing the two techniques and using a microphone array composed of 32 channels. In that case in particular, the Simultaneous Word Accuracy (where both talkers are correctly recognized simultaneously) in the ‘top 1’ hypothesis was 72.49%, and in the ‘top 3’ hypotheses was 86.25%, which were very promising results.

Hands-free speech recognition in real environments using microphone array and 2-levels MLLR adaptation as a front-end system for conversational TV

Hands-free speech recognition in real environments using microphone array and 2-levels MLLR adaptation as a front-end system for conversational TV

HMM-separation-based speech recognition for a distant moving speaker

This paper presents a hands-free speech recognition method based on HMM composition and separation for speech contaminated not only by additive noise but also by an acoustic transfer function. The method realizes an improved user interface such that a user is not encumbered by microphone equipment in noisy and reverberant environments. The use of HMM composition has already been proposed for countering additive noise. In this paper, the same approach is extended to handle convolutional acoustic distortion in a reverberant room, by using an HMM to model the acoustic transfer function. The states of this HMM correspond to different positions of the sound source. It can represent the positions of the sound sources, even if the speaker moves. This paper also proposes a new method, HMM separation, for estimating the HMM parameters of the acoustic transfer function on the basis of a maximum likelihood manner. The proposed method is obtained through the reverse of the process of HMM composition, where the model parameters are estimated by maximizing the likelihood of adaptation data uttered from an unknown position. Therefore, measurement of impulse responses is not required. The paper also describes the performance of the proposed methods for recognizing real distant-talking speech. The results of experiments clarify the effectiveness of the proposed method.

Frame-synchronous noise compensation for hands-free speech recognition in car environments

It has become increasingly important to develop hands-free speech recognition techniques for the human–computer interface in car environments. However, severe car noise degrades the speech recognition performance substantially. To compensate the performance loss, it is necessary to adapt the original speech hidden Markov models (HMMs) to meet changing car environments. A novel frame-synchronous adaptation mechanism for in-car speech recognition is presented. This mechanism is intended to perform unsupervised model adaptation efficiently on a frame-by-frame basis instead of a conventional adaptation algorithm relying on batch adaptation data and supervision information. The proposed adaptation scheme is performed during frame likelihood calculation where an optimal equalisation factor is first computed to equalise the model mean vector and the input frame vector. This equalisation factor then serves as a reference index to retrieve an additional bias vector for model mean adaptation. As a result, a rapid and flexible algorithm is exploited to establish a new robust likelihood measure. In experiments on hands-free in-car speech recognition with the microphone far from the talker, this framework is found to be effective in terms of recognition rate and computational cost under various driving speeds.

Method and apparatus including microphone arrays and neural networks for speech/speaker recognition systems

A neural network is trained to transform distant-talking cepstrum coefficients, derived from a microphone array receiving speech from a speaker distant therefrom, into a form substantially similar to close-talking cepstrum coefficients that would be derived from a microphone close to the speaker, for providing robust hands-free speech and speaker recognition in adverse practical environments with existing speech and speaker recognition systems which have been trained on close-talking speech.

Environmental conditions and acoustic transduction in hands-free speech recognition

Hands-free interaction represents a key-point for increase of flexibility of present applications and for the development of new speech recognition applications, where the user cannot be encumbered by either hand-held or head-mounted microphones. When the microphone is far from the speaker, the transduced signal is affected by degradation of different nature, that is often unpredictable. Special microphones and multi-microphone acquisition systems represent a way of reducing some environmental noise effects. Robust processing and adaptation techniques can be further used in order to compensate for different kinds of variability that may be present in the recognizer input. The purpose of this paper is to re-visit some of the assumptions about the different sources of this variability and to discuss both on special transducer systems and on compensation/adaptation techniques that can be adopted. In particular, the paper will refer to the use of multi-microphone systems to overcome some undesired effects caused by room acoustics (e.g. reverberation) and by coherent/incoherent noise (e.g. competitive talkers, computer fans). The paper concludes with the description of some experiments that were conducted both on real and simulated speech data.

Robust hands-free speech recognition

When speech recognition technology moves from the laboratory to real-world applications, there is increasing need for robustness. This paper describes a system of microphone arrays and neural networks (MANN) for robust hands-free speech recognition. MANN has the advantage that existing speech recognition systems can directly be deployed in practical adverse environments where distant-talking sound pickup is required. No retraining nor modification of the recognizers is necessary. MANN consists of two synergistic components: (1) signal enhancement by microphone arrays and (2) feature adaptation by neural network computing. High-quality sound capture by the microphone array enables successful feature adaptation by the neural network to mitigate environmental interference. Through neural network computation, a matched training and testing condition is approximated which typically elevates performance of speech recognition. Both computer-simulated and real-room speech input are used to evaluate the capability of MANN. Measurements of isolated-word recognition in noisy, reverberant, and distant-talking conditions show that MANN leads to a word recognition accuracy which is within 4%–6% of that obtained under a close-talking condition in quiet.