Distant-talking Speech Research Articles

Phase and reverberation aware DNN for distant-talking speech enhancement

Enhancing reverberant speech with Deep Neural Networks (DNNs) is an interesting yet challenging topic. The performance of speech enhancement degrades significantly when test and training conditions are mismatched. In this paper we propose a Static Reverberation Aware Training (SRAT)-based dereverberation through which the reverberation estimate is obtained by averaging over broken down frame. This method significantly reduces the input dimensions of the and enables the DNN to learn the relations between clean and reverberant speech more efficiently. Most speech enhancement approaches ignore phase information due to its complicated structure. As phase correlates closely to speech signal we exploited this relationship to achieve better performance using DNN. Phase information was augmented with magnitude information and used as the input for DNN. We denote this method as phase aware DNN. Finally, both phase information and reverberation were added to reverberant speech to achieve better speech enhancement performance in a distant-talking condition. Features of the reverberant speech, phase and reverberation were used during the training and testing stages. This is because the DNN could use both reverberation and phase information to better generalize the speech signal. The proposed method was evaluated using the REVERB CHALLENGE 2014 database. Results are significantly improved results with respect to both reconstructed speech quality (PESQ: Perceptual Evaluation of Speech Quality) and influence of reverberation (SRMR: Speech to Reverberation Modulation Energy Ratio). As compared to the conventional DNN-based approach, this proposed one improved SRMR from 4.84 to 5.92 and PESQ from 2.34 to 2.70, indicating that our proposed method could efficiently enhance speech severely corrupted by reverberation.

An evaluation of speech reconstruction for degraded speech of optical microphone with deep neural network

Measuring distant-talking speech with high accuracy is important in detecting criminal activity. Various microphones such as the parabolic and shotgun microphones have been developed for measuring it. However, most of them have difficulty in extracting distant-talking speech at a target position if they are surrounded by noisy sound sources. Therefore, this study focuses on a microphone system to extract the distant-talking speech by vibrating a papery object using laser light. This system is referred to as an optical microphone in this study. The sound quality of the optical microphone is especially degraded at higher frequencies because it utilizes an external diaphragm consisting of various materials as the vibrating papery object. In this study, we therefore propose a reconstruction method for degraded distant-talking speech observed with the optical microphone. The method is realized by using a deep neural network (DNN) that is trained as the system between clean and observed speech signals. In the proposed method, the log-power spectra of 11 frames are used for the input of the DNN. Finally, we confirmed the effectiveness of the proposed system through an evaluation experiment.

Environment-dependent denoising autoencoder for distant-talking speech recognition

In this paper, we propose an environment-dependent denoising autoencoder (DAE) and automatic environment identification based on a deep neural network (DNN) with blind reverberation estimation for robust distant-talking speech recognition. Recently, DAEs have been shown to be effective in many noise reduction and reverberation suppression applications because higher-level representations and increased flexibility of the feature mapping function can be learned. However, a DAE is not adequate in mismatched training and test environments. In a conventional DAE, parameters are trained using pairs of reverberant speech and clean speech under various acoustic conditions (that is, an environment-independent DAE). To address the above problem, we propose two environment-dependent DAEs to reduce the influence of mismatches between training and test environments. In the first approach, we train various DAEs using speech from different acoustic environments, and the DAE for the condition that best matches the test condition is automatically selected (that is, a two-step environment-dependent DAE). To improve environment identification performance, we propose a DNN that uses both reverberant speech and estimated reverberation. In the second approach, we add estimated reverberation features to the input of the DAE (that is, a one-step environment-dependent DAE or a reverberation-aware DAE). The proposed method is evaluated using speech in simulated and real reverberant environments. Experimental results show that the environment-dependent DAE outperforms the environment-independent one in both simulated and real reverberant environments. For two-step environment-dependent DAE, the performance of environment identification based on the proposed DNN approach is also better than that of the conventional DNN approach, in which only reverberant speech is used and reverberation is not blindly estimated. And, the one-step environment-dependent DAE significantly outperforms the two-step environment-dependent DAE.

Deep Learning Based Distant-talking Speech Processing in Real-world Sound Environments

Deep Learning Based Distant-talking Speech Processing in Real-world Sound Environments

On the relationship between Early-to-Late Ratio of Room Impulse Responses and ASR performance in reverberant environments

This work presents an experimental analysis of distant-talking speech recognition in a variety of reverberant conditions, correlating ASR performance to a compact representation of the propagation channel (i.e., the room impulse response).It is well known that reverberation and background noise degrade speech recognition performance, but few studies have investigated the relation between room impulse responses and recognition rates in a comprehensive manner. In particular, we show how the ASR accuracy is related to features derived from the structure of the early arrivals and the reverberation tail. A representation based on the combination of few parameters is hence proposed, analysing the impact of reverberation on different speech recognition tasks. Possible applications of the derived measure are in data contamination for acoustic modeling where this feature can be employed either to select the most suitable model for a given acoustic condition or to define the subset of room impulse responses to be used for the creation of partially matched reverberant models. Recognition results using different back-end solutions (GMM, DNN) on data generated with the image method and with real impulse responses validate the effectiveness of the approach.

Distant-talking accent recognition by combining GMM and DNN

Recently, automatic accent recognition has been paid more and more attentions. However, there are few researches focusing on accent recognition in distant-talking environment which is very important for improving distant-talking speech recognition performance with non-native accents. In this paper, we apply Gaussian Mixture Models (GMM) and Deep Neural Network (DNN) to identify the speaker accent in reverberant environments. The combination of likelihood with these two approaches is also proposed. In reverberant environment, the accent recognition rate was improved from 90.7 % with GMM to 93.0 % with DNN. The combination of GMM and DNN achieved recognition rate of 97.5 %, which outperformed than the individual GMM and DNN because the complementation of GMM and DNN. The relative error reduction is 73.1 % than the GMM-based method and 64.3 % than the DNN-based method, respectively.

Combination of bottleneck feature extraction and dereverberation for distant-talking speech recognition

The performance of speech recognition in distant-talking environments is severely degraded by the reverberation that can occur in enclosed spaces (e.g., meeting rooms). To mitigate this degradation...

Weight-Space Viterbi Decoding Based Spectral Subtraction for Reverberant Speech Recognition

A single-channel blind dereverberation algorithm is proposed in this letter for distant-talking speech recognition. The proposed method is based on spectral subtraction (SS) method, in which the spectrum of a late reverberant signal is estimated using a delayed and attenuated version of the reverberant signal. Through some assumptions, the conventional SS method regards the attenuation weight as a constant that is a function of reverberation time. However, these assumptions are not valid in real situations, and the ideal weight varies with the frame. Therefore, in the proposed method, the variable weight sequence is estimated using Viterbi decoding scheme based on the reverberation model. This weight sequence is then substituted for the fixed weight in the conventional SS method without explicitly estimating the reverberation time. The proposed method performs better than the conventional SS method in both isolated word recognition and connected digit recognition experiments in reverberant environments.

Single-channel Dereverberation for Distant-Talking Speech Recognition by Combining Denoising Autoencoder and Temporal Structure Normalization

In this paper, we propose a robust distant-talking speech recognition by combining cepstral domain denoising autoencoder (DAE) and temporal structure normalization (TSN) filter. For the proposed method, after applying a DAE in the cepstral domain of speech to suppress reverberation, we apply a post-processing technology based on temporal structure normalization (TSN) filter to reduce the noise and reverberation effects by normalizing the modulation spectra to reference spectra of clean speech. The proposed method was evaluated using speech in simulated and real reverberant environments. By combining a cepstral-domain DAE and TSN, the average Word Error Rate (WER) was reduced from 25.2% of the baseline system to 21.2% in simulated environments and from 47.5% to 41.3% in real environments, respectively.

Deep neural network-based bottleneck feature and denoising autoencoder-based dereverberation for distant-talking speaker identification

AbstractDeep neural network (DNN)-based approaches have been shown to be effective in many automatic speech recognition systems. However, few works have focused on DNNs for distant-talking speaker recognition. In this study, a bottleneck feature derived from a DNN and a cepstral domain denoising autoencoder (DAE)-based dereverberation are presented for distant-talking speaker identification, and a combination of these two approaches is proposed. For the DNN-based bottleneck feature, we noted that DNNs can transform the reverberant speech feature to a new feature space with greater discriminative classification ability for distant-talking speaker recognition. Conversely, cepstral domain DAE-based dereverberation tries to suppress the reverberation by mapping the cepstrum of reverberant speech to that of clean speech with the expectation of improving the performance of distant-talking speaker recognition. Since the DNN-based discriminant bottleneck feature and DAE-based dereverberation have a strong complementary nature, the combination of these two methods is expected to be very effective for distant-talking speaker identification. A speaker identification experiment was performed on a distant-talking speech set, with reverberant environments differing from the training environments. In suppressing late reverberation, our method outperformed some state-of-the-art dereverberation approaches such as the multichannel least mean squares (MCLMS). Compared with the MCLMS, we obtained a reduction in relative error rates of 21.4% for the bottleneck feature and 47.0% for the autoencoder feature. Moreover, the combination of likelihoods of the DNN-based bottleneck feature and DAE-based dereverberation further improved the performance.

Automatic Distance Compensation for Robust Voice-based Human-Computer Interaction

Distant-talking voice-based HCI system suffers from performance degradation due to mismatch between the acoustic speech (runtime) and the acoustic model (training). Mismatch is caused by the change in the power of the speech signal as observed at the microphones. This change is greatly influenced by the change in distance, affecting speech dynamics inside the room before reaching the microphones. Moreover, as the speech signal is reflected, its acoustical characteristic is also altered by the room properties. In general, power mismatch due to distance is a complex problem. This paper presents a novel approach in dealing with distance-induced mismatch by intelligently sensing instantaneous voice power variation and compensating model parameters. First, the distant-talking speech signal is processed through microphone array processing, and the corresponding distance information is extracted. Distance-sensitive Gaussian Mixture Models (GMMs), pre-trained to capture both speech power and room property are used to predict the optimal distance of the speech source. Consequently, pre-computed statistic priors corresponding to the optimal distance is selected to correct the statistics of the generic model which was frozen during training. Thus, model combinatorics are post-conditioned to match the power of instantaneous speech acoustics at runtime. This results to an improved likelihood in predicting the correct speech command at farther distances. We experiment using real data recorded inside two rooms. Experimental evaluation shows voice recognition performance using our method is more robust to the change in distance compared to the conventional approach. In our experiment, under the most acoustically challenging environment (i.e., Room 2: 2.5 meters), our method achieved 24.2% improvement in recognition performance against the best-performing conventional method. Keywords—Human Machine Interaction, Human Computer Interaction, Voice Recognition, Acoustic Model Compensation, Acoustic

Performance estimation of speech recognition based on Perceptual Evaluation of Speech Quality and acoustic parameters under noisy and reverberant environments with Corpus and Environment for Noisy Speech RECognition 4

CENSREC-4 evaluation framework has been distributed for evaluating distant-talking speech under various noisy and reverberant environments. It however has not been evaluated how variable noisy and reverberant features in this contains. We thus try to evaluate CENSREC-4 with our designed noisy and reverberant criteria based on PESQ and acoustic parameters. We specifically focus on criteria to represent the difficulty of noisy and reverberant speech recognition, and also confirm why it is difficult to easily evaluate the recognition accuracy in a part of CENSREC-4 corpus with our proposed noisy and reverberant criteria. We first designed the noisy and reverberant criteria using the relationship among the D value, the PESQ, and the ASR performance. We then tried to estimate the recognition accuracy in various noisy and reverberant environments with CENSREC-4. We carried out evaluation experiments to confirm the difficulty to easily evaluate the recognition accuracy in a part of CENSREC-4 courpus0. As a result of evaluation experiments, we confirmed that it was difficult to estimate the accuracy of noisy and reverberant speech recognition in heavy noisy and reverberant environment with CENSREC-4. We therefore confirmed that CENSREC-4 contained very challenging and variable noisy and reverberant data.

Making Machines Understand Us in Reverberant Rooms: Robustness Against Reverberation for Automatic Speech Recognition

Speech recognition technology has left the research laboratory and is increasingly coming into practical use, enabling a wide spectrum of innovative and exciting voice-driven applications that are radically changing our way of accessing digital services and information. Most of today's applications still require a microphone located near the talker. However, almost all of these applications would benefit from distant-talking speech capturing, where talkers are able to speak at some distance from the microphones without the encumbrance of handheld or body-worn equipment [1]. For example, applications such as meeting speech recognition, automatic annotation of consumer-generated videos, speech-to-speech translation in teleconferencing, and hands-free interfaces for controlling consumer-products, like interactive TV, will greatly benefit from distant-talking operation. Furthermore, for a number of unexplored but important applications, distant microphones are a prerequisite. This means that distant talking speech recognition technology is essential for extending the availability of speech recognizers as well as enhancing the convenience of existing speech recognition applications.

Distant-talking speech enhancement based on spectrum restoring with phoneme labels

With the development of communication and speech recognition technology, remote conference and robot communication systems have been developed in recent years. Although these are valid in close-talking speech, distant-talking speech degrades the performance of the systems. The reason is that it is affected by energy decay, noise and reverberations depending on the distances. To solve this problem, many noise reduction and speech enhancement approaches have been proposed. In the conventional approaches, the example-based speech enhancement is one of the effective noise reduction methods in distant-talking conditions. The most similar example of noisy speech to input signal is detected from model examples of clean and noisy speech signals. Thereby it can estimate and suppress the noise based on the clean speech. Although it can effectively reduce noise, it is not clear the performance in the distant-talking speech condition with the decay of speech energy. Thus in this paper, we proposed a distant-talking speech enhancement based on spectrum restoring with phoneme labels. It restores the clean spectrum from decayed spectrum based on spectrum envelopes for each phoneme label. We demonstrate the experiment to evaluate the effectiveness of proposed method. As a result, we confirmed that it can effectively enhance the target speech.

Real-time sound source localization based on multi-resolution scanning in frequency domain

Robust speech recognition is necessary for realizing useful speech interfaces. A microphone array is an effective item at capturing distant-talking speech with high-quality in noisy environments. It captures the target speech by localizing a talker and steering the directivity. These processing is usually executed in each frequency. For realizing useful speech interface, these processing must be finished in real-time. In the research into sound source localization, various methods have already been developed, and these methods localize a sound source based on acoustic space scanning by fixed resolution in each frequency. Therefore, calculation time is increased, and the real-time processing is difficult in higher spatial resolution. To overcome this problem, we proposed the localization method based on multi-resolution scanning in frequency domain. The lower frequency band has lower spatial resolution, and the higher frequency band has higher spatial resolution. The proposed method localizes the sound sou...

An identification of speaker-dependence in reverberant-robust speech recognition

In recent years, a hands-free speech device has been developed with improving speech-recognition techniques. There is, however, a problem that the reverberant speech degrades the recognition performance in the field of distant-talking speech recognition. It is possibly addressed by taking preventive measures against the degradation of recognition performance with the reverberant criteria to estimate the recognition performance. We have already proposed the method to estimate recognition performance with ISO3382 acoustic parameter based on an impulse response. In this method, the recognition performance was estimated without speech features. Identification of the speaker with robust or weak features against reverberation makes it possible to adapt acoustic model for each speaker toward improving the recognition performance. In this research, we designed the speaker-dependence criteria in reverberant speech recognition. We first investigated existence of the speaker with robust or weak features against reverberation in various reverberant environments. After that, we compared clean and reverberant speech data in terms of speech features such as MFCC, delta MFCC, delta power, and utterance speed to evaluate the effects of reverberation on speech recognition. An experimental result showed the utterance speed was one of the effective candidates for the identification of speaker-dependence in reverberant-robust speech recognition.

CENSREC-4: An evaluation framework for distant-talking speech recognition in reverberant environments

We have been distributing a new collection of databases and evaluation tools called CENSREC-4, which is a framework for evaluating distant-talking speech in reverberant environments. The data contained in CENSREC-4 are connected digit utterances as in CENSREC-1. Two subsets are included in the data: “basic data sets” and “extra data sets.” The basic data sets are used for evaluating the room impulse response-convolved speech data to simulate the various reverberations. The extra data sets consist of simulated data and corresponding real recorded data. Evaluation tools are presently only provided for the basic data sets and will be delivered to the extra data sets in the future. The task of CENSREC-4 with a basic data set appears simple; however, the results of experiments prove that CENSREC-4 provides a challenging reverberation speech-recognition task, in the sense that a traditional technique to improve recognition and a widely used criterion to represent the difficulty of recognition deliver poor performance. Within this context, this common framework can be an important step toward the future evolution of reverberant speech-recognition methodologies.

Distant-Talking Speech Recognition Based on Spectral Subtraction by Multi-Channel LMS Algorithm

SUMMARY We propose a blind dereverberation method based on spectral subtraction using a multi-channel least mean squares (MCLMS) algorithm for distant-talking speech recognition. In a distant-talking environment, the channel impulse response is longer than the short-term spectral analysis window. By treating the late reverberation as additive noise, a noise reduction technique based on spectral subtraction was proposed to estimate the power spectrum of the clean speech using power spectra of the distorted speech and the unknown impulse responses. To estimate the power spectra of the impulse responses, a variable step-size unconstrained MCLMS (VSS-UMCLMS) algorithm for identifying the impulse responses in a time domain is extended to a frequency domain. To reduce the effect of the estimation error of the channel impulse response, we normalize the early reverberation by cepstral mean normalization (CMN) instead of spectral subtraction using the estimated impulse response. Furthermore, our proposed method is combined with conventional delay-andsum beamforming. We conducted recognition experiments on a distorted speech signal simulated by convolving multi-channel impulse responses with clean speech. The proposed method achieved a relative error reduction rate of 22.4% in relation to conventional CMN. By combining the proposed method with beamforming, a relative error reduction rate of 24.5% in relation to the conventional CMN with beamforming was achieved using only an isolated word (with duration of about 0.6s) to estimate the spectrum of the impulse response.

Reverberation Model-Based Decoding in the Logmelspec Domain for Robust Distant-Talking Speech Recognition

The REMOS (REverberation MOdeling for Speech recognition) concept for reverberation-robust distant-talking speech recognition, introduced in “Distant-talking continuous speech recognition based on a novel reverberation model in the feature domain” (A. Sehr , in Proc. Interspeech, 2006, pp. 769-772) for melspectral features, is extended to logarithmic melspectral (logmelspec) features in this contribution. Thus, the favorable properties of REMOS, including its high flexibility with respect to changing reverberation conditions, become available in the more competitive logmelspec domain. Based on a combined acoustic model consisting of a hidden Markov model (HMM) network and a reverberation model (RM), REMOS determines clean-speech and reverberation estimates during recognition. Therefore, in each iteration of a modified Viterbi algorithm, an inner optimization operation maximizes the joint density of the current HMM output and the RM output subject to the constraint that their combination is equal to the current reverberant observation. Since the combination operation in the logmelspec domain is nonlinear, numerical methods appear necessary for solving the constrained inner optimization problem. A novel reformulation of the constraint, which allows for an efficient solution by nonlinear optimization algorithms, is derived in this paper so that a practicable implementation of REMOS for logmelspec features becomes possible. An in-depth analysis of this REMOS implementation investigates the statistical properties of its reverberation estimates and thus derives possibilities for further improving the performance of REMOS. Connected digit recognition experiments show that the proposed REMOS version in the logmelspec domain significantly outperforms the melspec version. While the proposed RMs with parameters estimated by straightforward training for a given room are robust to a mismatch of the speaker-microphone distance, their performance significantly decreases if they are used in a room with substantially different conditions. However, by training multi-style RMs with data from several rooms, good performance can be achieved across different rooms.

Robust Talker Direction Estimation Based on Weighted CSP Analysis and Maximum Likelihood Estimation

This paper describes a new talker direction estimation method for front-end processing to capture distant-talking speech by using a microphone array. The proposed method consists of two algorithms: One is a TDOA (Time Delay Of Arrival) estimation algorithm based on a weighted CSP (Cross-power Spectrum Phase) analysis with an average speech spectrum and CSP coefficient subtraction. The other is a talker direction estimation algorithm based on ML (Maximum Likelihood) estimation in a time sequence of the estimated TDOAs. To evaluate the effectiveness of the proposed method, talker direction estimation experiments were carried out in an actual office room. The results confirmed that the talker direction estimation performance of the proposed method is superior to that of the conventional methods in both diffused- and directional-noise environments.