Close-talking Microphone Research Articles

Improving phoneme recognition of throat microphone speech recordings using transfer learning

Throat microphones (TM) are a type of skin-attached non-acoustic sensors, which are robust to environmental noise but carry a lower signal bandwidth characterization than the traditional close-talk microphones (CM). Attaining high-performance phoneme recognition is a challenging task when the training data from a degrading channel, such as TM, is limited. In this paper, we address this challenge for the TM speech recordings using a transfer learning approach based on the stacked denoising auto-encoders (SDA). The proposed transfer learning approach defines an SDA-based domain adaptation framework to map the source domain CM representations and the target domain TM representations into a common latent space, where the mismatch across TM and CM is eliminated to better train an acoustic model and to improve the TM phoneme recognition. For the phoneme recognition task, we use the convolutional neural network (CNN) and the hidden Markov model (HMM) based CNN/HMM hybrid system, which delivers better acoustic modeling performance compared to the conventional Gaussian mixture model (GMM) based models. In the experimental evaluations, we observed more than 12% relative phoneme error rate (PER) improvement for the TM recordings with the proposed transfer learning approach compared to baseline performances.

Domain Adaptation for Food Intake Classification With Teacher/Student Learning

Automatic dietary monitoring (ADM) stands as a challenging application in wearable healthcare technologies. In this paper, we define an ADM to perform food intake classification (FIC) over throat microphone recordings. We investigate the use of transfer learning to design an improved FIC system. Although labeled data with acoustic close-talk microphones are abundant, throat data is scarce. Therefore, we propose a new adaptation framework based on teacher/student learning. The teacher network is trained over high-quality acoustic microphone recordings, whereas the student network distills deep feature extraction capacity of the teacher over a parallel dataset. Our approach allows us to transfer the representational capacity, adds robustness to the resulting model, and improves the FIC through throat microphone recordings. The classification problem is formulated as a spectra-temporal sequence recognition using the Convolutional LSTM (ConvLSTM) models. We evaluate the proposed approach using a large scale acoustic dataset collected from online recordings, an in-house food intake throat microphone dataset, and a parallel speech dataset. The bidirectional ConvLSTM network with the proposed domain adaptation approach consistently outperforms the SVM- and CNN-based baseline methods and attains 85.2% accuracy for the classification of 10 different food intake items. This translates to 17.8% accuracy improvement with the proposed domain adaptation.

End-to-end audiovisual speech activity detection with bimodal recurrent neural models

Speech activity detection (SAD) plays an important role in current speech processing systems, including automatic speech recognition (ASR). SAD is particularly difficult in environments with acoustic noise. A practical solution is to incorporate visual information, increasing the robustness of the SAD approach. An audiovisual system has the advantage of being robust to different speech modes (e.g., whisper speech) or background noise. Recent advances in audiovisual speech processing using deep learning have opened opportunities to capture in a principled way the temporal relationships between acoustic and visual features. This study explores this idea, proposing the bimodal recurrent neural network (BRNN) framework for SAD. The approach models the temporal dynamic of the sequential audiovisual data, improving the accuracy and robustness of the proposed SAD system. Instead of estimating hand-crafted features, the study investigates an end-to-end training approach, where acoustic and visual features are directly learned from the raw data during training. The experimental evaluation considers a large audiovisual corpus with over 60.8 h of recordings, collected from 105 speakers. The results demonstrate that the proposed framework leads to absolute improvements up to 1.2% under practical scenarios over a VAD baseline using only audio implemented with deep neural network (DNN). The proposed approach achieves 92.7% F1-score when it is evaluated using the sensors from a portable tablet under noisy acoustic environment, which is only 1.0% lower than the performance obtained under ideal conditions (e.g., clean speech obtained with a high definition camera and a close-talking microphone).

Estimating realistic speech levels for virtual acoustic environments

Speech intelligibility is commonly assessed in rather unrealistic acoustic environments at negative signal-to-noise ratios (SNRs). As a consequence, the results seem unlikely to reflect the subjects’ experience in the real world. To improve the ecological validity of speech tests, different sound reproduction techniques have been used by researchers to recreate field-recorded acoustic environments in the laboratory. Whereas the real-world sound pressure levels of these environments are usually known, this is not necessarily the case for the level of the target speech (and therefore the SNR). In this study, a two-talker conversation task is used to derive realistic target speech levels for given virtual acoustic environments. The talkers communicate with each other while listening to binaural recordings of the environments using highly open headphones. During the conversation their speech is recorded using close-talk microphones. Conversations between ten pairs of young normal-hearing talkers were recorded in this way in 12 different environments and the corresponding speech levels were derived. In this presentation, the methods are introduced and the derived speech levels are compared to results from the literature as well as from real sound-field recordings. The possibility of using this technique to generate environment-specific speech material with realistic vocal effort is discussed.

Microphone Array Processing for Distant Speech Recognition: From Close-Talking Microphones to Far-Field Sensors

Distant speech recognition (DSR) holds the promise of the most natural human computer interface because it enables man-machine interactions through speech, without the necessity of donning intrusive body- or head-mounted microphones. Recognizing distant speech robustly, however, remains a challenge. This contribution provides a tutorial overview of DSR systems based on microphone arrays. In particular, we present recent work on acoustic beam forming for DSR, along with experimental results verifying the effectiveness of the various algorithms described here; beginning from a word error rate (WER) of 14.3% with a single microphone of a linear array, our state-of-the-art DSR system achieved a WER of 5.3%, which was comparable to that of 4.2% obtained with a lapel microphone. Moreover, we present an emerging technology in the area of far-field audio and speech processing based on spherical microphone arrays. Performance comparisons of spherical and linear arrays reveal that a spherical array with a diameter of 8.4 cm can provide recognition accuracy comparable or better than that obtained with a large linear array with an aperture length of 126 cm.

Development of close-talking microphone for Olympic broadcasting

Development of close-talking microphone for Olympic broadcasting

The physiological microphone (PMIC): A competitive alternative for speaker assessment in stress detection and speaker verification

Interactive speech system scenarios exist which require the user to perform tasks which exert limitations on speech production, thereby causing speaker variability and reduced speech performance. In noisy stressful scenarios, even if noise could be completely eliminated, the production variability brought on by stress, including Lombard effect, has a more pronounced impact on speech system performance. Thus, in this study we focus on the use of a silent speech interface (PMIC), with a corresponding experimental assessment to illustrate its utility in the tasks of stress detection and speaker verification. This study focuses on the suitability of PMIC versus close-talk microphone (CTM), and reports that the PMIC achieves as good performance as CTM or better for a number of test conditions. PMIC reflects both stress-related information and speaker-dependent information to a far greater extent than the CTM. For stress detection performance (which is reported in % accuracy), PMIC performs at least on par or about 2% better than the CTM-based system. For a speaker verification application, the PMIC outperforms CTM for all matched stress conditions. The performance reported in terms of %EER is 0.91% (as compared to 1.69%), 0.45% (as compared to 1.49%), and 1.42% (as compared to 1.80%) for PMIC. This indicates that PMIC reflects speaker-dependent information. Also, another advantage of the PMIC is its ability to record the user physiology traits/state. Our experiments illustrate that PMIC can be an attractive alternative for stress detection as well as speaker verification tasks along with an advantage of its ability to record physiological information, in situations where the use of CTM may hinder operations (deep sea divers, fire-fighters in rescue operations, etc.).

Adaptive Beamforming With a Minimum Mutual Information Criterion

In this paper, we consider an acoustic beamforming application where two speakers are simultaneously active. We construct one subband-domain beamformer in generalized sidelobe canceller (GSC) configuration for each source. In contrast to normal practice, we then jointly optimize the active weight vectors of both GSCs to obtain two output signals with minimum mutual information (MMI). Assuming that the subband snapshots are Gaussian-distributed, this MMI criterion reduces to the requirement that the cross-correlation coefficient of the subband outputs of the two GSCs vanishes. We also compare separation performance under the Gaussian assumption with that obtained from several super-Gaussian probability density functions (pdfs), namely, the Laplace and pdfs. Our proposed technique provides effective nulling of the undesired source, but without the signal cancellation problems seen in conventional beamforming. Moreover, our technique does not suffer from the source permutation and scaling ambiguities encountered in conventional blind source separation algorithms. We demonstrate the effectiveness of our proposed technique through a series of far-field automatic speech recognition experiments on data from the PASCAL Speech Separation Challenge (SSC). On the SSC development data, the simple delay-and-sum beamformer achieves a word error rate (WER) of 70.4%. The MMI beamformer under a Gaussian assumption achieves a 55.2% WER, which is further reduced to 52.0% with a pdf, whereas the WER for data recorded with a close-talking microphone is 21.6%.

Speech Enhancement and Recognition in Meetings With an Audio–Visual Sensor Array

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> This paper addresses the problem of distant speech acquisition in multiparty meetings, using multiple microphones and cameras. Microphone array beamforming techniques present a potential alternative to close-talking microphones by providing speech enhancement through spatial filtering. Beamforming techniques, however, rely on knowledge of the speaker location. In this paper, we present an integrated approach, in which an audio– visual multiperson tracker is used to track active speakers with high accuracy. Speech enhancement is then achieved using microphone array beamforming followed by a novel postfiltering stage. Finally, speech recognition is performed to evaluate the quality of the enhanced speech signal. The approach is evaluated on data recorded in a real meeting room for stationary speaker, moving speaker, and overlapping speech scenarios. The results show that the speech enhancement and recognition performance achieved using our approach are significantly better than a single table-top microphone and are comparable to a lapel microphone for some of the scenarios. The results also indicate that the audio–visual-based system performs significantly better than audio-only system, both in terms of enhancement and recognition. This reveals that the accurate speaker tracking provided by the audio–visual sensor array proved beneficial to improve the recognition performance in a microphone array-based speech recognition system. </para>

Short-Term Spatio–Temporal Clustering Applied to Multiple Moving Speakers

Distant microphones permit to process spontaneous multiparty speech with very little constraints on speakers, as opposed to close-talking microphones. Minimizing the constraints on speakers permits a large diversity of applications, including meeting summarization and browsing, surveillance, hearing aids, and more natural human-machine interaction. Such applications of distant microphones require to determine where and when the speakers are talking. This is inherently a multisource problem, because of background noise sources, as well as the natural tendency of multiple speakers to talk over each other. Moreover, spontaneous speech utterances are highly discontinuous, which makes it difficult to track the multiple speakers with classical filtering approaches, such as Kalman filtering of particle filters. As an alternative, this paper proposes a probabilistic framework to determine the trajectories of multiple moving speakers in the short-term only, i.e., only while they speak. Instantaneous location estimates that are close in space and time are grouped into ldquoshort-term clustersrdquo in a principled manner. Each short-term cluster determines the precise start and end times of an utterance and a short-term spatial trajectory. Contrastive experiments clearly show the benefit of using short-term clustering, on real indoor recordings with seated speakers in meetings, as well as multiple moving speakers.

Portable communication apparatus and microphone device for the apparatus

The invention relates to a portable communication apparatus and a microphone device for the apparatus. In the portable communication apparatus of the invention, an openable/closable member (2) is attached in an openable and closable manner to a main unit (1) of the portable apparatus. The openable/closable member (2) incorporates a bidirectional close-talking microphone (5). An inner sound hole (26) and an outer sound hole (27) are opened in an inner wall (22) and an outer wall (24) of the openable/closable member (2), respectively. The inner wall (22) faces the front face of a diaphragm of the close-talking microphone (5) via a sound path (64), and the outer sound hole (27) faces the rear face of the diaphragm of the close-talking microphone (5) via a sound path (65). The inner sound hole (26) is biasedly located in a position which is close to a free end of the openable/closable member (2), and the outer sound hole (27) is biasedly located in a position which is close to the center position of opening and closing movements of the openable/closable member (2). The portable communication apparatus of the invention can obtain a speech quality of a very high S/N ratio at a substantially same degree in both the cases where the openable/closable member (2) is opened, and where the member is closed.

CENSREC-3: An Evaluation Framework for Japanese Speech Recognition in Real Car-Driving Environments

This paper introduces a common database, an evaluation framework, and its baseline recognition results for in-car speech recognition, CENSREC-3, as an outcome of the IPSJ-SIG SLP Noisy Speech Recognition Evaluation Working Group. CENSREC-3, which is a sequel to AURORA-2J, has been designed as the evaluation framework of isolated word recognition in real car-driving environments. Speech data were collected using two microphones, a close-talking microphone and a hands-free microphone, under 16 carefully controlled driving conditions, i.e., combinations of three car speeds and six car conditions. CENSREC-3 provides six evaluation environments designed using speech data collected in these conditions.

Single-Channel Multiple Regression for In-Car Speech Enhancement

We address issues for improving hands-free speech enhancement and speech recognition performance in different car environments using a single distant microphone. This paper describes a new single-channel in-car speech enhancement method that estimates the log spectra of speech at a close-talking microphone based on the nonlinear regression of the log spectra of noisy signal captured by a distant microphone and the estimated noise. The proposed method provides significant overall quality improvements in our subjective evaluation on the regression-enhanced speech, and performed best in most objective measures. Based on our isolated word recognition experiments conducted under 15 real car environments, the proposed adaptive nonlinear regression approach shows an advantage in average relative word error rate (WER) reductions of 50.8% and 13.1%, respectively, compared to original noisy speech and ETSI advanced front-end (ETSI ES 202 050).

A self-steering close-talking microphone array*

Abstract For communication in extremely noisy environments, close-talking microphone arrays are useful. Differential microphone array systems possessing excellent nearfield directivity and strong farfield-noise reduction capability are very suitable for close-talking situations. In order to improve the focusing effect at the desired bearing in the nearfield we propose a novel self-steering system of close-talking arrays with the first-order differential sub-arrays in this paper. Calculations of directivity patterns and directivity factors show that when a kind of optimized beamforming technique is adopted even small arrays with a focusing configuration will exhibit a satisfactory nearfield directivity which is electronically steerable within a specific range. The superiority of the proposed array system is verified by experiments with real acoustic data. * Supported by National Science Foundation of China (Grant Nos. 60272037 and 60340420325), the Motorola UPR Project, and the Research Fund for the Doctor...

Position independent close-talking microphone

Close-talking microphones are required for speech pick-up in high noise environments and are based on the assumption that unwanted noise sources are distant from a desired nearfield source. Common close-talking microphones in use today have to be carefully oriented relative to the desired nearfield source. To address this problem, a new close-talking microphone array is proposed that is orientation-invariant with respect to the attenuation of farfield noise sources. It is shown that by appropriate processing of the close-talking microphone array signals, one can adaptively compensate for the distance and orientation of the microphone for a nearfield source. The proposed close-talking array is based on an spatially orthonormal decomposition of the soundfield for a nearfield source. It consists of four or more pressure microphones that are mounted in the surface of a rigid sphere with a diameter on the order of a centimeter. Finally, it is shown that decomposing the soundfield into spatially orthonormal modes leads to a computationally efficient implementation.

Adaptive Nonlinear Regression Using Multiple Distributed Microphones for In-Car Speech Recognition

In this paper, we address issues in improving hands-free speech recognition performance in different car environments using multiple spatially distributed microphones. In the previous work, we proposed the multiple linear regression of the log spectra (MRLS) for estimating the log spectra of speech at a close-talking microphone. In this paper, the concept is extended to nonlinear regressions. Regressions in the cepstrum domain are also investigated. An effective algorithm is developed to adapt the regression weights automatically to different noise environments. Compared to the nearest distant microphone and adaptive beamformer (Generalized Sidelobe Canceller), the proposed adaptive nonlinear regression approach shows an advantage in the average relative word error rate (WER) reductions of 58.5% and 10.3%, respectively, for isolated word recognition under 15 real car environments.

Performance of real-time source-location estimators for a large-aperture microphone array

A large array of microphones is being studied as a possible means of acquiring data in offices, conference rooms, and auditoria without requiring close-talking microphones. An array that surrounds all possible sources has a large aperture and such arrays have attractive properties for accurate spatial resolution and significant signal-to-noise enhancement. For the first time, this paper presents all the details of a real-time, source-location algorithm (LEMSalg) based on time-of-arrival delays derived from a phase transform applied to the generalized cross-power spectrum. It is being used successfully in a representative environment where microphone SNRs are below 0 dB. We have found that many small features are required to make a useful location estimating algorithm work and work well in real-time. We present an experimental evaluation of the current algorithm's performance using data taken with the Huge Microphone Array (HMA) system, which has 448 microphones in a noisy, reverberant environment. Using off-line computation, we also compared the LEMSalg to two alternative methods. The first of these adds local beamforming to the preprocessing of the base algorithm, increasing performance significantly at modest additional computational cost. The second algorithm maximizes the total steered-response power in the same phase transform. While able to derive good position estimates from shorter data runs, this method is two orders of magnitude more computationally expensive and is not yet suitable for real-time use.

Adaptive log-spectral regression for in-car speech recognition using multiple distributed microphones

This letter addresses issues in improving hands-free speech recognition performance in different car environments. We propose a new speech-enhancement approach based on optimizing regression of the log-spectra, which is used to estimate the log-spectra of speech at a close-talking microphone by using multiple spatially distributed microphones. The regression weights can be adapted automatically for different noise environments. Compared to the nearest distant microphone and adaptive beamformer generalized sidelobe canceller (GSC), the proposed approach shows an advantage in the average relative word error rate (WER) reduction of 58.5 and 10.3%, respectively, for isolated word recognition under 15 real-car environments.

Multiple Regression of Log Spectra for In-Car Speech Recognition Using Multiple Distributed Microphones

This paper describes a new multi-channel method of noisy speech recognition, which estimates the log spectrum of speech at a closetalking microphone based on the multiple regression of the log spectra (MRLS) of noisy signals captured by distributed microphones. The advantages of the proposed method are as follows: 1) The method does not require a sensitive geometric layout, calibration of the sensors nor additional pre-processing for tracking the speech source; 2) System works in very small computation amounts; and 3) Regression weights can be statistically optimized over the given training data. Once the optimal regression weights are obtained by regression learning, they can be utilized to generate the estimated log spectrum in the recognition phase, where the speech of close-talking is no longer required. The performance of the proposed method is illustrated by speech recognition of real in-car dialogue data. In comparison to the nearest distant microphone and multi-microphone adaptive beamformer, the proposed approach obtains relative word error rate (WER) reductions of 9.8% and 3.6%, respectively.

Factors affecting the performance of large-aperture microphone arrays.

Large arrays of microphones have been proposed and studied as a possible means of acquiring data in offices, conference rooms, and auditoria without requiring close-talking microphones. When such an array essentially surrounds all possible sources, it is said to have a large aperture. Large-aperture arrays have attractive properties of spatial resolution and signal-to-noise enhancement. This paper presents a careful comparison of theoretical and measured performance for an array of 256 microphones using simple delay-and-sum beamforming. This is the largest currently functional, all digital-signal-processing array that we know of. The array is wall-mounted in the moderately adverse environment of a general-purpose laboratory (8 m x 8 m x 3 m). The room has a T60 reverberation time of 550 ms. Reverberation effects in this room severely impact the array's performance. However, the width of the main lobe remains comparable to that of a simplified prediction. Broadband spatial resolution shows a single central peak with 10 dB gain about 0.4 m in diameter at the -3 dB level. Away from that peak, the response is approximately flat over most of the room. Optimal weighting for signal-to-noise enhancement degrades the spatial resolution minimally. Experimentally, we verify that signal-to-noise gain is less than proportional to the square root of the number of microphones probably due to the partial correlation of the noise between channels, to variation of signal intensity with polar angle about the source, and to imperfect correlation of the signal over the array caused by reverberations. We show measurements of the relative importance of each effect in our environment.