Automatic Speech Recognition Results Research Articles

Is the Same Performance Really the Same?: Understanding How Listeners Perceive ASR Results Differently According to the Speaker's Accent

Research suggests that automatic speech recognition (ASR) systems, which automatically convert speech to text, show different performances according to various input classes (e.g., accent, age), requiring attention to building fairer AI systems that would perform similarly across various input classes. However, would an AI system with the same performance regardless of input classes really be perceived as fair enough? To this end, we investigate how listeners perceive the ASR system of the same result differently according to whether the speaker is a native speaker (NS) or a non-native speaker (NNS), which may lead to unfair situations. We conducted a study (n = 420), where participants were given one of the ten speech recordings with various accents of the same script along with the same captions. We found that even with the same ASR output, listeners perceive the ASR results differently. They found captions to be more useful for NNS's speech and blamed NNS more for the errors than NS. Based on the findings, we present design implications suggesting that we should take a step further than just achieving the same performance across various input classes to build a fair ASR system.

Deep Neural Networks-based Classification Methodologies of Speech, Audio and Music, and its Integration for Audio Metadata Tagging

Videos contain visual and auditory information. Visual information in a video can include images of people, objects, and the landscape, whereas auditory information includes voices, sound effects, background music, and the soundscape. The audio content can provide detailed information on the story by conducting a voice and atmosphere analysis of the sound effects and soundscape. Metadata tags represent the results of a media analysis as text. The tags can classify video content on social networking services, like YouTube. This paper presents the methodologies of speech, audio, and music processing. Also, we propose integrating these audio tagging methods and applying them in an audio metadata generation system for video storytelling. The proposed system automatically creates metadata tags based on speech, sound effects, and background music information from the audio input. The proposed system comprises five subsystems: (1) automatic speech recognition, which generates text from the linguistic sounds in the audio, (2) audio event classification for the type of sound effect, (3) audio scene classification for the type of place from the soundscape, (4) music detection for the background music, and (5) keyword extraction from the automatic speech recognition results. First, the audio signal is converted into a suitable form, which is subsequently combined from each subsystem to create metadata for the audio content. We evaluated the proposed system using video logs (vlogs) on YouTube. The proposed system exhibits a similar accuracy to handcrafted metadata for the audio content, and for a total of 104 YouTube vlogs, achieves an accuracy of 65.83%.

Accented Speech Recognition Based on End-to-End Domain Adversarial Training of Neural Networks

The performance of automatic speech recognition (ASR) may be degraded when accented speech is recognized because the speech has some linguistic differences from standard speech. Conventional accented speech recognition studies have utilized the accent embedding method, in which the accent embedding features are directly fed into the ASR network. Although the method improves the performance of accented speech recognition, it has some restrictions, such as increasing the computational costs. This study proposes an efficient method of training the ASR model for accented speech in a domain adversarial way based on the Domain Adversarial Neural Network (DANN). The DANN plays a role as a domain adaptation in which the training data and test data have different distributions. Thus, our approach is expected to construct a reliable ASR model for accented speech by reducing the distribution differences between accented speech and standard speech. DANN has three sub-networks: the feature extractor, the domain classifier, and the label predictor. To adjust the DANN for accented speech recognition, we constructed these three sub-networks independently, considering the characteristics of accented speech. In particular, we used an end-to-end framework based on Connectionist Temporal Classification (CTC) to develop the label predictor, a very important module that directly affects ASR results. To verify the efficiency of the proposed approach, we conducted several experiments of accented speech recognition for four English accents including Australian, Canadian, British (England), and Indian accents. The experimental results showed that the proposed DANN-based model outperformed the baseline model for all accents, indicating that the end-to-end domain adversarial training effectively reduced the distribution differences between accented speech and standard speech.

Multi-Angle Lipreading with Angle Classification-Based Feature Extraction and Its Application to Audio-Visual Speech Recognition

Recently, automatic speech recognition (ASR) and visual speech recognition (VSR) have been widely researched owing to the development in deep learning. Most VSR research works focus only on frontal face images. However, assuming real scenes, it is obvious that a VSR system should correctly recognize spoken contents from not only frontal but also diagonal or profile faces. In this paper, we propose a novel VSR method that is applicable to faces taken at any angle. Firstly, view classification is carried out to estimate face angles. Based on the results, feature extraction is then conducted using the best combination of pre-trained feature extraction models. Next, lipreading is carried out using the features. We also developed audio-visual speech recognition (AVSR) using the VSR in addition to conventional ASR. Audio results were obtained from ASR, followed by incorporating audio and visual results in a decision fusion manner. We evaluated our methods using OuluVS2, a multi-angle audio-visual database. We then confirmed that our approach achieved the best performance among conventional VSR schemes in a phrase classification task. In addition, we found that our AVSR results are better than ASR and VSR results.

The relationship between word error rate and perceptual judgment

The aim of the present study is to examine the relationship between word error rate (WER) from an automatic speech recognition system and perceptual judgments (foreign-accentedness, fluency, and comprehensibility) from human raters. In a previous study, Franco et al. (1997) used HMM-derived scores based on posterior probabilities of phone segments, and Deville et al. (1999) used an HMM/ANN recognition approach to show how the results of automatic speech recognition can be used for perceptual judgments. Park and Culnan (2019) showed the possibility to assimilate human raters' perceptual judgments by using neural network models only with the speech signal, and suggested that the model worked better on accentedness judgments than fluency judgments. In this study, we will examine whether WERs of English sentences produced by three language groups (American, Korean, Chinese) are significantly different, and if there is any difference, we will analyze the correlations between WER and perceptual judgments. The perceptual data used in Park and Culnan (2019) will be used for the analysis. The preliminary results of this study will be used to find important features to build more accurate automatic proficiency judgment models.

Reviewing Speech Input with Audio

Speech input is a primary method of interaction for blind mobile device users, yet the process of dictating and reviewing recognized text through audio only (i.e., without access to visual feedback) has received little attention. A recent study found that sighted users could identify only about half of automatic speech recognition (ASR) errors when listening to text-to-speech output of the ASR results. Blind screen reader users, in contrast, may be better able to identify ASR errors through audio due to their greater use of speech interaction and increased ability to comprehend synthesized speech. To compare the experiences of blind and sighted users with speech input and ASR errors, as well as to compare their ability to identify ASR errors through audio-only interaction, we conducted a lab study with 12 blind and 12 sighted participants. The study included a semi-structured interview portion to qualitatively understand experiences with ASR, followed by a controlled speech input task to quantitatively compare participants’ ability to identify ASR errors in their dictated text. Findings revealed differences between blind and sighted participants in terms of how they use speech input and their level of concern for ASR errors (e.g., blind users were more highly concerned). In the speech input task, blind participants were able to identify only 40% of ASR errors, which, counter to our hypothesis, was not significantly different from sighted participants’ performance. In depth analysis of speech input, ASR errors, and strategy of identifying ASR errors scrutinized how participants entered a text with speech input and reviewed it. Our findings indicate the need for future work on how to support blind users in confidently using speech input to generate accurate, error-free text.

Speech Enhancement Based on Denoising Autoencoder With Multi-Branched Encoders

Deep learning-based models have greatly advanced the performance of speech enhancement (SE) systems. However, two problems remain unsolved, which are closely related to model generalizability to noisy conditions: (1) mismatched noisy condition during testing, i.e., the performance is generally sub-optimal when models are tested with unseen noise types that are not involved in the training data; (2) local focus on specific noisy conditions, i.e., models trained using multiple types of noises cannot optimally remove a specific noise type even though the noise type has been involved in the training data. These problems are common in real applications. In this article, we propose a novel denoising autoencoder with a multi-branched encoder (termed DAEME) model to deal with these two problems. In the DAEME model, two stages are involved: training and testing. In the training stage, we build multiple component models to form a multi-branched encoder based on a decision tree (DSDT). The DSDT is built based on prior knowledge of speech and noisy conditions (the speaker, environment, and signal factors are considered in this paper), where each component of the multi-branched encoder performs a particular mapping from noisy to clean speech along the branch in the DSDT. Finally, a decoder is trained on top of the multi-branched encoder. In the testing stage, noisy speech is first processed by each component model. The multiple outputs from these models are then integrated into the decoder to determine the final enhanced speech. Experimental results show that DAEME is superior to several baseline models in terms of objective evaluation metrics, automatic speech recognition results, and quality in subjective human listening tests.

Automatic perceptual judgment using neural networks

The aim of the present study is to investigate the performance of automatic perceptual judgment models built with neural networks. In previous studies, Franco et al. (1997) used HMM-derived scores based on posterior probabilities of phone segments, demonstrating a high correlation with human raters. Deville et al. (1999) also used an HMM/ANN recognition approach, and showed how the results of automatic speech recognition can be used for perceptual judgments. However, since most previous studies made use of automatic speech recognition in their analysis, the present study provides a different approach: using features and raw data. Native speakers of English will listen to English sentences produced by native and non-native speakers of English, transcribe what they heard, and respond to one of three perceptual judgements: foreign-accentedness, fluency, and comprehensibility. The data will be fed into prediction models in three different ways; one with annotated features (pauses, durations, etc), another with Mel Frequency Cepstral Coefficients (MFCC), and the other with Mel-spectrograms. The performance of the models will be measured by analyzing the correlation between the judgments by models and by human raters. The preliminary results of this study will be used to build more accurate automatic proficiency judgment models.

Building and evaluation of a real room impulse response dataset

This paper presents BUT ReverbDB - a dataset of real room impulse responses (RIR), background noises and re-transmitted speech data. The retransmitted data includes LibriSpeech test-clean, 2000 HUB5 English evaluation and part of 2010 NIST Speaker Recognition Evaluation datasets. We provide a detailed description of RIR collection (hardware, software, post-processing) that can serve as a "cook-book" for similar efforts. We also validate BUT ReverbDB in two sets of automatic speech recognition (ASR) experiments and draw conclusions for augmenting ASR training data with real and artificially generated RIRs. We show that a limited number of real RIRs, carefully selected to match the target environment, provide results comparable to a large number of artificially generated RIRs, and that both sets can be combined to achieve the best ASR results. The dataset is distributed for free under a non-restrictive license and it currently contains data from 8 rooms, which is growing. The distribution package also contains a Kaldi-based recipe for augmenting publicly available AMI close-talk meeting data and test the results on an AMI single distant microphone set, allowing it to reproduce our experiments.

Speech-driven mobile games for speech therapy: User experiences and feasibility

Purpose: To assist in remote treatment, speech-language pathologists (SLPs) rely on mobile games, which though entertaining, lack feedback mechanisms. Games integrated with automatic speech recognition (ASR) offer a solution where speech productions control gameplay. We therefore performed a feasibility study to assess children’s and SLPs’ experiences towards speech-controlled games, game feature preferences and ASR accuracy.Method: Ten children with childhood apraxia of speech (CAS), six typically developing (TD) children and seven SLPs trialled five games and answered questionnaires. Researchers also compared the results of ASR to perceptual judgment.Result: Children and SLPs found speech-controlled games interesting and fun, despite ASR–human disagreements. They preferred games with rewards, challenge and multiple difficulty levels. Automatic speech recognition–human agreement was higher for SLPs than children, similar between TD and CAS and unaffected by CAS severity (77% TD, 75% CAS – incorrect; 51% TD, 47% CAS, 71% SLP – correct). Manual stop recording yielded higher agreement than automatic. Word length did not influence agreement.Conclusion: Children’s and SLPs’ positive responses towards speech-controlled games suggest that they can engage children in higher intensity practice. Our findings can guide future improvements to the ASR, recording methods and game features to improve the user experience and therapy adherence.

Predicting speech intelligibility with deep neural networks

An accurate objective prediction of human speech intelligibility is of interest for many applications such as the evaluation of signal processing algorithms. To predict the speech recognition threshold (SRT) of normal-hearing listeners, an automatic speech recognition (ASR) system is employed that uses a deep neural network (DNN) to convert the acoustic input into phoneme predictions, which are subsequently decoded into word transcripts. ASR results are obtained with and compared to data presented in Schubotz et al. (2016), which comprises eight different additive maskers that range from speech-shaped stationary noise to a single-talker interferer and responses from eight normal-hearing subjects. The task for listeners and ASR is to identify noisy words from a German matrix sentence test in monaural conditions. Two ASR training schemes typically used in applications are considered: (A) matched training, which uses the same noise type for training and testing and (B) multi-condition training, which covers all eight maskers. For both training schemes, ASR-based predictions outperform established measures such as the extended speech intelligibility index (ESII), the multi-resolution speech envelope power spectrum model (mr-sEPSM) and others. This result is obtained with a speaker-independent model that compares the word labels of the utterance with the ASR transcript, which does not require separate noise and speech signals. The best predictions are obtained for multi-condition training with amplitude modulation features, which implies that the noise type has been seen during training. Predictions and measurements are analyzed by comparing speech recognition thresholds and individual psychometric functions to the DNN-based results.

Spoken Term Detection Using Spoken Document Index Based on Keywords Collected from Automatic Speech Recognition Result

Spoken Term Detection Using Spoken Document Index Based on Keywords Collected from Automatic Speech Recognition Result

음성인식 기반 응급상황관제

In emergency dispatching at 119 Command & Dispatch Center, some inconsistencies between the 'standard emergency aid system' and 'dispatch protocol,' which are both mandatory to follow, cause inefficiency in the dispatcher's performance. If an emergency dispatch system uses automatic speech recognition (ASR) to process the dispatcher's protocol speech during the case registration, it instantly extracts and provides the required information specified in the 'standard emergency aid system,' making the rescue command more efficient. For this purpose, we have developed a Korean large vocabulary continuous speech recognition system for 400,000 words to be used for the emergency dispatch system. The 400,000 words include vocabulary from news, SNS, blogs and emergency rescue domains. Acoustic model is constructed by using 1,300 hours of telephone call (8 kHz) speech, whereas language model is constructed by using 13 GB text corpus. From the transcribed corpus of 6,600 real telephone calls, call logs with emergency rescue command class and identified major symptom are extracted in connection with the rescue activity log and National Emergency Department Information System (NEDIS). ASR is applied to emergency dispatcher's repetition utterances about the patient information. Based on the Levenshtein distance between the ASR result and the template information, the emergency patient information is extracted. Experimental results show that 9.15% Word Error Rate of the speech recognition performance and 95.8% of emergency response detection performance are obtained for the emergency dispatch system.

Enhancements in Statistical Spoken Language Translation by De-normalization of ASR Results

Spoken language translation (SLT) has become very important in an increasingly globalized world. Machine translation (MT) for automatic speech recognition (ASR) systems is a major challenge of great interest. This research investigates that automatic sentence segmentation of speech that is important for enriching speech recognition output and for aiding downstream language processing. This article focuses on the automatic sentence segmentation of speech and improving MT results. We explore the problem of identifying sentence boundaries in the transcriptions produced by automatic speech recognition systems in the Polish language. We also experiment with reverse normalization of the recognized speech samples.

Using automatic speech recognition to identify pediatric speech errors

Speech delay is a childhood language problem that might resolve without intervention, but might alternatively presage continued speech and language deficits. Thus, early detection through screening might help to identify children for whom intervention is warranted. The goal of this work is to develop Automatic Speech Recognition (ASR) methods to partially automate screening for speech delay in young children. Speech data were recorded from typically developing and speech delayed children (N = 63) aged 6 to 9 years old during administration of the Goldman Fristoe Test of Articulation (GFTA). Monophone Hidden Markov Model (HMM) acoustic models were trained on speech data obtained from 207 typically developing children in the same age range. These training data consisted of a total of about 18,000 single-word utterances. The HMMs were then used to develop an utterance verification system to distinguish correct versus error productions. Several variations of the recognition strategy, feature extraction, and scoring methods were investigated. The best overall ASR result for distinguishing normal versus abnormal speech is approximately 86%. It is hypothesized that the ASR methods could approach the level of accuracy of speech therapists for this task (agreement among multiple therapists is over 95%), but a much larger database may be needed.

Improving Domain-independent Cloud-Based Speech Recognition with Domain-Dependent Phonetic Post-Processing

Automatic speech recognition (ASR) technology has been developed to such a level that off-the-shelf distributed speech recognition services are available (free of cost), which allow researchers to integrate speech into their applications with little development effort or expert knowledge leading to better results compared with previously used open-source tools. Often, however, such services do not accept language models or grammars but process free speech from any domain. While results are very good given the enormous size of the search space, results frequently contain out-of-domain words or constructs that cannot be understood by subsequent domain-dependent natural language understanding (NLU) components. We present a versatile post-processing technique based on phonetic distance that integrates domain knowledge with open-domain ASR results, leading to improved ASR performance. Notably, our technique is able to make use of domain restrictions using various degrees of domain knowledge, ranging from pure vocabulary restrictions via grammars or N-Grams to restrictions of the acceptable utterances. We present results for a variety of corpora (mainly from human-robot interaction) where our combined approach significantly outperforms Google ASR as well as a plain open-source ASR solution.

On Grounding Natural Kind Terms in Human-Robot Communication

Our contribution situates Human-Robot Communication, especially the grounding of Natural Kind Terms, in the interface of Artificial Intelligence, Cognitive Psychology, Philosophy, Robotics and Semantics. We investigate whether a robot can be grounded in the sense favoured in Artificial Intelligence and Philosophy.We thus extend the notion of grounding to social symbol grounding using an interactive perspective addressing the question how grounding can be achieved in detail in interaction. For the acquisition of Natural Kind Terms we establish the notions of foundational common ground and foundational grounding in contrast to the established common ground and grounding. We introduce the robot setting used and provide a deep evaluation of a tutorial dialogue between a user and the robot. We investigate these Human-Robot Communication data from an ethno-methodological and an “omniscient” perspective (the latter amounting to consideration of automatic speech recognition results) and test whether these perspectives matter for analysing grounding. We show that the robot has acquired a partial concept of a Natural Kind Term—represented by statistics over visual object features—and that this is shared knowledge, hence the first step of a grounding sequence. Finally, we argue that grounding of robots can be achieved and extended to situated structures of considerable complexity.

Experimental research of influence of acoustic noises of different types on results of automatic speech recognition

Experimental research of influence of acoustic noises of different types on results of automatic speech recognition

A Joint Approach for Single-Channel Speaker Identification and Speech Separation

In this paper, we present a novel system for joint speaker identification and speech separation. For speaker identification a single-channel speaker identification algorithm is proposed which provides an estimate of signal-to-signal ratio (SSR) as a by-product. For speech separation, we propose a sinusoidal model-based algorithm. The speech separation algorithm consists of a double-talk/single-talk detector followed by a minimum mean square error estimator of sinusoidal parameters for finding optimal codevectors from pre-trained speaker codebooks. In evaluating the proposed system, we start from a situation where we have prior information of codebook indices, speaker identities and SSR-level, and then, by relaxing these assumptions one by one, we demonstrate the efficiency of the proposed fully blind system. In contrast to previous studies that mostly focus on automatic speech recognition (ASR) accuracy, here, we report the objective and subjective results as well. The results show that the proposed system performs as well as the best of the state-of-the-art in terms of perceived quality while its performance in terms of speaker identification and automatic speech recognition results are generally lower. It outperforms the state-of-the-art in terms of intelligibility showing that the ASR results are not conclusive. The proposed method achieves on average, 52.3% ASR accuracy, 41.2 points in MUSHRA and 85.9% in speech intelligibility.

Automatic speech recognition performance in different room acoustic environments with and without dereverberation preprocessing

The performance of recent dereverberation methods for reverberant speech preprocessing prior to Automatic Speech Recognition (ASR) is compared for an extensive range of room and source-receiver configurations. It is shown that room acoustic parameters such as the clarity (C50) and the definition (D50) correlate well with the ASR results. When available, such room acoustic parameters can provide insight into reverberant speech ASR performance and potential improvement via dereverberation preprocessing. It is also shown that the application of a recent dereverberation method based on perceptual modelling can be used in the above context and achieve significant Phone Recognition (PR) improvement, especially under highly reverberant conditions.