Use Of Automatic Speech Recognition Research Articles

Ensuring Safety for Artificial-Intelligence-Based Automatic Speech Recognition in Air Traffic Control Environment

This paper describes the safety assessment conducted in SESAR2020 project PJ.10-W2-96 ASR on automatic speech recognition (ASR) technology implemented for air traffic control (ATC) centers. ASR already now enables the automatic recognition of aircraft callsigns and various ATC commands including command types based on controller–pilot voice communications for presentation at the controller working position. The presented safety assessment process consists of defining design requirements for ASR technology application in normal, abnormal, and degraded modes of ATC operations. A total of eight functional hazards were identified based on the analysis of four use cases. The safety assessment was supported by top-down and bottom-up modelling and analysis of the causes of hazards to derive system design requirements for the purposes of mitigating the hazards. Assessment of achieving the specified design requirements was supported by evidence generated from two real-time simulations with pre-industrial ASR prototypes in approach and en-route operational environments. The simulations, focusing especially on the safety aspects of ASR application, also validated the hypotheses that ASR reduces controllers’ workload and increases situational awareness. The missing validation element, i.e., an analysis of the safety effects of ASR in ATC, is the focus of this paper. As a result of the safety assessment activities, mitigations were derived for each hazard, demonstrating that the use of ASR does not increase safety risks and is, therefore, ready for industrialization.

Effects of Language Ontology on Transatlantic Automatic Speech Understanding Research Collaboration in the Air Traffic Management Domain

As researchers around the globe develop applications for the use of Automatic Speech Recognition and Understanding (ASRU) in the Air Traffic Management (ATM) domain, Air Traffic Control (ATC) language ontologies will play a critical role in enabling research collaboration. The MITRE Corporation (MITRE) and the German Aerospace Center (DLR), having independently developed ATC language ontologies for specific applications, recently compared these ontologies to identify opportunities for improvement and harmonization. This paper extends the topic in two ways. First, this paper describes the specific ways in which ontologies facilitate the sharing of and collaboration on data, models, algorithms, metrics, and applications in the ATM domain. Second, this paper provides comparative analysis of word frequencies in ATC speech in the United States and Europe to illustrate that, whereas methods and tools for evaluating ASRU applications can be shared across researchers, the specific models would not work well between regions due to differences in the underlying corpus data.

Dictating translations with automatic speech recognition: Effects on translators' performance.

Technologies can greatly improve translators' productivity and reduce their workload. Previous research has found that the use of automatic speech recognition (ASR) tools for dictating translations can increase productivity. However, these studies often had small sample sizes and did not consider other important aspects of translators' performance, such as translation quality and cognitive effort. This study aims to investigate the impact of text input method on translators' performance in terms of task duration, time allocation, editing operations, cognitive effort, and translation quality, as well as whether text difficulty affects these factors. To do this, 60 Chinese translation trainees were randomly assigned to either a dictation group or a typing group, and completed two English-Chinese translations of varying levels of source-text difficulty. Data were collected using keylogging, subjective ratings, screen recording, and a questionnaire. The results showed that using ASR reduced the typing effort of participants without negatively affecting translation quality, but did not save time or reduce cognitive effort. No effect of text difficulty was observed. Analysis of the revisions made by the dictation group and the results of the post-test questionnaire provide insights into how ASR systems can be optimized for translation purposes.

Consensus automatic speech recognition (CASR) in the California Cognitive Assessment Battery (CCAB).

Recent reports have investigated the use of automatic speech recognition (ASR) to analyze and score verbal responses in cognitive tests. ASR scoring is objective, permits the efficient computerized administration of verbal tests, and generates timestamps that enable the detailed temporal analysis of responses. However, ASR transcription accuracy varies by engine, task, and participant, and ASR can incorrectly score responses from participants with atypical speech patterns. Here we describe the speech-transcription pipeline of the California Cognitive Assessment Battery (CCAB), which incorporates consensus ASR (CASR) to produce more accurate transcripts than possible with any single ASR engine. We also developed a Transcript Review Tool (TRT) which facilitates the manual correction of mis-transcribed words in problem subjects. Figure 1 shows the CCAB speech transcription pipeline. Realtime ASR transcriptions are obtained along with the transcriptions of the digital recordings of responses using six cloud-based ASR engines (e.g., Google, etc.). Individual transcripts are then combined to produce a "consensus" transcript, and a transcription confidence measure based primarily on the agreement between ASR engines (Figure 2). If needed, "consensus" transcripts can be manually corrected using the Transcript Review Tool which enables the review of all words or just those words below a predefined CASR confidence threshold (Figure 3). ASR transcriptions were obtained from 442 healthy adults (mean age = 65.1 ±14.4) who each underwent three days of cognitive testing that included 25 verbal tests. In all, approximately 276 hours of speech were transcribed. Preliminary analyses show that CASR transcription accuracy surpassed 99% for tests with limited response sets (e.g., digit span, verbal list learning, face-name binding, etc.) and exceeded 95% for discursive speech tests (e.g., picture description and logical memory). CASR transcription is more accurate than that of any single ASR engine. When combined with the TRT, "consensus" ASR can produce error-free, timestamped transcripts that enable the detailed analysis of verbal responses from older individuals at risk of cognitive decline.

A Study of Interpreter Training in the Changing Digital Environment : Focused on the use of Automatic Speech Recognition

A Study of Interpreter Training in the Changing Digital Environment : Focused on the use of Automatic Speech Recognition

Automatic Assessment of Intelligibility in Noise in Parkinson Disease: Validation Study.

Most individuals with Parkinson disease (PD) experience a degradation in their speech intelligibility. Research on the use of automatic speech recognition (ASR) to assess intelligibility is still sparse, especially when trying to replicate communication challenges in real-life conditions (ie, noisy backgrounds). Developing technologies to automatically measure intelligibility in noise can ultimately assist patients in self-managing their voice changes due to the disease. The goal of this study was to pilot-test and validate the use of a customized web-based app to assess speech intelligibility in noise in individuals with dysarthria associated with PD. In total, 20 individuals with dysarthria associated with PD and 20 healthy controls (HCs) recorded a set of sentences using their phones. The Google Cloud ASR API was used to automatically transcribe the speakers' sentences. An algorithm was created to embed speakers' sentences in +6-dB signal-to-noise multitalker babble. Results from ASR performance were compared to those from 30 listeners who orthographically transcribed the same set of sentences. Data were reduced into a single event, defined as a success if the artificial intelligence (AI) system transcribed a random speaker or sentence as well or better than the average of 3 randomly chosen human listeners. These data were further analyzed by logistic regression to assess whether AI success differed by speaker group (HCs or speakers with dysarthria) or was affected by sentence length. A discriminant analysis was conducted on the human listener data and AI transcriber data independently to compare the ability of each data set to discriminate between HCs and speakers with dysarthria. The data analysis indicated a 0.8 probability (95% CI 0.65-0.91) that AI performance would be as good or better than the average human listener. AI transcriber success probability was not found to be dependent on speaker group. AI transcriber success was found to decrease with sentence length, losing an estimated 0.03 probability of transcribing as well as the average human listener for each word increase in sentence length. The AI transcriber data were found to offer the same discrimination of speakers into categories (HCs and speakers with dysarthria) as the human listener data. ASR has the potential to assess intelligibility in noise in speakers with dysarthria associated with PD. Our results hold promise for the use of AI with this clinical population, although a full range of speech severity needs to be evaluated in future work, as well as the effect of different speaking tasks on ASR.

Self-conducted speech audiometry using automatic speech recognition: Simulation results for listeners with hearing loss

Speech-in-noise tests are an important tool for assessing hearing impairment, the successful fitting of hearing aids, as well as for research in psychoacoustics. An important drawback of many speech-based tests is the requirement of an expert to be present during the measurement, in order to assess the listener’s performance. This drawback may be largely overcome through the use of automatic speech recognition (ASR), which utilizes automatic response logging. However, such an unsupervised system may reduce the accuracy due to the introduction of potential errors. In this study, two different ASR systems are compared for automated testing: A system with a feed-forward deep neural network (DNN) from a previous study (Ooster et al., 2018), as well as a state-of-the-art system utilizing a time-delay neural network (TDNN). The dynamic measurement procedure of the speech intelligibility test was simulated considering the subjects’ hearing loss and selecting from real recordings of test participants. The ASR systems’ performance is investigated based on responses of 73 listeners, ranging from normal-hearing to severely hearing-impaired as well as read speech from cochlear implant listeners. The feed-forward DNN produced accurate testing results for NH and unaided HI listeners but a decreased measurement accuracy was found in the simulation of the adaptive measurement procedure when considering aided severely HI listeners, recorded in noisy environments with a loudspeaker setup. The TDNN system produces error rates of 0.6% and 3.0% for deletion and insertion errors, respectively. We estimate that the SRT deviation with this system is below 1.38 dB for 95% of the users. This result indicates that a robust unsupervised conduction of the matrix sentence test is possible with a similar accuracy as with a human supervisor even when considering noisy conditions and altered or disordered speech from elderly severely HI listeners and listeners with a CI.

SoK: A Modularized Approach to Study the Security of Automatic Speech Recognition Systems

With the wide use of Automatic Speech Recognition (ASR) in applications such as human machine interaction, simultaneous interpretation, audio transcription, and so on, its security protection becomes increasingly important. Although recent studies have brought to light the weaknesses of popular ASR systems that enable out-of-band signal attack, adversarial attack, and so on, and further proposed various remedies (signal smoothing, adversarial training, etc.), a systematic understanding of ASR security (both attacks and defenses) is still missing, especially on how realistic such threats are and how general existing protection could be. In this article, we present our systematization of knowledge for ASR security and provide a comprehensive taxonomy for existing work based on a modularized workflow. More importantly, we align the research in this domain with that on security in Image Recognition System (IRS), which has been extensively studied, using the domain knowledge in the latter to help understand where we stand in the former. Generally, both IRS and ASR are perceptual systems. Their similarities allow us to systematically study existing literature in ASR security based on the spectrum of attacks and defense solutions proposed for IRS, and pinpoint the directions of more advanced attacks and the directions potentially leading to more effective protection in ASR. In contrast, their differences, especially the complexity of ASR compared with IRS, help us learn unique challenges and opportunities in ASR security. Particularly, our experimental study shows that transfer attacks across ASR models are feasible, even in the absence of knowledge about models (even their types) and training data.

The Potential of ASR for Improving English Pronunciation: A Review

To pronounce well is a complex task, requiring students not only to possess knowledge of the appropriate sounds in a given context, but also to learn to use their vocal apparatus to make those sounds, equipped with extensive practice and feedback. Students in these situations require autonomous monitoring experiences to receive tailored feedback.. One of the technological tools learners can use to improve their pronunciation is Automatic Speech Recognition (ASR). This provides learners with individual practice and feedback to assist them s to accomplish their language goals. This study examines the database of research on the use of ASR in pronunciation instruction and learning available on Google Scholar, Springer Link, Education Resources Information Center (ERIC), Taylor & Francis Online and Directory of Open Access Journal (DOAJ). To help the process of identification, some procedures and criteria were employedThe results revealed that ten articles met the eligibility criteria. The procedures of utilizing ASR to improve students' pronunciation competency are then discussed in this study. 
 Keywords: teaching speaking, teaching pronunciation, Automatic Speech Recognition

OPRA-RS: A Hearing-Aid Fitting Method Based on Automatic Speech Recognition and Random Search.

Hearing-aid (HA) prescription rules (such as NAL-NL2, DSL-v5, and CAM2) are used by HA audiologists to define initial HA settings (e.g., insertion gains, IGs) for patients. This initial fitting is later individually adjusted for each patient to improve clinical outcomes in terms of speech intelligibility and listening comfort. During this fine-tuning stage, speech-intelligibility tests are often carried out with the patient to assess the benefits associated with different HA settings. As these tests tend to be time-consuming and performance on them depends on the patient's level of fatigue and familiarity with the test material, only a limited number of HA settings can be explored. Consequently, it is likely that a suboptimal fitting is used for the patient. Recent studies have shown that automatic speech recognition (ASR) can be used to predict the effects of IGs on speech intelligibility for patients with age-related hearing loss (ARHL). The aim of the present study was to extend this approach by optimizing, in addition to IGs, compression thresholds (CTs). However, increasing the number of parameters to be fitted increases exponentially the number of configurations to be assessed. To limit the number of HA settings to be tested, three random-search (RS) genetic algorithms were used. The resulting new HA fitting method, combining ASR and RS, is referred to as “objective prescription rule based on ASR and random search" (OPRA-RS). Optimal HA settings were computed for 12 audiograms, representing average and individual audiometric profiles typical for various levels of ARHL severity, and associated ASR performances were compared to those obtained with the settings recommended by CAM2. Each RS algorithm was run twice to assess its reliability. For all RS algorithms, ASR scores obtained with OPRA-RS were significantly higher than those associated with CAM2. Each RS algorithm converged on similar optimal HA settings across repetitions. However, significant differences were observed between RS algorithms in terms of maximum ASR performance and processing costs. These promising results open the way to the use of ASR and RS algorithms for the fine-tuning of HAs with potential speech-intelligibility benefits for the patient.

Automatic speech recognition and text-to-speech technologies for L2 pronunciation improvement

This paper presents a reflection on two technologies – automatic speech recognition (ASR) and Text-to-Speech (TTS) – to improve learners’ pronunciation, aiming for successful spoken communication. It sheds some light on the practical usage of these technologies, demonstrating their effectiveness, qualities, and limitations to assist teachers in deciding the most efficient digital resources applied to their students’ needs. A review of literature on previous empirical studies was carried out, with quantitative and/or qualitative studies conducted by researchers in the field, investigating teachers’ and learners' perceptions and the use of ASR and TTS as a pedagogical tool for pronunciation practice. As a result, it was concluded that a) the presented resources seem to have the potential to enhance pronunciation practice, both in terms of perception and production; b) technology can result in considerable benefits to learners, mainly as a supplement to pronunciation teaching; and c) the use of these digital resources is a way of giving learners the opportunity to focus on their specific difficulties and receive personalized feedback while becoming more autonomous in their learning process.

Microscopic and Blind Prediction of Speech Intelligibility: Theory and Practice.

Being able to estimate speech intelligibility without the need for listening tests would confer great benefits for a wide range of speech processing applications. Many attempts have therefore been made to introduce an objective, and ideally referencefree measure for this purpose. Most works analyze speech intelligibility prediction (SIP) methods from a macroscopic point of view, averaging over longer time spans. This paper, in contrast, presents a theoretical framework for the microscopic evaluation of SIP methods. Within our framework, a Statistically estimated Accuracy based on Theory (StAT) is derived, which numerically quantifies the statistical limitations inherent in microscopic SIP. A state-of-the-art approach to microscopic SIP, namely, the use of automatic speech recognition (ASR) to directly predict listening test results, is evaluated within this framework. The practical results are in good agreement with the theory. As the final contribution, a fully blind DIscriminative Speech intelligibility Predictor (DISP) is introduced and is also evaluated within the StAT framework. It is shown that this novel, blind estimator can predict intelligibility as well as-and often even with better accuracy than-the non-blind ASR-based approach, and that its results are again in good agreement with its theoretically derived performance potential.

Student Perceptions of Mobile Automated Speech Recognition for Pronunciation Study and Testing

This study utilized Automated Speech Recognition technology to determine the potential utility and acceptance of such technology in the English as a Foreign Language classroom. Learners were made aware of the Automatic Speech Recognition potential of their mobile devices and provided with some direction in, and incentive for, its use. Participants were then scored on their assessment of the technology according to the Technology Acceptance Model. Participants showed a marked appreciation for the ease and utility of the technology with over 72% agreeing that the technology was both accessible and useful. Support for the use of Automatic Speech Recognition as a testing method was somewhat mixed, with 75% of participants agreeing that the testing was fair, but only 60% reporting that they felt they did well on the test. As a secondary point of interest, this study examined the potential use of Automatic Speech Recognition technology for teaching and testing pronunciation.

Representation transfer learning from deep end-to-end speech recognition networks for the classification of health states from speech

Representation transfer learning has been widely used across a range of machine learning tasks. One such notable approach seen in the speech literature is the use of Convolutional Neural Networks, pre-trained for image classification tasks, to extract features from spectrograms of speech signals. Interestingly, despite the strong performance of such approaches, there have been minimal research efforts exploring the suitability of using speech-specific networks to perform feature extraction. In this regard, a novel feature representation learning framework is presented herein. This approach is comprising the use of Automatic Speech Recognition (ASR) deep neural networks as feature extractors, the fusion of several extracted feature representations using Compact Bilinear Pooling (CBP), and finally inference via a specially optimised Recurrent Neural Network (RNN) classifier. To determine the usefulness of these feature representations, they are comprehensively tested on two representative speech-health classification tasks, namely the food-type being eaten and speaker intoxication. Key results indicate the promise of the extracted features, demonstrating comparable results to other state-of-the-art approaches in the literature.

Effects of Automatic Speech Recognition Software on Pronunciation for Adults With Different Learning Styles

This study investigated how learning styles (visual/verbal) and the use of Automatic Speech Recognition (ASR) software affect English as a Second Language adult learners’ improvement during a 12-week course focusing on pronunciation. In the control group (n = 28), the teacher corrected and gave feedback on the adult learners’ pronunciation; experimental group 1 (n = 33) used dictation ASR along with peers’ correction; and experimental group 2 (n = 31) used dictation ASR alone. Their pre- and post-tests on pronunciation in reading tasks and live conversation were analyzed with their learning styles taken into account, using 2-way ANCOVA. The results suggest that learning styles made a significant difference in the pronunciation performance of the reading task in all groups. Visual style learners outperformed verbal style learners in the reading task. The combination of ASR and peer correction yielded the highest improvement in both reading tasks and live conversation.

Automatic speech recognition in the operating room – An essential contemporary tool or a redundant gadget? A survey evaluation among physicians in form of a qualitative study

IntroductionFor decades, automatic speech recognition (ASR) has been the subject of research and its range of applications broadened. Presently, ASR among physicians is mainly used to convert speech into text but not to implement instructions in the operating room (OR). This study aimed to evaluate physicians of different surgical professions on their personal experience and posture towards ASR. MethodsA 16-item survey was distributed electronically to hospitals and outpatient clinics in southern Germany addressing physicians on the potential applications of ASR in the OR. ResultsThe survey was responded by 185 of 2693 physicians (response rate: 6.9%) with a mean age of 41.8 ± 9.8 years. ASR is desirable in the OR regardless of the field of speciality (93.7%). While only 2.7% have used ASR, 87.9% evaluate its future potential as high. 91.0% of those working in a university hospital would consider testing ASR, while 67.5% of those in non-university hospitals and practices (p = 0.001). 90.1% of responders of strictly surgical specialities see potential in ASR while 73.7% in non-surgical specialities evaluate its future potential as high (p = 0.01). 58.3% of those over the age of 60 consider the use of ASR without a headset to be imaginable, while 96.3% among those under the age of 60. There were no statistically significant differences regarding sex and professional position. ConclusionForeseeably, ASR is anticipated to be integrated into ORs and valued at a high market potential. Our study provides information about physicians’ individual preferences from various surgical disciplines regarding ASR.

Vocabulary Size Influences Spontaneous Speech in Native Language Users: Validating the Use of Automatic Speech Recognition in Individual Differences Research.

Previous research has shown that vocabulary size affects performance on laboratory word production tasks. Individuals who know many words show faster lexical access and retrieve more words belonging to pre-specified categories than individuals who know fewer words. The present study examined the relationship between receptive vocabulary size and speaking skills as assessed in a natural sentence production task. We asked whether measures derived from spontaneous responses to everyday questions correlate with the size of participants' vocabulary. Moreover, we assessed the suitability of automatic speech recognition (ASR) for the analysis of participants' responses in complex language production data. We found that vocabulary size predicted indices of spontaneous speech: individuals with a larger vocabulary produced more words and had a higher speech-silence ratio compared to individuals with a smaller vocabulary. Importantly, these relationships were reliably identified using manual and automated transcription methods. Taken together, our results suggest that spontaneous speech elicitation is a useful method to investigate natural language production and that automatic speech recognition can alleviate the burden of labor-intensive speech transcription.

Speech Audiometry at Home: Automated Listening Tests via Smart Speakers With Normal-Hearing and Hearing-Impaired Listeners.

Speech audiometry in noise based on sentence tests is an important diagnostic tool to assess listeners’ speech recognition threshold (SRT), i.e., the signal-to-noise ratio corresponding to 50% intelligibility. The clinical standard measurement procedure requires a professional experimenter to record and evaluate the response (expert-conducted speech audiometry). The use of automatic speech recognition enables self-conducted measurements with an easy-to-use speech-based interface. This article compares self-conducted SRT measurements using smart speakers with expert-conducted laboratory measurements. With smart speakers, there is no control over the absolute presentation level, potential errors from the automated response logging, and room acoustics. We investigate the differences between highly controlled measurements in the laboratory and smart speaker-based tests for young normal-hearing (NH) listeners as well as for elderly NH, mildly and moderately hearing-impaired listeners in low, medium, and highly reverberant room acoustics. For the smart speaker setup, we observe an overall bias in the SRT result that depends on the hearing loss. The bias ranges from +0.7 dB for elderly moderately hearing-impaired listeners to +2.2 dB for young NH listeners. The intrasubject standard deviation is close to the clinical standard deviation (0.57/0.69 dB for the young/elderly NH compared with 0.5 dB observed for clinical tests and 0.93/1.09 dB for the mild/moderate hearing-impaired listeners compared with 0.9 dB). For detecting a clinically elevated SRT, the speech-based test achieves an area under the curve value of 0.95 and therefore seems promising for complementing clinical measurements.

Automatic perceptual judgment using neural networks

The aim of the present study is to investigate the performance of automatic perceptual judgment models built with neural networks. In previous studies, Franco et al. (1997) used HMM-derived scores based on posterior probabilities of phone segments, demonstrating a high correlation with human raters. Deville et al. (1999) also used an HMM/ANN recognition approach, and showed how the results of automatic speech recognition can be used for perceptual judgments. However, since most previous studies made use of automatic speech recognition in their analysis, the present study provides a different approach: using features and raw data. Native speakers of English will listen to English sentences produced by native and non-native speakers of English, transcribe what they heard, and respond to one of three perceptual judgements: foreign-accentedness, fluency, and comprehensibility. The data will be fed into prediction models in three different ways; one with annotated features (pauses, durations, etc), another with Mel Frequency Cepstral Coefficients (MFCC), and the other with Mel-spectrograms. The performance of the models will be measured by analyzing the correlation between the judgments by models and by human raters. The preliminary results of this study will be used to build more accurate automatic proficiency judgment models.

Enhanced Automatic Speech Recognition System Based on Enhancing Power-Normalized Cepstral Coefficients

Many new consumer applications are based on the use of automatic speech recognition (ASR) systems, such as voice command interfaces, speech-to-text applications, and data entry processes. Although ASR systems have remarkably improved in recent decades, the speech recognition system performance still significantly degrades in the presence of noisy environments. Developing a robust ASR system that can work in real-world noise and other acoustic distorting conditions is an attractive research topic. Many advanced algorithms have been developed in the literature to deal with this problem; most of these algorithms are based on modeling the behavior of the human auditory system with perceived noisy speech. In this research, the power-normalized cepstral coefficient (PNCC) system is modified to increase robustness against the different types of environmental noises, where a new technique based on gammatone channel filtering combined with channel bias minimization is used to suppress the noise effects. The TIDIGITS database is utilized to evaluate the performance of the proposed system in comparison to the state-of-the-art techniques in the presence of additive white Gaussian noise (AWGN) and seven different types of environmental noises. In this research, one word is recognized from a set containing 11 possibilities only. The experimental results showed that the proposed method provides significant improvements in the recognition accuracy at low signal to noise ratios (SNR). In the case of subway noise at SNR = 5 dB, the proposed method outperforms the mel-frequency cepstral coefficient (MFCC) and relative spectral (RASTA)–perceptual linear predictive (PLP) methods by 55% and 47%, respectively. Moreover, the recognition rate of the proposed method is higher than the gammatone frequency cepstral coefficient (GFCC) and PNCC methods in the case of car noise. It is enhanced by 40% in comparison to the GFCC method at SNR 0dB, while it is improved by 20% in comparison to the PNCC method at SNR −5dB.