Automatic Speech Recognition Service Research Articles

Measuring the Accuracy of Automatic Speech Recognition Solutions

For d/Deaf and hard of hearing (DHH) people, captioning is an essential accessibility tool. Significant developments in artificial intelligence mean that automatic speech recognition (ASR) is now a part of many popular applications. This makes creating captions easy and broadly available—but transcription needs high levels of accuracy to be accessible. Scientific publications and industry report very low error rates, claiming that artificial intelligence has reached human parity or even outperforms manual transcription. At the same time, the DHH community reports serious issues with the accuracy and reliability of ASR. There seems to be a mismatch between technical innovations and the real-life experience for people who depend on transcription. Independent and comprehensive data is needed to capture the state of ASR. We measured the performance of 11 common ASR services with recordings of Higher Education lectures. We evaluated the influence of technical conditions like streaming, the use of vocabularies, and differences between languages. Our results show that accuracy ranges widely between vendors and for the individual audio samples. We also measured a significant lower quality for streaming ASR, which is used for live events. Our study shows that despite the recent improvements of ASR, common services lack reliability in accuracy.

Using HIPAA (Health Insurance Portability and Accountability Act)-Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study.

Automatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription services among different diagnostic groups in a mental health setting. There has also been little research into the types of words ASR transcriptions mistakenly generate or omit. This study compared the WER of 3 ASR transcription services (Amazon Transcribe [Amazon.com, Inc], Zoom-Otter AI [Zoom Video Communications, Inc], and Whisper [OpenAI Inc]) in interviews across 2 different clinical categories (controls and participants experiencing a variety of mental health conditions). These ASR transcription services were also compared with a commercial human transcription service, Rev (Rev.Com, Inc). Words that were either included or excluded by the error in the transcripts were systematically analyzed by their Linguistic Inquiry and Word Count categories. Participants completed a 1-time research psychiatric interview, which was recorded on a secure server. Transcriptions created by the research team were used as the gold standard from which WER was calculated. The interviewees were categorized into either the control group (n=18) or the mental health condition group (n=47) using the Mini-International Neuropsychiatric Interview. The total sample included 65 participants. Brunner-Munzel tests were used for comparing independent sets, such as the diagnostic groupings, and Wilcoxon signed rank tests were used for correlated samples when comparing the total sample between different transcription services. There were significant differences between each ASR transcription service's WER (P<.001). Amazon Transcribe's output exhibited significantly lower WERs compared with the Zoom-Otter AI's and Whisper's ASR. ASR performances did not significantly differ across the 2 different clinical categories within each service (P>.05). A comparison between the human transcription service output from Rev and the best-performing ASR (Amazon Transcribe) demonstrated a significant difference (P<.001), with Rev having a slightly lower median WER (7.6%, IQR 5.4%-11.35 vs 8.9%, IQR 6.9%-11.6%). Heat maps and spider plots were used to visualize the most common errors in Linguistic Inquiry and Word Count categories, which were found to be within 3 overarching categories: Conversation, Cognition, and Function. Overall, consistent with previous literature, our results suggest that the WER between manual and automated transcription services may be narrowing as ASR services advance. These advances, coupled with decreased cost and time in receiving transcriptions, may make ASR transcriptions a more viable option within health care settings. However, more research is required to determine if errors in specific types of words impact the analysis and usability of these transcriptions, particularly for specific applications and in a variety of populations in terms of clinical diagnosis, literacy level, accent, and cultural origin.

Performance Disparities between Accents in Automatic Speech Recognition (Student Abstract)

In this work, we expand the discussion of bias in Automatic Speech Recognition (ASR) through a large-scale audit. Using a large and global data set of speech, we perform an audit of some of the most popular English ASR services. We show that, even when controlling for multiple linguistic covariates, ASR service performance has a statistically significant relationship to the political alignment of the speaker's birth country with respect to the United States' geopolitical power.

Automatic speech recognition in neurodegenerative disease

Automatic speech recognition (ASR) could potentially improve communication by providing transcriptions of speech in real time. ASR is particularly useful for people with progressive disorders that lead to reduced speech intelligibility or difficulties performing motor tasks. ASR services are usually trained on healthy speech and may not be optimized for impaired speech, creating a barrier for accessing augmented assistance devices. We tested the performance of three state-of-the-art ASR platforms on two groups of people with neurodegenerative disease and healthy controls. We further examined individual differences that may explain errors in ASR services within groups, such as age and sex. Speakers were recorded while reading a standard text. Speech was elicited from individuals with multiple sclerosis, Friedreich’s ataxia, and healthy controls. Recordings were manually transcribed and compared to ASR transcriptions using Amazon Web Services, Google Cloud, and IBM Watson. Accuracy was measured as the proportion of words that were correctly classified. ASR accuracy was higher for controls than clinical groups, and higher for multiple sclerosis compared to Friedreich’s ataxia for all ASR services. Amazon Web Services and Google Cloud yielded higher accuracy than IBM Watson. ASR accuracy decreased with increased disease duration. Age and sex did not significantly affect ASR accuracy. ASR faces challenges for people with neuromuscular disorders. Until improvements are made in recognizing less intelligible speech, the true value of ASR for people requiring augmented assistance devices and alternative communication remains unrealized. We suggest potential methods to improve ASR for those with impaired speech.

Automatized analysis of children's exposure to child-directed speech in reschool settings: Validation and application.

The present study explored whether a tool for automatic detection and recognition of interactions and child-directed speech (CDS) in preschool classrooms could be developed, validated, and applied to non-coded video recordings representing children's classroom experiences. Using first-person video recordings collected by 13 preschool children during a morning in their classrooms, we extracted high-level audiovisual features from recordings using automatic speech recognition and computer vision services from a cloud computing provider. Using manual coding for interactions and transcriptions of CDS as reference, we trained and tested supervised classifiers and linear mappings to measure five variables of interest. We show that the supervised classifiers trained with speech activity, proximity, and high-level facial features achieve adequate accuracy in detecting interactions. Furthermore, in combination with an automatic speech recognition service, the supervised classifier achieved error rates for CDS measures that are in line with other open-source automatic decoding tools in early childhood settings. Finally, we demonstrate our tool's applicability by using it to automatically code and transcribe children's interactions and CDS exposure vertically within a classroom day (morning to afternoon) and horizontally over time (fall to winter). Developing and scaling tools for automatized capture of children's interactions with others in the preschool classroom, as well as exposure to CDS, may revolutionize scientific efforts to identify precise mechanisms that foster young children's language development.

Knowledge Transfer Using User-Generated Data within Real-Time Cloud Services

When automatic speech recognition (ASR) is provided as a cloud service, it is easy to collect voice and application domain data from users. Harnessing these data will facilitate the provision of more personalized services. In this paper, we demonstrate our transfer learning-based knowledge service that built with the user-generated data collected through our novel system that deliveries personalized ASR service. First, we discuss the motivation, challenges, and prospects of building up such a knowledge-based service-oriented system. Second, we present a Quadruple Transfer Learning (QTL) method that can learn a classification model from a source domain and transfer it to a target domain. Third, we provide an overview architecture of our novel system that collects voice data from mobile users, labels the data via crowdsourcing, utilises these collected user-generated data to train different machine learning models, and delivers the personalised real-time cloud services. Finally, we use the E-Book data collected from our system to train classification models and apply them in the smart TV domain, and the experimental results show that our QTL method is effective in two classification tasks, which confirms that the knowledge transfer provides a value-added service for the upper-layer mobile applications in different domains.

Enhancing listening capability of humanoid robot by reduction of stationary ego‐noise

Speech interfaces for household robots utilizing third‐party automatic speech recognition (ASR) services face the challenge of overcoming stationary ego‐noise that decreases ASR accuracy. Previous studies on signal processing have proposed numerous noise reduction methods that increase the signal‐to‐noise ratio of speech audio and subjective speech clarity. However, severe limitations on the cost of hardware of household robots and the use of closed ‘black box’ ASR services require us to re‐examine the efficacy of noise reduction methods in this context. Here we compare the effect of several basic noise filters on the performance of ASR services when speech sounds include the stationary ego‐noise of a humanoid Pepper robot. The result revealed that a spectrum subtraction filter improves the accuracy of ASR services best. We also demonstrate that the filter improves ASR performance on an actual Pepper robot system. This study not only provides practical knowledge on the selection of noise filters for a robot system but also discusses further improvements to the listening capabilities of the robot utilizing ASR. © 2019 Institute of Electrical Engineers of Japan. Published by John Wiley &amp; Sons, Inc.

V-Speech

Smart glasses are often used in public environments or industrial scenarios that are relatively noisy. Background noise and sound from competing speakers deteriorate voice communication or performance of automatic speech recognition (ASR). Typically, signal processing techniques are used to reduce noise and enhance voice quality, but they have limitations in performance, hardware and/or computing resources. Voice capturing techniques using bone conducting on the head have been proposed in some experimental and commercial devices, with good robustness against environmental noise, but limited by signal distortions inherent to the capturing method. We present V-Speech, a novel sensing and signal processing solution that enables speech recognition and human-to-human communication in very noisy environments. It captures the voice signal with a vibration sensor located in the nasal pads of smart glasses and performs a transformation to the sensor signal in order to mimic that of a regular microphone in low noise conditions. The signal transformation is key, as it eliminates the "nasal distortion" that is introduced for nasal phonemes in the speech induced vibrations of the nasal bone. The output of V-Speech has low noise, sounds natural, and can be used in voice communication or as input to an off-the-shelf ASR service. We evaluated V-Speech in noise-free and noisy conditions with 30 volunteer speakers uttering 145 phrases and validated its performance on ASR engines and with assessments of voice quality using the Perceptual Evaluation of Speech Quality (PESQ) metric. The results show in extreme noise conditions a mean improvement of 50% for Word Error Rate (WER), and 1.0 on a scale of 5.0 for PESQ. In addition, real life recordings were made under various representative noise conditions, some with sound pressure levels of 93 dBA, which require hearing protection. Subjective listening tests were conducted according to a modified ITU P.835 approach to determine intelligibility, naturalness and overall quality. Under these extreme conditions, where V-Speech achieved 30 dB SNR, subjective results show the speech is intelligible, and the naturalness of the speech is rated as fair to good. This enables clear voice communication in challenging work environments, for example in places with industrial, factory, mining and construction noise. With our proposed smart switching technique between a regular microphone signal and V-Speech, the optimal quality can be maintained from low to high noise conditions.

System and method for providing remote automatic speech recognition services via a packet network

A system and method of operating an automatic speech recognition service using a client-server architecture is used to make ASR services accessible at a client location remote from the location of the main ASR engine. The present invention utilizes client-server communications over a packet network, such as the Internet, where the ASR server receives a grammar from the client, receives information representing speech from the client, performs speech recognition, and returns information based upon the recognized speech to the client.