Automatic Speech Recognition Technology Research Articles

How People Living With Amyotrophic Lateral Sclerosis Use Personalized Automatic Speech Recognition Technology to Support Communication.

Amyotrophic lateral sclerosis (ALS) is a progressive, ultimately fatal disease causing progressive muscular weakness. Most people living with ALS (plwALS) experience dysarthria, eventually becoming unable to communicate using natural speech. Many wish to use speech for as long as possible. Personalized automated speech recognition (ASR) model technology, such as Google's Project Relate, is argued to better recognize speech with dysarthria, supporting maintenance of understanding through real-time captioning. The objectives of this study are how plwALS and communication partners use Relate in everyday conversation over a period of up to 12 months and how it may change with any decline in speech over time. This study videoed interactions between three plwALS and communication partners. We assessed ASR caption accuracy and how well they preserved meaning. Conversation analysis was used to identify participants' own organizational practices in the accomplishment of interaction. Thematic analysis was used to understand better the participants' experiences of using ASR captions. All plwALS reported lower-than-expected ASR accuracy when used in conversation and felt ASR captioning was only useful in certain contexts. All participants liked the concept of live captioning and were hopeful that future improvements to ASR accuracy may support their communication in everyday life. Training is needed on best practices for customization and practical use of ASR technology and for the limitations of ASR in conversational settings. Support is needed for those less confident with technology and to reduce misplaced allocation of ownership of captioning errors, risking negative effects on psychological well-being.

Monitoring student behavior in autonomous automatic speech recognition-based pronunciation practice

A growing number of studies are investigating the role of Automatic Speech Recognition (ASR) technology in second language (L2) pronunciation learning, highlighting its potential to promote autonomous learning and enhance the development of L2 pronunciation. However, how learners interact with ASR systems remains largely unknown. The current study examined the behavior of 46 English-as-a-foreign language learners over six sessions of autonomous ASR-based pronunciation practice. It also explored potential relationship between learners' behavior and individual differences in motivation and pronunciation attitude. Analyses of the learners' screen recordings shed light on their interaction with the ASR output, including the proportion of words successfully transcribed on the first attempt and unsuccessful cases requiring subsequent attempts—whether repaired, ignored, or remaining unrepaired despite several attempts. Results from motivation and attitude questionnaires revealed that learners with a higher pronunciation attitude at the study's onset were more likely to reattempt a word not successfully transcribed by the ASR system. Motivation, however, was not found to be linked to learner behavior. The findings contribute to the expanding research on ASR technology in L2 pronunciation learning by emphasizing the influence of learner behaviors and attitudes, and providing important insights for L2 practitioners and researchers.

Advancing Accessibility through Automatic Speech Recognition and NLP Integration

Abstract: The integration of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) technologies has the potential to revolutionize accessibility and inclusive communication. This article explores the fundamentals of ASR, including acoustic modeling, language modeling, and speech signal processing techniques, and discusses the challenges posed by language diversity in developing accurate ASR systems. The advent of ASR technology has opened up numerous possibilities for enhancing accessibility across various domains, such as assistive technology, education, healthcare, and media. The integration of ASR with NLP techniques enables the processing and analysis of spoken language data, leading to the development of voice-enabled virtual assistants, conversational AI systems, and cross-lingual communication tools. However, several challenges remain, including the need for robust and accurate ASR systems, privacy and security concerns, and ethical considerations in the development and deployment of these technologies. The article also presents future directions, such as the integration of ASR and NLP with emotion recognition and sentiment analysis, advances in deep learning techniques, and the application of these technologies in healthcare and accessibility. Overall, the integration of ASR and NLP holds immense promise for creating more natural, empathetic, and inclusive communication systems, but their development and deployment must be approached with care to ensure fairness, transparency, and user privacy.

Intelligent Clinical Documentation: Harnessing Generative AI for Patient-Centric Clinical Note Generation

Comprehensive clinical documentation is crucial for effective healthcare delivery, yet it poses a significant burden on healthcare professionals, leading to burnout, increased medical errors, and compromised patient safety. This paper explores the potential of generative AI (Artificial Intelligence) to streamline the clinical documentation process, specifically focusing on generating SOAP (Subjective, Objective, Assessment, Plan) and BIRP (Behavior, Intervention, Response, Plan) notes. We present a case study demonstrating the application of natural language processing (NLP) and automatic speech recognition (ASR) technologies to transcribe patient-clinician interactions, coupled with advanced prompting techniques to generate draft clinical notes using large language models (LLMs). The study highlights the benefits of this approach, including time savings, improved documentation quality, and enhanced patient-centered care. Additionally, we discuss ethical considerations, such as maintaining patient confidentiality and addressing model biases, underscoring the need for responsible deployment of generative AI in healthcare settings. The findings suggest that generative AI has the potential to revolutionize clinical documentation practices, alleviating administrative burdens and enabling healthcare professionals to focus more on direct patient care.

Automatic speech recognition technology: History, applications and improvements

In todays world, automatic speech recognition(ASR) has been an important part of artificial intelligence. It has been recognized as an extremely difficult highly challenging high-tech topic. It mainly converts the vocabulary content in human speech into computer-readable input, which is generally understandable text content, and may also be binary encoding or character sequences. Since the 1950s, ASR has been continuously developing from simple systems for pronunciation of 10 English numbers to the rise of multiple frameworks and different neural networks. The process of ASR is constantly becoming diversified and specialized. Based on the analysis of existing literature, this article will briefly describe the history of speech recognition technology, the current development status of speech recognition, various applications in daily life and advanced areas, and methods for improvements. It indicates that nowadays automatic technology has become an essential part in peoples daily lives. Simple methods for eliminating echoes and noise to improve system performance and user experience are also an important part that should be considered in the use of ASR.

Linguistic analysis of human-computer interaction

This article reviews recent literature investigating speech variation in production and comprehension during spoken language communication between humans and devices. Human speech patterns toward voice-AI presents a test to our scientific understanding about speech communication and language use. First, work exploring how human-AI interactions are similar to, or different from, human-human interactions in the realm of speech variation is reviewed. In particular, we focus on studies examining how users adapt their speech when resolving linguistic misunderstandings by computers and when accommodating their speech toward devices. Next, we consider work that investigates how top-down factors in the interaction can influence users’ linguistic interpretations of speech produced by technological agents and how the ways in which speech is generated (via text-to-speech synthesis, TTS) and recognized (using automatic speech recognition technology, ASR) has an effect on communication. Throughout this review, we aim to bridge both HCI frameworks and theoretical linguistic models accounting for variation in human speech. We also highlight findings in this growing area that can provide insight to the cognitive and social representations underlying linguistic communication more broadly. Additionally, we touch on the implications of this line of work for addressing major societal issues in speech technology.

System dedicated to Polish Automatic Speech Recognition - overview of solutions

The paper presents the analysis of modern Artificial Intelligence algorithms for the automated system supporting human beings during their conversation in Polish language. Their task is to perform Automatic Speech Recognition (ASR) and process it further, for instance fill the computer-based form or perform the Natural Language Processing (NLP) to assign the conversation to one of predefined categories. The State-of-the-Art review is required to select the optimal set of tools to process speech in the difficult conditions, which degrade accuracy of ASR. The paper presents the top-level architecture of the system applicable for the task. Characteristics of Polish language are discussed. Next, existing ASR solutions and architectures with the End-To-End (E2E) deep neural network (DNN) based ASR models are presented in detail. Differences between Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN) and Transformers in the context of ASR technology are also discussed.

Video Summarization for Marathi Language

The Video Summarization Platform using Python Flask is a comprehensive tool designed to summarize Marathi and English videos while providing summaries in Hindi, Marathi, and English languages. Leveraging machine learning and natural language processing (NLP) techniques, this platform offers a sophisticated solution for efficiently extracting key information from videos. The platform begins by transcribing the audio content of the video into text using automatic speech recognition (ASR) technology. This transcription process ensures that the platform can accurately analyze and summarize the video's content. Next, the text is translated into the target languages, namely Hindi, Marathi, and English, enabling users from diverse linguistic backgrounds to access the summarized content. To generate concise and informative summaries, advanced NLP algorithm is applied. This algorithm analyze the transcribed text to identify the most significant phrases, sentences, and concepts. By considering factors such as keyword frequency, semantic relevance, and context, the platform effectively distils the video's content into digestible summaries. Additionally, machine learning models are employed to classify the type of video content. These models are trained on diverse datasets encompassing various video genres and topics. By recognizing patterns and features within the video content, the platform can accurately categorize videos into distinct types, such as news, interviews, tutorials, or entertainment. The platform's user interface, powered by Python Flask, offers a seamless experience for users to upload videos, select their preferred language for summarization, and receive concise summaries in their chosen languages. The intuitive design ensures accessibility and ease of use, catering to both novice and advanced users. Overall, the Video Summarization Platform serves as a valuable resource for individuals seeking efficient ways to consume multimedia content. Whether for educational, informational, or entertainment purposes, this platform empowers users to access summarized video content in multiple languages, facilitated by cutting-edge machine learning and NLP technologies. Key Words: Transcription, Marathi-speaking users, Marathi YouTube videos, video content, transcription, summary, translation, Natural Language Toolkit (NLTK), content comprehension, user interaction data, past summaries, recommendation

MDVC corpus: empowering Moroccan Darija speech recognition

Automatic speech recognition (ASR) technology has significantly transformed human-machine interactions, but it remains limited in its representation of diverse languages and dialects. Moroccan Darija, the lively Moroccan dialect, has long been underrepresented in the realm of language technology. To address this gap, we present a novel corpus of audio files accompanied by meticulously transcribed Moroccan Darija speech. The corpus comprises 1,000 hours of diverse content, featuring multiple Moroccan accents, extracted from 80 YouTube channels. To standardize the representation of Moroccan Darija in our corpus, we made efforts to establish consistent writing norms and conventions. In addition to the dataset creation, we applied fine-tuning using the Wav2Vec2 model on the Moroccan Darija voice corpus (MDVC) dataset achieving a remarkable word error rate (WER) of 9%. This article discusses the current state of Moroccan Darija research, highlighting the scarcity of resources and the need for robust ASR systems. Our contribution offers a valuable resource for researchers and developers, and by standardizing the Darija language, we strive to improve ASR system for this low resource language.

Customization of the ASR System for ATC Speech with Improved Fusion

In recent years, automatic speech recognition (ASR) technology has improved significantly. However, the training process for an ASR model is complex, involving large amounts of data and a large number of algorithms. The task of training a new model for air traffic control (ATC) is considerable, as it may require many researchers for its maintenance and upgrading. In this paper, we developed an improved fusion method that can adapt the language model (LM) in ASR to the domain of air traffic control. Instead of using vocabulary in traditional fusion, this method uses the ATC instructions to improve the LM. The perplexity shows that the LM of the improved fusion is much better than that of the use of vocabulary. With vocabulary fusion, the CER in the ATC corpus decreases from 0.3493 to 0.2876. The improved fusion reduces the CER of the ATC corpora from 0.3493 to 0.2761. Although there is only a difference of less than 2% between the two fusions, the perplexity shows that the LM of the improved fusion is much better.

ANALYSIS OF WHISPER AUTOMATIC SPEECH RECOGNITION PERFORMANCE ON LOW RESOURCE LANGUAGE

Implementing Automatic Speech Recognition Technology in daily life could give convenience to its users. However, speeches that can be recognized accurately by the ASR model right now are in languages considered high resources, like English. In previous research, a few regional languages like Javanese, Sundanese, Balinese and Btaknese are used in automatic speech recognition. This research aim is to improve speech recognition using the ASR model on low-resource language. The dataset used in this research is the Javanese dataset specifically because there is a high-quality Javanese speech dataset provided by previous research. The method used is fine-tuning the Whisper model which has been trained on 680,000 hours of multilingual voice data using a Javanese speech dataset. To reduce computation requirements, parameter efficient fine-tuning (PEFT) implemented in the fine-tuning process. The trainable parameter is reduced to <1% because the implementation of PEFT reduces the computation required by the model for fine-tuning. The best WER evaluation result is 13.77%, achieved by the fine-tuned Whisper large-v2 model compared to the base model of Whisper large-v2, which achieves 89.40% in WER evaluation. Performance improvement in WER evaluation showed that fine-tuning effectively improves the performance of the Whisper automatic speech recognition model on recognizing speeches in low-resource languages like the Javanese language compared to the Original Whisper model performance with minimal computational cost needed for fine-tuning large model.

Automatic Speech Recognition System to Record Progress Notes in a Mobile EHR: A Pilot Study.

Creating notes in the EHR is one of the most problematic aspects for health professionals. The main challenges are the time spent on this task and the quality of the records. Automatic speech recognition technologies aim to facilitate clinical documentation for users, optimizing their workflow. In our hospital, we internally developed an automatic speech recognition system (ASR) to record progress notes in a mobile EHR. The objective of this article is to describe the pilot study carried out to evaluate the implementation of ASR to record progress notes in a mobile EHR application. As a result, the specialty that used ASR the most was Home Medicine. The lack of access to a computer at the time of care and the need to perform short and fast evolutions were the main reasons for users to use the system.

Linguistic disparities in cross-language automatic speech recognition transfer from Arabic to Tashlhiyt

Tashlhiyt is a low-resource language with respect to acoustic databases, language corpora, and speech technology tools, such as Automatic Speech Recognition (ASR) systems. This study investigates whether a method of cross-language re-use of ASR is viable for Tashlhiyt from an existing commercially-available system built for Arabic. The source and target language in this case have similar phonological inventories, but Tashlhiyt permits typologically rare phonological patterns, including vowelless words, while Arabic does not. We find systematic disparities in ASR transfer performance (measured as word error rate (WER) and Levenshtein distance) for Tashlhiyt across word forms and speaking style variation. Overall, performance was worse for casual speaking modes across the board. In clear speech, performance was lower for vowelless than for voweled words. These results highlight systematic speaking mode- and phonotactic-disparities in cross-language ASR transfer. They also indicate that linguistically-informed approaches to ASR re-use can provide more effective ways to adapt existing speech technology tools for low resource languages, especially when they contain typologically rare structures. The study also speaks to issues of linguistic disparities in ASR and speech technology more broadly. It can also contribute to understanding the extent to which machines are similar to, or different from, humans in mapping the acoustic signal to discrete linguistic representations.

Flask-based ASR for Automated Disorder Speech Recognition

Speech disorders encompass a diverse range of conditions hindering effective communication, stemming from developmental, neurological, or physical factors. These challenges impact daily life and opportunities. Automatic Speech Recognition (ASR) technology emerges as a pivotal solution. ASR, powered by sophisticated algorithms, transcribes spoken language into written text, transcending traditional communication barriers. Tailorable to accommodate unique speech disorders, ASR offers the promise of empowering individuals to articulate their thoughts. Our Python Flask-based ASR(FBASR) applications meticulously crafted to cater to individuals, including children, grappling with speech disorders. It provides a user-friendly platform. Users simply submit a speech audio file in WAV format, and the application transcribes it. Alongside recognized text, it furnishes vital audio metrics. Additionally, it generates visual audio signal and spectrogram representations, all stored efficiently for future reference. The application's profound advantage lies in granting a voice to individuals with speech disorders, facilitating education, employment, and healthcare interactions. Furthermore, it serves as a resource for research and therapy development, promising inclusivity and enriching lives. The proposed work concludes by discussing the future outlook of cloud-based ASR and identifying some of the key areas of research that need to be addressed to make cloud-based DSR systems more accurate, reliable, and accessible.

Toward Effective Aircraft Call Sign Detection Using Fuzzy String-Matching between ASR and ADS-B Data

Recently, artificial intelligence and data science have witnessed dramatic progress and rapid growth, especially Automatic Speech Recognition (ASR) technology based on Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs). Consequently, new end-to-end Recurrent Neural Network (RNN) toolkits were developed with higher speed and accuracy that can often achieve a Word Error Rate (WER) below 10%. These toolkits can nowadays be deployed, for instance, within aircraft cockpits and Air Traffic Control (ATC) systems in order to identify aircraft and display recognized voice messages related to flight data, especially for airports not equipped with radar. Hence, the performance of air traffic controllers and pilots can ultimately be improved by reducing workload and stress and enforcing safety standards. Our experiment conducted at Tangier’s International Airport ATC aimed to build an ASR model that is able to recognize aircraft call signs in a fast and accurate way. The acoustic and linguistic models were trained on the Ibn Battouta Speech Corpus (IBSC), resulting in an unprecedented speech dataset with approved transcription that includes real weather aerodrome observation data and flight information with a call sign captured by an ADS-B receiver. All of these data were synchronized with voice recordings in a structured format. We calculated the WER to evaluate the model’s accuracy and compared different methods of dataset training for model building and adaptation. Despite the high interference in the VHF radio communication channel and fast-speaking conditions that increased the WER level to 20%, our standalone and low-cost ASR system with a trained RNN model, supported by the Deep Speech toolkit, was able to achieve call sign detection rate scores up to 96% in air traffic controller messages and 90% in pilot messages while displaying related flight information from ADS-B data using the Fuzzy string-matching algorithm.

Distributed Communicative Language Training Platform Using Automatic Speech Recognition Technology for Smart University

The purpose of this research is to achieve the following objectives: 1) Synthesize documents and international research on the characteristics of a smart university. 2) Synthesize the processes of distributed communicative language training (DCLT). 3) Design the system architecture of a DCLT platform that utilizes automatic speech recognition (ASR) technology for a smart university. 4) Evaluate the appropriateness of a DCLT platform that utilizes ASR technology for a smart university. Nine experts were selected for this research. They were required to have more than five years of relevant experience in the field, including expertise in system architecture, distributed enterprise, language teaching, and ASR. The research instruments included a suitable assessment form for evaluating the system architecture of a DCLT platform that utilizes ASR technology for a smart university. The results of this research indicate that the DCLT platform, which utilizes ASR technology, was considered suitable for a smart university.

Ensuring Safety for Artificial-Intelligence-Based Automatic Speech Recognition in Air Traffic Control Environment

This paper describes the safety assessment conducted in SESAR2020 project PJ.10-W2-96 ASR on automatic speech recognition (ASR) technology implemented for air traffic control (ATC) centers. ASR already now enables the automatic recognition of aircraft callsigns and various ATC commands including command types based on controller–pilot voice communications for presentation at the controller working position. The presented safety assessment process consists of defining design requirements for ASR technology application in normal, abnormal, and degraded modes of ATC operations. A total of eight functional hazards were identified based on the analysis of four use cases. The safety assessment was supported by top-down and bottom-up modelling and analysis of the causes of hazards to derive system design requirements for the purposes of mitigating the hazards. Assessment of achieving the specified design requirements was supported by evidence generated from two real-time simulations with pre-industrial ASR prototypes in approach and en-route operational environments. The simulations, focusing especially on the safety aspects of ASR application, also validated the hypotheses that ASR reduces controllers’ workload and increases situational awareness. The missing validation element, i.e., an analysis of the safety effects of ASR in ATC, is the focus of this paper. As a result of the safety assessment activities, mitigations were derived for each hazard, demonstrating that the use of ASR does not increase safety risks and is, therefore, ready for industrialization.

Speaking to remember: Model-based adaptive vocabulary learning using automatic speech recognition

Memorizing vocabulary is a crucial aspect of learning a new language. While personalized learning- or intelligent tutoring systems can assist learners in memorizing vocabulary, the majority of such systems are limited to typing-based learning and do not allow for speech practice. Here, we aim to compare the efficiency of typing- and speech based vocabulary learning. Furthermore, we explore the possibilities of improving such speech-based learning using an adaptive algorithm based on a cognitive model of memory retrieval. We combined a response time-based algorithm for adaptive item scheduling that was originally developed for typing-based learning with automatic speech recognition technology and tested the system with 50 participants. We show that typing- and speech-based learning result in similar learning outcomes and that using a model-based, adaptive scheduling algorithm improves recall performance relative to traditional learning in both modalities, both immediately after learning and on follow-up tests. These results can inform the development of vocabulary learning applications that–unlike traditional systems–allow for speech-based input.

Using HIPAA (Health Insurance Portability and Accountability Act)-Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study.

Automatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription services among different diagnostic groups in a mental health setting. There has also been little research into the types of words ASR transcriptions mistakenly generate or omit. This study compared the WER of 3 ASR transcription services (Amazon Transcribe [Amazon.com, Inc], Zoom-Otter AI [Zoom Video Communications, Inc], and Whisper [OpenAI Inc]) in interviews across 2 different clinical categories (controls and participants experiencing a variety of mental health conditions). These ASR transcription services were also compared with a commercial human transcription service, Rev (Rev.Com, Inc). Words that were either included or excluded by the error in the transcripts were systematically analyzed by their Linguistic Inquiry and Word Count categories. Participants completed a 1-time research psychiatric interview, which was recorded on a secure server. Transcriptions created by the research team were used as the gold standard from which WER was calculated. The interviewees were categorized into either the control group (n=18) or the mental health condition group (n=47) using the Mini-International Neuropsychiatric Interview. The total sample included 65 participants. Brunner-Munzel tests were used for comparing independent sets, such as the diagnostic groupings, and Wilcoxon signed rank tests were used for correlated samples when comparing the total sample between different transcription services. There were significant differences between each ASR transcription service's WER (P<.001). Amazon Transcribe's output exhibited significantly lower WERs compared with the Zoom-Otter AI's and Whisper's ASR. ASR performances did not significantly differ across the 2 different clinical categories within each service (P>.05). A comparison between the human transcription service output from Rev and the best-performing ASR (Amazon Transcribe) demonstrated a significant difference (P<.001), with Rev having a slightly lower median WER (7.6%, IQR 5.4%-11.35 vs 8.9%, IQR 6.9%-11.6%). Heat maps and spider plots were used to visualize the most common errors in Linguistic Inquiry and Word Count categories, which were found to be within 3 overarching categories: Conversation, Cognition, and Function. Overall, consistent with previous literature, our results suggest that the WER between manual and automated transcription services may be narrowing as ASR services advance. These advances, coupled with decreased cost and time in receiving transcriptions, may make ASR transcriptions a more viable option within health care settings. However, more research is required to determine if errors in specific types of words impact the analysis and usability of these transcriptions, particularly for specific applications and in a variety of populations in terms of clinical diagnosis, literacy level, accent, and cultural origin.

Development of DelTalk (an English learning application) using Agile Method

English has become a world language that must be mastered by many people because English has dominated the era of communication to connect and transfer knowledge to the entire world. When it comes to the development of mobile applications for language learning, previous research has tended to emphasize testing on various technologies such as multimedia, virtual and augmented reality, conversational agents and artificial intelligence-based systems. This research, therefore, aimed to develop a mobile-based English learning application that provided English learning in the form of stories called the Deltalk. This mobile-based English learning application provides features in which the users can practice their speaking in English. This app was developed by integrating ASR (Automatic Speech Recognition) technology provided by NOVO Learning. The ASR was integrated with Deltalk through the API (Application Programming Interface) and Websocket. Deltalk development adapted two agile frameworks, including Scrum and Lean Software Development by performing MVP process one time.