Automatic Speech Recognition Applications Research Articles

Intelligent Clinical Documentation: Harnessing Generative AI for Patient-Centric Clinical Note Generation

Comprehensive clinical documentation is crucial for effective healthcare delivery, yet it poses a significant burden on healthcare professionals, leading to burnout, increased medical errors, and compromised patient safety. This paper explores the potential of generative AI (Artificial Intelligence) to streamline the clinical documentation process, specifically focusing on generating SOAP (Subjective, Objective, Assessment, Plan) and BIRP (Behavior, Intervention, Response, Plan) notes. We present a case study demonstrating the application of natural language processing (NLP) and automatic speech recognition (ASR) technologies to transcribe patient-clinician interactions, coupled with advanced prompting techniques to generate draft clinical notes using large language models (LLMs). The study highlights the benefits of this approach, including time savings, improved documentation quality, and enhanced patient-centered care. Additionally, we discuss ethical considerations, such as maintaining patient confidentiality and addressing model biases, underscoring the need for responsible deployment of generative AI in healthcare settings. The findings suggest that generative AI has the potential to revolutionize clinical documentation practices, alleviating administrative burdens and enabling healthcare professionals to focus more on direct patient care.

Improving Medical Speech-to-Text Accuracy using Vision-Language Pre-training Models.

Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.

A Spiking LSTM Accelerator for Automatic Speech Recognition Application Based on FPGA

Long Short-Term Memory (LSTM) finds extensive application in sequential learning tasks, notably in speech recognition. However, existing accelerators tailored for traditional LSTM networks grapple with high power consumption, primarily due to the intensive matrix–vector multiplication operations inherent to LSTM networks. In contrast, the spiking LSTM network has been designed to avoid these multiplication operations by replacing multiplication and nonlinear functions with addition and comparison. In this paper, we present an FPGA-based accelerator specifically designed for spiking LSTM networks. Firstly, we employ a low-cost circuit in the LSTM gate to significantly reduce power consumption and hardware cost. Secondly, we propose a serial–parallel processing architecture along with hardware implementation to reduce inference latency. Thirdly, we quantize and efficiently deploy the synapses of the spiking LSTM network. The power consumption of the accelerator implemented on Artix-7 and Zynq-7000 is only about 1.1 W and 0.84 W, respectively, when performing the inference for speech recognition with the Free Spoken Digit Dataset (FSDD). Additionally, the energy consumed per inference is remarkably efficient, with values of 87 µJ and 66 µJ, respectively. In comparison with dedicated accelerators designed for traditional LSTM networks, our spiking LSTM accelerator achieves a remarkable reduction in power consumption, amounting to orders of magnitude.

Flask-based ASR for Automated Disorder Speech Recognition

Speech disorders encompass a diverse range of conditions hindering effective communication, stemming from developmental, neurological, or physical factors. These challenges impact daily life and opportunities. Automatic Speech Recognition (ASR) technology emerges as a pivotal solution. ASR, powered by sophisticated algorithms, transcribes spoken language into written text, transcending traditional communication barriers. Tailorable to accommodate unique speech disorders, ASR offers the promise of empowering individuals to articulate their thoughts. Our Python Flask-based ASR(FBASR) applications meticulously crafted to cater to individuals, including children, grappling with speech disorders. It provides a user-friendly platform. Users simply submit a speech audio file in WAV format, and the application transcribes it. Alongside recognized text, it furnishes vital audio metrics. Additionally, it generates visual audio signal and spectrogram representations, all stored efficiently for future reference. The application's profound advantage lies in granting a voice to individuals with speech disorders, facilitating education, employment, and healthcare interactions. Furthermore, it serves as a resource for research and therapy development, promising inclusivity and enriching lives. The proposed work concludes by discussing the future outlook of cloud-based ASR and identifying some of the key areas of research that need to be addressed to make cloud-based DSR systems more accurate, reliable, and accessible.

Ensuring Safety for Artificial-Intelligence-Based Automatic Speech Recognition in Air Traffic Control Environment

This paper describes the safety assessment conducted in SESAR2020 project PJ.10-W2-96 ASR on automatic speech recognition (ASR) technology implemented for air traffic control (ATC) centers. ASR already now enables the automatic recognition of aircraft callsigns and various ATC commands including command types based on controller–pilot voice communications for presentation at the controller working position. The presented safety assessment process consists of defining design requirements for ASR technology application in normal, abnormal, and degraded modes of ATC operations. A total of eight functional hazards were identified based on the analysis of four use cases. The safety assessment was supported by top-down and bottom-up modelling and analysis of the causes of hazards to derive system design requirements for the purposes of mitigating the hazards. Assessment of achieving the specified design requirements was supported by evidence generated from two real-time simulations with pre-industrial ASR prototypes in approach and en-route operational environments. The simulations, focusing especially on the safety aspects of ASR application, also validated the hypotheses that ASR reduces controllers’ workload and increases situational awareness. The missing validation element, i.e., an analysis of the safety effects of ASR in ATC, is the focus of this paper. As a result of the safety assessment activities, mitigations were derived for each hazard, demonstrating that the use of ASR does not increase safety risks and is, therefore, ready for industrialization.

Vowel classification with combining pitch detection and one‐dimensional convolutional neural network based classifier for gender identification

AbstractHuman speech signals may contain specific information regarding a speaker's characteristics, and these signals can be very useful in applications involving interactive voice response (IVR) and automatic speech recognition (ASR). For IVR and ASR applications, speaker classification into different ages and gender groups can be applied in human–machine interaction or computer‐based interaction systems for customised advertisement, translation (text generation), machine dialog systems, or self‐service applications. Hence, an IVR‐based system dictates that ASR should function through users' voices (specific voice‐frequency bands) to identify customers' age and gender and interact with a host system. In the present study, we intended to combine a pitch detection (PD)‐based extractor and a voice classifier for gender identification. The Yet Another Algorithm for Pitch Tracking (YAAPT)‐based PD method was designed to extract the voice fundamental frequency (F0) from non‐stationary speaker's voice signals, allowing us to achieve gender identification, by distinguishing differences in F0 between adult females and males, and classify voices into adult and children groups. Then, in vowel voice signal classification, a one‐dimensional (1D) convolutional neural network (CNN), consisted of a multi‐round 1D kernel convolutional layer, a 1D pooling process, and a vowel classifier that could preliminary divide feature patterns into three level ranges of F0, including adult and children groups. Consequently, a classifier was used in the classification layer to identify the speakers' gender. The proposed PD‐based extractor and voice classifier could reduce complexity and improve classification efficiency. Acoustic datasets were selected from the Hillenbrand database for experimental tests on 12 vowels classifications, and K‐fold cross‐validations were performed. The experimental results demonstrated that our approach is a very promising method to quantify the proposed classifier's performance in terms of recall (%), precision (%), accuracy (%), and F1 score.

Mobile robot: automatic speech recognition application for automation and STEM education

Nowadays, robots are widely applied in life as well as industrial production, medicine, rescue, learning, and entertainment. There are many kinds of robots using different modern technologies such as manipulators, movement, and biomimetic robots; each type is applied according to its own role. Robots are also applied in training, especially in STEM training implemented with mobile robots. In this paper, we proposed models to design mobile robot combined with Android OS Tablet with a user interaction screen, voice control model, and AIDL IPC interactive model for remote control. We research the mobile robot using RockChip AI Processor RK3399Pro, which focuses on understanding the structure and operating mechanism of the AI Processor that RockChip robot uses, along with researching Android with IPC AIDL architecture and Automatic Voice Recognition technology to control robot to turn left, turn right, move forward, backward, and stop the movement by voice with three languages: Korean, English, and Vietnamese. We experimented on 1850 recording audio files including 1013 female voices and 837 male voices with ‘latest_short,’ ‘latest_long,’ and ‘command_and_search’ models. The results obtained a fairly high average confidence of 95% for English and Vietnamese languages and 87% for Korean. From results of this study, it will serve as a foundation for the research and development of a system to support students in learning by interacting with mobile robots through voice with multi-language. STEM education in the form of gamification of the curriculum, especially for extensive research on training programs can be applied in mobile robots with more flexibility.

CORAA ASR: a large corpus of spontaneous and prepared speech manually validated for speech recognition in Brazilian Portuguese

Automatic Speech recognition (ASR) is a complex and challenging task. In recent years, there have been significant advances in the area. In particular, for the Brazilian Portuguese (BP) language, there were around 376 h publicly available for the ASR task until the second half of 2020. With the release of new datasets in early 2021, this number increased to 574 h. The existing resources, however, are composed of audios containing only read and prepared speech. There is a lack of datasets including spontaneous speech, which are essential in several ASR applications. This paper presents CORAA (Corpus of Annotated Audios) ASR with 290 h, a publicly available dataset for ASR in BP containing validated pairs of audio-transcription. CORAA ASR also contains European Portuguese audios (4.6 h). We also present a public ASR model based on Wav2Vec 2.0 XLSR-53, fine-tuned over CORAA ASR. Our model achieved a Word Error Rate (WER) of 24.18% on CORAA ASR test set and 20.08% on Common Voice test set. When measuring the Character Error Rate (CER), we obtained 11.02% and 6.34% for CORAA ASR and Common Voice, respectively. CORAA ASR corpora were assembled to both improve ASR models in BP with phenomena from spontaneous speech and motivate young researchers to start their studies on ASR for Portuguese. All the corpora are publicly available at https://github.com/nilc-nlp/CORAA under the CC BY-NC-ND 4.0 license.

An exploratory study on the accuracy of three speech recognition software programs for young Taiwanese EFL learners

ABSTRACT Automatic speech recognition (ASR) technology affords language learners the ability to evaluate their pronunciation accuracy by comparing their intended spoken output with the transcribed text produced by ASR-based dictation applications. However, earlier dictation tools were criticized for producing low level recognition rates for non-native speech. Further investigation on the transcription accuracy of new ASR dictation tools for non-native speech is therefore necessary. This study aimed to evaluate the accuracy of three modern ASR dictation software programs with young Taiwanese learners of English. Capabilities of these ASR applications were also explored by assessing whether the transcription accuracy of high and low proficiency learners’ can be differentiated. Thirty junior high school students dictated the same 60 sentences used in Derwing et al.’s (2000) original dictation study, and their outputs were further analyzed. The results indicated that the transcription accuracy rates of high proficiency learners were comparable to the accuracy rates of native English speakers in previous studies, suggesting that current dictation programs have become more accurate in transcribing non-native speech. Results also show that low proficiency learners’ accuracy rates were significantly lower compared to high proficiency learners, indicating that the programs were able to differentiate proficient and less proficient learners.

Identifying Language Disorder in Bilingual Children Using Automatic Speech Recognition.

The differential diagnosis of developmental language disorder (DLD) in bilingual children represents a unique challenge due to their distributed language exposure and knowledge. The current evidence indicates that dual-language testing yields the most accurate classification of DLD among bilinguals, but there are limited personnel and resources to support this practice. The purpose of this study was therefore to determine the feasibility of dual-language automatic speech recognition (ASR) for identifying DLD in bilingual children. Eighty-four Spanish-English bilingual second graders with (n = 25) and without (n = 59) confirmed diagnoses of DLD completed the Bilingual English-Spanish Assessment-Middle Extension Morphosyntax in both languages. Their responses on a subset of items were scored manually by human examiners and programmatically by a researcher-developed ASR application employing a commercial speech-to-text algorithm. Results demonstrated moderate overall item-by-item scoring agreement (k = .54) and similar classification accuracy values (human = 92%, ASR = 88%) between the two methods using the best-language score. Classification accuracy of the ASR method increased to 94% of cases correctly classified when test items with poorer discrimination in the ASR condition were eliminated. This study provides preliminary support for the technical feasibility of ASR as a bilingual expressive language assessment tool. https://doi.org/10.23641/asha.20249994.

Combining Frame-Synchronous and Label-Synchronous Systems for Speech Recognition

Commonly used automatic speech recognition (ASR) systems can be classified into frame-synchronous and label-synchronous categories, based on whether the speech is decoded on a per-frame or per-label basis. Frame-synchronous systems, such as traditional hidden Markov model systems, can easily incorporate existing knowledge and can support streaming ASR applications. Label-synchronous systems, based on attention-based encoder-decoder models, can jointly learn the acoustic and language information with a single model, which can be regarded as audio-grounded language models. In this paper, we propose rescoring the N-best hypotheses or lattices produced by a first-pass frame-synchronous system with a label-synchronous system in a second-pass. By exploiting the complementary modelling of the different approaches, the combined two-pass systems achieve competitive performance without using any extra speech or text data on two standard ASR tasks. For the 80-hour AMI IHM dataset, the combined system has a 13.7% word error rate (WER) on the evaluation set, which is up to a 29% relative WER reduction over the individual systems. For the 300-hour Switchboard dataset, the WERs of the combined system are 5.7% and 12.1% on Switchboard and CallHome subsets of Hub5'00, and 13.2% and 7.6% on Switchboard Cellular and Fisher subsets of RT03, up to a 33% relative reduction in WER over the individual systems.

Turkish Speech Recognition Techniques and Applications of Recurrent Units (LSTM and GRU)

A typical solution of Automatic Speech Recognition (ASR) problems is realized by feature extraction, feature classification, acoustic modeling and language modeling steps. In classification and modeling steps, Deep Learning Methods have become popular and give more successful recognition results than conventional methods. In this study, an application for solving ASR problem in Turkish Language has been developed. The data sets and studies related to Turkish Language ASR problem are examined. Language models in the ASR problems of agglutative language groups such as Turkish, Finnish and Hungarian are examined. Subword based model is chosen in order not to decrease recognition performance and prevent large vocabulary. The recogniton performance is increased by Deep Learning Methods called Long-Short Term Memory (LSTM) Neural Networks and Gated Recurrent Unit (GRU) in the classification and acoustic modeling steps. The recognition performances of systems including LSTM and GRU are compared with the the previous studies using traditional methods and Deep Neural Networks. When the results were evaluated, it is seen that LSTM and GRU based Speech Recognizers performs better than the recognizers with previous methods. Final Word Error Rate (WER) values were obtained for LSTM and GRU as 10,65% and 11,25%, respectively. GRU based systems have similar performance when compared to LSTM based systems. However, it has been observed that the training periods are short. Computation times are 73.518 and 61.020 seconds respectively. The study gave detailed information about the applicability of the latest methods to Turkish ASR research and applications.

Effectiveness of Mobile Assisted Pronunciation Training in the Acquisition of English Vowels, /o/ and /ɔ/

With the wide availability of mobile devices, efforts to accommodate these technological devices have been made in pronunciation training. This study aimed to investigate the effectiveness of mobile apps in improving the perception and production of the English vowels, /o/ and /ɔ/, which are difficult to discriminate and produce for Korean learners of English. The participants were 24 college freshmen enrolled in a general English course. They shadowed the model pronunciation from the application and corrected their production after instant feedback from the automatic speech recognition application. Their perception was evaluated in the preand post-tests, and the production was visually examined on the formant chart. The training was held for 5 minutes each day, for a month. The results showed that the training made positive effects on improving students’ perception of the target vowels. There was also some progress in their production although the improvement was not as noticeable as in perception. A discrepancy in the production was discovered between male and female participants with the latter performing better.

Detecting and correcting automatic speech recognition errors with a new model

The purpose of automatic speech recognition (ASR) systems is to recognize speech signals obtained from people and convert them into text so that they can be processed by a computer. Although many ASR applications are versatile and widely used in the real world, they still generate relatively inaccurate results. They tend to generate spelling errors in recognized words, especially in noisy environments, in situations where the vocabulary size is increased, and at times when the input speech is of poor quality. The permanent presence of errors in ASR systems has led to the need to find alternative methods for automatic detection and correction of such errors. In this study, the basic principles of ASR evaluation are first summarized, and then a new approach based on the suggestion of an alternative hypothesis is proposed for the detection and correction of these errors generated by ASR systems. The proposed method involves a series of processes such as identifying incorrect words, selecting the ones that can be corrected, and identifying candidate words to replace these words. As a result of the tests carried out by creating different test environments, significant performance improvements for Turkish were achieved and an average of 4.60 % performance improvement was provided.

Design of ASR Software for Recognition of the Russian Language Variants

The article first touches on the problem of speech recognition of Russian language variants. With the development and growing (ASR) popularity of the automatic speech recognition (ASR) technology, more and more attention is now being paid to the problems related to the incompatibility of modern applications to work with non-standard language varieties. This question is especially relevant for Russian, as it is, contrary to the conservative statement about its homogeneity, represented by many forms that differ from the standard one, which generally have a wide distribution in various regions of Russia and throughout the world. The study of various aspects of the interaction of ACER algorithms with non-standard varieties of the Russian language, as well as existing approaches to creating an ASR product that can process such idioms, today seems to be an urgent direction. The aim of the work is to analyze in detail the methods for developing ASR systems capable of performing the task of recognizing and processing speech samples of speakers of Russian language forms different from the standard, which may contribute to further research on this topic. The research material is based on the software interface of the SOVA ASR application for automatic speech recognition, as well as a selection of audio recordings of speech of native speakers of the Central Asian and Ukrainian versions of Russian, and the corresponding transcription texts. The research methods such as the study and analysis of specialized literature, data collection for subsequent software processing, qualitative and quantitative analysis, and experimental data are used.

A Model for the Application of Automatic Speech Recognition for Generating Lesson Summaries

A Model for the Application of Automatic Speech Recognition for Generating Lesson Summaries

SPEESH: speech-based mobile application for dysarthric speech recognition

There are numerous automatic speech recognition applications available in market, but these applications are quite challenging to be used for individual with speech disability. Due to this demand, SPEESH, a speech recognition system is developed for individuals with dysarthria disorders. The main objective of this project is to study the requirement of proposed system and characteristics of individuals with dysarthria. Interview with speech therapists in Malaysia has been conducted on the phone to elicit the requirements for the system. Based on the interview, we acquired numerous common words that are used by dysarthric patients. These findings are crucial in designing the application to ease the user’s experience. The system consists of two phases on how it works; text dependent and text independent. First, in a text dependent phase, the system will learn to recognize specific words from the speaker and register them into database. In the next phase, speaker model will now consist characteristics of voice of the person and it can identify what one is saying. SPEESH application have been designed to be able to translate words from individuals with dysarthria or any speech disorders. All the functionalities and features are all developed in a way such that SPEESH will work as a communication platform between dysarthria individuals and their surroundings.

La tecnología habla-texto como herramienta de documentación para intérpretes: Nuevo método para compilar un corpus ad hoc y extraer terminología a partir de discursos orales en vídeo

Although interpreting has not yet benefited from technology as much as its sister field, translation, interest in developing tailor-made solutions for interpreters has risen sharply in recent years. In particular, Automatic Speech Recognition (ASR) is being used as a central component of Computer-Assisted Interpreting (CAI) tools, either bundled or standalone. This study pursues three main aims: (i) to establish the most suitable ASR application for building ad hoc corpora by comparing several ASR tools and assessing their performance; (ii) to use ASR in order to extract terminology from the transcriptions obtained from video-recorded speeches, in this case talks on climate change and adaptation; and (iii) to promote the adoption of ASR as a new documentation tool among interpreters. To the best of our knowledge, this is one of the first studies to explore the possibility of Speech-to-Text (S2T) technology for meeting the preparatory needs of interpreters as regards terminology and background/domain knowledge.

Mutual-learning sequence-level knowledge distillation for automatic speech recognition

Automatic speech recognition (ASR) is a crucial technology for man-machine interaction. End-to-end models have been studied recently in deep learning for ASR. However, these models are not suitable for the practical application of ASR due to their large model sizes and computation costs. To address this issue, we propose a novel mutual-learning sequence-level knowledge distillation framework enjoying distinct student structures for ASR. Trained mutually and simultaneously, each student learns not only from the pre-trained teacher but also from its distinct peers, which can improve the generalization capability of the whole network, through making up for the insufficiency of each student and bridging the gap between each student and the teacher. Extensive experiments on the TIMIT and large LibriSpeech corpuses show that, compared with the state-of-the-art methods, the proposed method achieves an excellent balance between recognition accuracy and model compression.

Automatic Re-Formulation of user’s Irrational Behavior in Speech Recognition using Acoustic Nudging Model

In automatic speech recognition for development of automatic speech recognition applications, there has been numerous claims on the presence of speech recognition errors known as classified into lexical and acoustic errors. These errors distort speech signals thereby depreciating the accuracy and performance rate of speech recognition applications. Even though lexical speech recognition error problem has been partially combated, acoustic speech recognition error referred to as user’s acoustic irrational behavior is being ignored causing high error rate with low accuracy which is the bone of contention and an impediment factor in the wide adoption of speech recognition technology. Users do not always behave in a rational manner especially when dealing with a particular speech recognition application. The persistent presence of these user’s acoustic irrational behavior in speech have intensified the essential need to automatically detect and correct such errors, as current researches only focus on detecting user’s acoustic irrational behavior but not correcting/reformulating/re-sizing this error. Hence, this paper provides an acoustic nudging model that will perform automatic correction/reformulation of user’s acoustic irrational behavior in speech to achieve higher performance and accuracy using different acoustic parameters which are based in Pitch, Time gaps between words, Timbre descend and ascend time and Loudness. This study was able to discover a foundation for reducing error rate and achieve higher performance, as well as improve accuracy in speech recognition applications through detection and re-formulation of user’s acoustic irrational behavior in speech signal automatically, thereby making the model applicable to any speech recognition applications. The outcome of this study would be useful in enhancing accuracy and performance in the context of automatic speech recognition.