Average Word Error Rate Research Articles

CAM: A cross-lingual adaptation framework for low-resource language speech recognition

In this paper, a novel cross-lingual adaptation framework called CAM is presented for low-resource language speech recognition (LLSR). It is based on the recent popular adapter method. CAM is achieved by adapting self-supervised speech models (SSMs) from source languages to target low-resource languages in a two-stage process. CAM fills two research gaps existing in current methods: (i) language similarity is not effectively considered; (ii) the performance-efficiency trade-off is not well balanced. Specifically, two key components, a similarity-aware fusion module (SAFM) and an adapter weight-sharing strategy (AWSS), are designed. A well-trained adapter is introduced by SAFM to compute precise language similarities via dot product. AWSS takes both performance and efficiency into consideration by sharing adapter weights. Experimental results on two corpora, FLEURS and Common Voice, demonstrate that CAM equipped with these two designs, denoted as performance-oriented CAM (P-CAM), obtains state-of-the-art (SOTA) performance with satisfactory efficiency compared to current leading methods. Besides, an efficiency-oriented CAM (E-CAM) by introducing a weight-space fusion module (WSFM) is presented. The core of WSFM is to average the weights of multiple adapters into a new adapter. Compared to full fine-tuning, only 5.0% of trainable parameters are required for E-CAM while a 2.3% relative average word error rate (WER) reduction is obtained. CAM brings a performance- or efficiency-oriented trade-off to meet the different needs of speech recognition systems.

Language fusion via adapters for low-resource speech recognition

Data scarcity makes low-resource speech recognition systems suffer from severe overfitting. Although fine-tuning addresses this issue to some extent, it leads to parameter-inefficient training. In this paper, a novel language knowledge fusion method, named LanFusion, is proposed. It is built on the recent popular adapter-tuning technique, thus maintaining better parameter efficiency compared with conventional fine-tuning methods. LanFusion is a two-stage method. Specifically, multiple adapters are first trained on several source languages to extract language-specific and language-invariant knowledge. Then, the trained adapters are re-trained on the target low-resource language to fuse the learned knowledge. Compared with Vanilla-adapter, LanFusion obtains a relative average word error rate (WER) reduction of 9.8% and 8.6% on the Common Voice and FLEURS corpora, respectively. Extensive experiments demonstrate the proposed method is not only simple and effective but also parameter-efficient. Besides, using source languages that are geographically similar to the target language yields better results on both datasets.

Deep causal speech enhancement and recognition using efficient long-short term memory Recurrent Neural Network.

Long short-term memory (LSTM) has been effectively used to represent sequential data in recent years. However, LSTM still struggles with capturing the long-term temporal dependencies. In this paper, we propose an hourglass-shaped LSTM that is able to capture long-term temporal correlations by reducing the feature resolutions without data loss. We have used skip connections in non-adjacent layers to avoid gradient decay. In addition, an attention process is incorporated into skip connections to emphasize the essential spectral features and spectral regions. The proposed LSTM model is applied to speech enhancement and recognition applications. The proposed LSTM model uses no future information, resulting in a causal system suitable for real-time processing. The combined spectral feature sets are used to train the LSTM model for improved performance. Using the proposed model, the ideal ratio mask (IRM) is estimated as a training objective. The experimental evaluations using short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) have demonstrated that the proposed model with robust feature representation obtained higher speech intelligibility and perceptual quality. With the TIMIT, LibriSpeech, and VoiceBank datasets, the proposed model improved STOI by 16.21%, 16.41%, and 18.33% over noisy speech, whereas PESQ is improved by 31.1%, 32.9%, and 32%. In seen and unseen noisy situations, the proposed model outperformed existing deep neural networks (DNNs), including baseline LSTM, feedforward neural network (FDNN), convolutional neural network (CNN), and generative adversarial network (GAN). With the Kaldi toolkit for automated speech recognition (ASR), the proposed model significantly reduced the word error rates (WERs) and reached an average WER of 15.13% in noisy backgrounds.

Automated Speech Audiometry: Can It Work Using Open-Source Pre-Trained Kaldi-NL Automatic Speech Recognition?

A practical speech audiometry tool is the digits-in-noise (DIN) test for hearing screening of populations of varying ages and hearing status. The test is usually conducted by a human supervisor (e.g., clinician), who scores the responses spoken by the listener, or online, where software scores the responses entered by the listener. The test has 24-digit triplets presented in an adaptive staircase procedure, resulting in a speech reception threshold (SRT). We propose an alternative automated DIN test setup that can evaluate spoken responses whilst conducted without a human supervisor, using the open-source automatic speech recognition toolkit, Kaldi-NL. Thirty self-reported normal-hearing Dutch adults (19-64 years) completed one DIN + Kaldi-NL test. Their spoken responses were recorded and used for evaluating the transcript of decoded responses by Kaldi-NL. Study 1 evaluated the Kaldi-NL performance through its word error rate (WER), percentage of summed decoding errors regarding only digits found in the transcript compared to the total number of digits present in the spoken responses. Average WER across participants was 5.0% (range 0-48%, SD = 8.8%), with average decoding errors in three triplets per participant. Study 2 analyzed the effect that triplets with decoding errors from Kaldi-NL had on the DIN test output (SRT), using bootstrapping simulations. Previous research indicated 0.70 dB as the typical within-subject SRT variability for normal-hearing adults. Study 2 showed that up to four triplets with decoding errors produce SRT variations within this range, suggesting that our proposed setup could be feasible for clinical applications.

Improving Dysarthric Speech Segmentation With Emulated and Synthetic Augmentation.

Acoustic features extracted from speech can help with the diagnosis of neurological diseases and monitoring of symptoms over time. Temporal segmentation of audio signals into individual words is an important pre-processing step needed prior to extracting acoustic features. Machine learning techniques could be used to automate speech segmentation via automatic speech recognition (ASR) and sequence to sequence alignment. While state-of-the-art ASR models achieve good performance on healthy speech, their performance significantly drops when evaluated on dysarthric speech. Fine-tuning ASR models on impaired speech can improve performance in dysarthric individuals, but it requires representative clinical data, which is difficult to collect and may raise privacy concerns. This study explores the feasibility of using two augmentation methods to increase ASR performance on dysarthric speech: 1) healthy individuals varying their speaking rate and loudness (as is often used in assessments of pathological speech); 2) synthetic speech with variations in speaking rate and accent (to ensure more diverse vocal representations and fairness). Experimental evaluations showed that fine-tuning a pre-trained ASR model with data from these two sources outperformed a model fine-tuned only on real clinical data and matched the performance of a model fine-tuned on the combination of real clinical data and synthetic speech. When evaluated on held-out acoustic data from 24 individuals with various neurological diseases, the best performing model achieved an average word error rate of 5.7% and a mean correct count accuracy of 94.4%. In segmenting the data into individual words, a mean intersection-over-union of 89.2% was obtained against manual parsing (ground truth). It can be concluded that emulated and synthetic augmentations can significantly reduce the need for real clinical data of dysarthric speech when fine-tuning ASR models and, in turn, for speech segmentation.

Can Same-right-and-different-left Gestures Be Recognized with Only Right-hand Signals?

Sign language serves as a bridge between the hearing-impaired and other people. Existing sensor-based approaches tend to only collect data from the dominant hand. Does this signal collection method affect the accuracy of gesture recognition, especially gestures where the dominant hand has the same movement while the non-dominant hand has different movements? The specific gestures are called same-right-and-different-left (SRDL) where the right hand is dominant. This article is the first to propose an SRDL-aware sign language recognition system. First, an SRDL discriminator based on an autoencoder and range classifier is designed to determine whether the gesture is SRDL. Second, an SRDL feature selector based on clustering relationship is presented. Multivariate variational mode decomposition and fast fourier transform are used to obtain the feature expression. Moreover, a clustering relationship algorithm is proposed to dynamically select features for every group of SRDL gestures in the feature expression. Finally, the experimental results show that the average word error rate is 14.3% and decreases by 8.5% and 12.1% compared with Signspeaker and MyoSign, respectively.

Design of Intelligent Reflecting Surface Phase Shifts for Uplink SCMA

In this letter, we consider an intelligent reflecting surface (IRS) assisted uplink sparse code multiple access (SCMA) system. We first derive an upper bound on word error rate (WER), based on which we design IRS phase shifts to minimize WER. Specifically, we propose two IRS design algorithms, semidefinite relaxation (SDR)-based approach and gradient ascent approach. We also provide computational complexity analysis. Then we derive an upper bound on average WER and analyse error performance. Finally, simulations demonstrate that both algorithms have performance similar to the known approach, while the proposed gradient ascent approach has the least computational complexity.

MSilent

Silent speech recognition (SSR) allows users to speak to the device without making a sound, avoiding being overheard or disturbing others. Compared to the video-based approach, wireless signal-based SSR can work when the user is wearing a mask and has fewer privacy concerns. However, previous wireless-based systems are still far from well-studied, e.g. they are only evaluated in corpus with highly limited size, making them only feasible for interaction with dozens of deterministic commands. In this paper, we present mSilent, a millimeter-wave (mmWave) based SSR system that can work in the general corpus containing thousands of daily conversation sentences. With the strong recognition capability, mSilent not only supports the more complex interaction with assistants, but also enables more general applications in daily life such as communication and input. To extract fine-grained articulatory features, we build a signal processing pipeline that uses a clustering-selection algorithm to separate articulatory gestures and generates a multi-scale detrended spectrogram (MSDS). To handle the complexity of the general corpus, we design an end-to-end deep neural network that consists of a multi-branch convolutional front-end and a Transformer-based sequence-to-sequence back-end. We collect a general corpus dataset of 1,000 daily conversation sentences that contains 21K samples of bi-modality data (mmWave and video). Our evaluation shows that mSilent achieves a 9.5% average word error rate (WER) at a distance of 1.5m, which is comparable to the performance of the state-of-the-art video-based approach. We also explore deploying mSilent in two typical scenarios of text entry and in-car assistant, and the less than 6% average WER demonstrates the potential of mSilent in general daily applications.

Coded Asymmetric Modulation Design for Block-Fading Channels With Nonlinear Energy Harvesting

In this letter, the design of coded modulation for simultaneous wireless information and power transfer (SWIPT) block-fading channels is proposed, for both power-splitting (PS) and time-switching (TS) nonlinear energy harvesting (EH). With PS, although the diversity-rate tradeoff is not affected, nonlinear EH requires asymmetric modulations design which leads to an information transfer (IT) performance penalty. We thus design an asymmetric circular quadrature amplitude modulation (ACQAM) scheme that provides large performance gains compared to conventional modulations. On the other hand, we show that TS negatively impacts the diversity-rate tradeoff of the coded modulation, while an optimal QAM can be employed in the IT phase. We next derive an upper bound on the average word error rate (WER) based on a pairwise error probability analysis for both EH methods. Finally, WER simulation results are shown together with the derived bounds.

Multilingual Speech Recognition for Turkic Languages

The primary aim of this study was to contribute to the development of multilingual automatic speech recognition for lower-resourced Turkic languages. Ten languages—Azerbaijani, Bashkir, Chuvash, Kazakh, Kyrgyz, Sakha, Tatar, Turkish, Uyghur, and Uzbek—were considered. A total of 22 models were developed (13 monolingual and 9 multilingual). The multilingual models that were trained using joint speech data performed more robustly than the baseline monolingual models, with the best model achieving an average character and word error rate reduction of 56.7%/54.3%, respectively. The results of the experiment showed that character and word error rate reduction was more likely when multilingual models were trained with data from related Turkic languages than when they were developed using data from unrelated, non-Turkic languages, such as English and Russian. The study also presented an open-source Turkish speech corpus. The corpus contains 218.2 h of transcribed speech with 186,171 utterances and is the largest publicly available Turkish dataset of its kind. The datasets and codes used to train the models are available for download from our GitHub page.

Mandarin Electro-Laryngeal Speech Enhancement Using Cycle-Consistent Generative Adversarial Networks

Electro-laryngeal (EL) speech has poor intelligibility and naturalness, which hampers the popular use of the electro-larynx. Voice conversion (VC) can enhance EL speech. However, if the EL speech to be enhanced is with complicated tone variation rules in Mandarin, the enhancement will be less effective. This is because the source speech (Mandarin EL speech) and the target speech (normal speech) are not strictly parallel. We propose using cycle-consistent generative adversarial networks (CycleGAN, a parallel-free VC framework) to enhance continuous Mandarin EL speech, which can solve the above problem. In the proposed framework, the generator is designed based on the neural networks of a 2D-Conformer-1D-Transformer-2D-Conformer. Then, we used Mel-Spectrogram instead of traditional acoustic features (fundamental frequency, Mel-Cepstrum parameters and aperiodicity parameters). At last, we converted the enhanced Mel-Spectrogram into waveform signals using WaveNet. We undertook both subjective and objective tests to evaluate the proposed approach. Compared with traditional approaches to enhance continuous Mandarin EL speech with variable tone (the average tone accuracy being 71.59% and average word error rate being 10.85%), our framework increases the average tone accuracy by 12.12% and reduces the average errors of word perception by 9.15%. Compared with the approaches towards continuous Mandarin EL speech with fixed tone (the average tone accuracy being 29.89% and the average word error rate being 10.74%), our framework increases the average tone accuracy by 42.38% and reduces the average errors of word perception by 8.59%. Our proposed framework can effectively address the problem that the source and target speech are not strictly parallel. The intelligibility and naturalness of Mandarin EL speech have been further improved.

The development of an automatic speech recognition model using interview data from long-term care for older adults

In long-term care (LTC) for older adults, interviews are used to collect client perspectives that are often recorded and transcribed verbatim, which is a time-consuming, tedious task. Automatic speech recognition (ASR) could provide a solution; however, current ASR systems are not effective for certain demographic groups. This study aims to show how data from specific groups, such as older adults or people with accents, can be used to develop an effective ASR. An initial ASR model was developed using the Mozilla Common Voice dataset. Audio and transcript data (34 h) from interviews with residents, family, and care professionals on quality of care were used. Interview data were continuously processed to reduce the word error rate (WER). Due to background noise and mispronunciations, an initial ASR model had a WER of 48.3% on interview data. After finetuning using interview data, the average WER was reduced to 24.3%. When tested on speech data from the interviews, a median WER of 22.1% was achieved, with residents displaying the highest WER (22.7%). The resulting ASR model was at least 6 times faster than manual transcription. The current method decreased the WER substantially, verifying its efficacy. Moreover, using local transcription of audio can be beneficial to the privacy of participants. The current study shows that interview data from LTC for older adults can be effectively used to improve an ASR model. While the model output does still contain some errors, researchers reported that it saved much time during transcription.

Malaysian English Large Vocabulary Continuous Speech Recognizer: An Improvement Using Acoustic Model Adapation

This research project aims to develop Malaysian English Continuous Speech Recognition system by adapting US English acoustic model with Malaysian English speech corpus using Maximum a posteriori reasoning (MAP) and Maximum Likelihood Linear Regression (MLLR). During feature extraction stage, the Mel-Frequency Cepstral Coefficients (MFCC) technique was used. The Hidden Markov Model was used as the back end pattern comparison technique. For the purpose of implementation, the CMU Sphinx toolkit, which includes Pocketsphinx and Sphinxtrain as well as an acoustic model, was used to develop a speech recognition system for Malaysian English. Malaysian English speech sample will be recorded and transcribed to produce the training database required for acoustic model adaptation. The adaptation speech corpus were collected from a number of speakers. The outcome of this research could increase the application of Malaysian English speech recognition in Malaysia due to accent problem. As a result, speech recognition systems that have gone through the MAP adaptation had the best performance. Its average word error rate achieved was 32.84%. Average word recognition rate was 72.52% and average sentence error rate was 78.89%.

Artificial Intelligence Biosensing System on Hand Gesture Recognition for the Hearing Impaired

AI technologies have the potential to help deaf individuals communicate. Due to the complexity of sign fragmentation and the inadequacy of capturing hand gestures, the authors present a sign language recognition (SLR) system and wearable surface electromyography (sEMG) biosensing device based on a Deep SLR that converts sign language into printed message or speech, allowing people to better understand sign language and hand motions. On the forearms, two armbands containing a biosensor and multi-channel sEMG sensors are mounted to capture quite well arm and finger actions. Deep SLR was tested on an Android and iOS smartphone, and its usefulness was determined by comprehensive testing. Sign Speaker has a considerable limitation in terms of recognising two-handed signs with smartphone and smartwatch. To solve these issues, this research proposes a new real-time end-to-end SLR method. The average word error rate of continuous sentence recognition is 9.6%, and detecting signals and recognising a sentence with six sign words takes less than 0.9 s, demonstrating Deep SLR's recognition.

Two-Step Joint Optimization with Auxiliary Loss Function for Noise-Robust Speech Recognition.

In this paper, a new two-step joint optimization approach based on the asynchronous subregion optimization method is proposed for training a pipeline model composed of two different models. The first-step processing of the proposed joint optimization approach trains the front-end model only, and the second-step processing trains all the parameters of the combined model together. In the asynchronous subregion optimization method, the first-step processing only supports the goal of the front-end model. However, the first-step processing of the proposed approach works with a new loss function to make the front-end model support the goal of the back-end model. The proposed optimization approach was applied, here, to a pipeline composed of a deep complex convolutional recurrent network (DCCRN)-based speech enhancement model and a conformer-transducer-based ASR model as a front-end and a back-end, respectively. Then, the performance of the proposed two-step joint optimization approach was evaluated on the LibriSpeech automatic speech recognition (ASR) corpus in noisy environments by measuring the character error rate (CER) and word error rate (WER). In addition, an ablation study was carried out to examine the effectiveness of the proposed optimization approach on each of the processing blocks in the conformer-transducer ASR model. Consequently, it was shown from the ablation study that the conformer-transducer-based ASR model with the joint network trained only by the proposed optimization approach achieved the lowest average CER and WER. Moreover, the proposed optimization approach reduced the average CER and WER on the Test-Noisy dataset under matched noise conditions by 0.30% and 0.48%, respectively, compared to the approach of separate optimization of speech enhancement and ASR. Compared to the conventional two-step joint optimization approach, the proposed optimization approach provided average CER and WER reductions of 0.22% and 0.31%, respectively. Moreover, it was revealed that the proposed optimization approach achieved a lower average CER and WER, by 0.32% and 0.43%, respectively, than the conventional optimization approach under mismatched noise conditions.

Addressing Clinical Documentation Challenges: The Role of AIPowered Digital Scribes

In the realm of healthcare, the burden of clinical documentation has long been a challenge for healthcare professionals. This paper presents a comprehensive study on the development and evaluation of a unified digital scribe system aimed at automating the process of recording, transcribing, and annotating clinical narratives in Spanish. The proposed system integrates cutting-edge technologies, including speech recognition and Named Entity Recognition (NER), to streamline the documentation workflow. The study evaluates the performance of the system in a simulated environment, encompassing transcription accuracy and entity recognition across medical and dental domains. Notably, the average Word Error Rates (WER) for different domains reveal promising results, showcasing the system's potential to accurately transcribe spoken clinical narratives. Furthermore, the NER model demonstrates its capability to identify key medical entities such as diseases, body parts, and medications, with F1 scores indicating strong performance. The paper also outlines the development of a prototype platform that unifies both the transcription and NER modules. This platform offers healthcare professionals a seamless solution for capturing, transcribing, and annotating clinical information, ultimately aiming to reduce documentation burden and improve clinical workflow efficiency. In conclusion, this research presents a robust foundation for the development of a digital scribe system in the Spanish healthcare system, offering promising solutions to alleviate clinical documentation challenges, reduce professional burnout, and enhance patient care through streamlined documentation practices.

Design of an Intelligent Reflecting Surface Aided mmWave Massive MIMO Using X-Precoding

In this paper, we consider an intelligent reflecting surface (IRS) aided millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) system using hybrid beamforming/combining. To enhance error performance, we adopt X-code (or X-precoder), a low-complexity precoding technique for traditional MIMO channels, to encode information symbols. We first derive an upper bound on word error rate (WER), based on which we design jointly IRS phase shifts and X-code (or X-precoder) to minimize WER. Specifically, we propose two algorithms for IRS design exploiting alternating optimization and gradient ascent optimization methods. Then we devise X-code and X-precoder, respectively, by minimizing average WER over all channel realizations and WER for each channel realization. We also provide their diversity analysis. Further, we present the procedure of decoupling fully digital beamformer/combiner at transceiver into the optimal hybrid one. Finally, simulation results show that both IRS optimization algorithms have similar WERs whereas gradient ascent approach has a lower computational complexity. Simulations demonstrate that the designed X-code (or X-precoder) provides a significant performance gain.

What does parity mean? A detailed comparison of ASR and human transcription errors

Automatic speech recognition (ASR) has seen dramatic improvements as a result of advances in deep learning. This has led several recent studies to conclude that ASR is approaching parity with human performance, at least in certain speech contexts. These studies use average word error rate (WER) as their primary evaluation metric, dividing the total insertions + substitutions + deletions by the reference length (WER = (I + D+S)/Nr), to compare ASR systems to human transcribers. Averaging combined error types obscures important differences between human and ASR error patterns which impact human comprehension of the output. Human transcribers tend to delete pragmatic and discourse markers (such as fillers and backchannels) and disfluencies, whereas ASR tends to make more substitution errors on words. In conversational settings listeners are able to recover from missing discourse markers, function words, or backchannels, but word substitutions are harder to recover from because they interrupt the information flow. WER is a reasonable first pass metric of ASR performance, but when it comes to communicative parity, it averages out important ways in which it differs from human transcription.

BackSwipe: Back-of-device Word-Gesture Interaction on Smartphones.

Back-of-device interaction is a promising approach to interacting on smartphones. In this paper, we create a back-of-device command and text input technique called BackSwipe, which allows a user to hold a smartphone with one hand, and use the index finger of the same hand to draw a word-gesture anywhere at the back of the smartphone to enter commands and text. To support BackSwipe, we propose a back-of-device word-gesture decoding algorithm which infers the keyboard location from back-of-device gestures, and adjusts the keyboard size to suit the gesture scales; the inferred keyboard is then fed back into the system for decoding. Our user study shows BackSwipe is feasible and a promising input method, especially for command input in the one-hand holding posture: users can enter commands at an average accuracy of 92% with a speed of 5.32 seconds/command. The text entry performance varies across users. The average speed is 9.58 WPM with some users at 18.83 WPM; the average word error rate is 11.04% with some users at 2.85%. Overall, BackSwipe complements the extant smartphone interaction by leveraging the back of the device as a gestural input surface.

Automatic Speech Recognition for Indoor HRI Scenarios

This article presents a stand-alone automatic speech recognition system that accounts for listener movement, time-varying reverberation effects, environmental noise, and user position information for beamforming approaches in an HRI setting. We raise the importance of replacing the classical black-box integration of automatic speech recognition technology in HRI applications with the incorporation of the acoustic environment representation and modeling, and of the target source direction. Test data were recorded on a real robot under various moving conditions. For addressing the time-varying acoustic channel problem and incorporating environmental effect during training, clean speech samples were passed through estimated static channel responses and noise was added. Beamforming is investigated regarding oracle source tracking using, for instance, image processing. The proposed strategy is interesting for the robotics community, because it allows the development of voice-based HRI with limited training data and without relying on third-party technologies or Internet access eliminating the need to upload data to the cloud. In our mobile HRI scenario, the resulting speech recognition engine provided an average word error rate that is at least 19% and 34% lower than publicly available speech recognition APIs with the playback (i.e., loudspeaker) and human testing modalities, respectively.