Low-resource Speech Recognition Research Articles

Meta-Adaptable-Adapter: Efficient adaptation of self-supervised models for low-resource speech recognition

Self-supervised models have demonstrated remarkable performance in speech processing by learning latent representations from large amounts of unlabeled data. Adapting these models to low-resource languages yields promising results, but the computational cost of fine-tuning all model parameters is prohibitively high. Adapters offer a solution by introducing lightweight bottleneck structures into pre-trained models for downstream tasks, enabling efficient parameter adaptation. However, randomly initialized adapters often underperform in extremely low-resource scenarios. To address this issue, we explore the Meta-Adapter for self-supervised models and analyzed some limitations of Meta-Adapter including poor learning in language-specific knowledge and meta-overfitting problems. To relieve these problems, we propose the Meta-Adaptable-Adapter (MAA), a new meta leaning algorithm that adapts to low-resource languages quickly and effectively. MAA learns task-specific adapters for feature extraction, and task-independent adapters for feature combination. The experiments on three datasets show superior performance on 31 low-resource languages across seven different language families compared to other adapters, showing better generalization and extensibility.

A Lightweight Task-Agreement Meta Learning for Low-Resource Speech Recognition

Meta-learning has proven to be a powerful paradigm for transferring knowledge from prior tasks to facilitate the quick learning of new tasks in automatic speech recognition. However, the differences between languages (tasks) lead to variations in task learning directions, causing the harmful competition for model’s limited resources. To address this challenge, we introduce the task-agreement multilingual meta-learning (TAMML), which adopts the gradient agreement algorithm to guide the model parameters towards a direction where tasks exhibit greater consistency. However, the computation and storage cost of TAMML grows dramatically with model’s depth increases. To address this, we further propose a simplification called TAMML-Light which only uses the output layer for gradient calculation. Experiments on three datasets demonstrate that TAMML and TAMML-Light achieve outperform meta-learning approaches, yielding superior results.Furthermore, TAMML-Light can reduce at least 80 %\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} of the relative increased computation expenses compared to TAMML.

CAM: A cross-lingual adaptation framework for low-resource language speech recognition

In this paper, a novel cross-lingual adaptation framework called CAM is presented for low-resource language speech recognition (LLSR). It is based on the recent popular adapter method. CAM is achieved by adapting self-supervised speech models (SSMs) from source languages to target low-resource languages in a two-stage process. CAM fills two research gaps existing in current methods: (i) language similarity is not effectively considered; (ii) the performance-efficiency trade-off is not well balanced. Specifically, two key components, a similarity-aware fusion module (SAFM) and an adapter weight-sharing strategy (AWSS), are designed. A well-trained adapter is introduced by SAFM to compute precise language similarities via dot product. AWSS takes both performance and efficiency into consideration by sharing adapter weights. Experimental results on two corpora, FLEURS and Common Voice, demonstrate that CAM equipped with these two designs, denoted as performance-oriented CAM (P-CAM), obtains state-of-the-art (SOTA) performance with satisfactory efficiency compared to current leading methods. Besides, an efficiency-oriented CAM (E-CAM) by introducing a weight-space fusion module (WSFM) is presented. The core of WSFM is to average the weights of multiple adapters into a new adapter. Compared to full fine-tuning, only 5.0% of trainable parameters are required for E-CAM while a 2.3% relative average word error rate (WER) reduction is obtained. CAM brings a performance- or efficiency-oriented trade-off to meet the different needs of speech recognition systems.

Exploration of Whisper fine-tuning strategies for low-resource ASR

Limited data availability remains a significant challenge for Whisper’s low-resource speech recognition performance, falling short of practical application requirements. While previous studies have successfully reduced the recognition error rates of target language speech through fine-tuning, a comprehensive exploration and analysis of Whisper’s fine-tuning capabilities and the advantages and disadvantages of various fine-tuning strategies are still lacking. This paper aims to fill this gap by conducting comprehensive experimental exploration for Whisper’s low-resource speech recognition performance using five fine-tuning strategies with limited supervised data from seven low-resource languages. The results and analysis demonstrate that all fine-tuning strategies explored in this paper significantly enhance Whisper’s performance. However, different strategies vary in their suitability and practical effectiveness, highlighting the need for careful selection based on specific use cases and resources available.

Exploring Generation of Pronunciation Lexicon for Low-Resource Language Automatic Speech Recognition Based on Generic Phone Recognizer

Exploring Generation of Pronunciation Lexicon for Low-Resource Language Automatic Speech Recognition Based on Generic Phone Recognizer

Improving cross-lingual low-resource speech recognition by Task-based Meta PolyLoss

Multilingual meta learning has emerged as a promising paradigm for transferring knowledge from source languages to facilitate the learning of low-resource target languages. Loss functions are a type of meta-knowledge that is crucial to the effective training of neural networks. However, the misalignment between the loss functions and the learning paradigms of meta learning degrades the network’s performance. To address this challenge, we propose a new method called Task-based Meta PolyLoss (TMPL) for meta learning. By regarding speech recognition tasks as normal samples and applying PolyLoss to the meta loss function, TMPL can be denoted as a linear combination of polynomial functions based on task query loss. Theoretical analysis shows that TMPL improves meta learning by enabling attention adjustment across different tasks, which can be tailored for different datasets. Experiments on three datasets demonstrated that gradient-based meta learning methods achieve superior performance with TMPL. Furthermore, our experiments validate that the task-based loss function effectively mitigates the misalignment issue.

A comparative study of machine learning-based Chinese dialect speech recognition

As the official language of China, Mandarin is used on all formal occasions. However, due to China's rich history and vast population, dialects have become the primary language spoken by many people. While not taught in school, dialects serve as unofficial languages in China and are passed down through word of mouth. In recent years, the use of dialects has significantly declined as people have adapted to using official Mandarin in all settings. However, among the elderly, dialects remain the primary language of communication. The great need to use dialects as the primary mode of communication has laid the foundation for the importance and high demand for dialect speech recognition. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, research on this topic is further analysed and compared. The results suggest that there has been a certain degree of development and progress in dialect speech recognition technology. However, further research is needed to overcome the limitations of low-resource language speech recognition.

Language fusion via adapters for low-resource speech recognition

Data scarcity makes low-resource speech recognition systems suffer from severe overfitting. Although fine-tuning addresses this issue to some extent, it leads to parameter-inefficient training. In this paper, a novel language knowledge fusion method, named LanFusion, is proposed. It is built on the recent popular adapter-tuning technique, thus maintaining better parameter efficiency compared with conventional fine-tuning methods. LanFusion is a two-stage method. Specifically, multiple adapters are first trained on several source languages to extract language-specific and language-invariant knowledge. Then, the trained adapters are re-trained on the target low-resource language to fuse the learned knowledge. Compared with Vanilla-adapter, LanFusion obtains a relative average word error rate (WER) reduction of 9.8% and 8.6% on the Common Voice and FLEURS corpora, respectively. Extensive experiments demonstrate the proposed method is not only simple and effective but also parameter-efficient. Besides, using source languages that are geographically similar to the target language yields better results on both datasets.

Task-based Meta Focal Loss for Multilingual Low-resource Speech Recognition

Low-resource automatic speech recognition is a challenging task due to a lack of labeled training data. To resolve this issue, multilingual meta-learning learns a better model initialization from many source language tasks for fast adaptation to unseen target languages. However, for diverse source languages, the quantity and difficulty vary greatly because of their different data scales and phonological systems. These differences lead to task-quantity and task-difficulty imbalance issues and thus a failure of multilingual meta-learning ASR. In this work, we propose a task-based meta focal loss (TMFL) approach to address this tough challenge. Specifically, we introduce a hard-task moderator and update the meta-parameters using gradients from both the support set and query set. Our proposed approach focuses more on hard tasks and makes full use of the data from hard tasks. Moreover, we analyze the significance of the hard task moderator and interpret its significance at the sample level. Experiment results show that the proposed method, TMFL, significantly outperforms the state-of-the-art multilingual meta-learning on all target languages for the IARPA BABEL and OpenSLR datasets, especially under very-low-resource conditions. In particular, it can reduce character error rate from 72% to 60% by fine-tuning the pre-trained model with about 22 hours of Vietnamese data.

Frontier Research on Low-Resource Speech Recognition Technology.

With the development of continuous speech recognition technology, users have put forward higher requirements in terms of speech recognition accuracy. Low-resource speech recognition, as a typical speech recognition technology under restricted conditions, has become a research hotspot nowadays because of its low recognition rate and great application value. Under the premise of low-resource speech recognition technology, this paper reviews the research status of feature extraction and acoustic models, and conducts research on resource expansion. Especially in terms of the technical challenges faced by this technology, solutions are proposed, and future research directions are prospected.

Meta adversarial learning improves low-resource speech recognition

Low-resource automatic speech recognition is a challenging task. To resolve this issue, multilingual meta-learning learns a better model initialization from many source languages, allowing for rapid adaption to target languages. However, differences in data scales and learning difficulties vary greatly from one language to another. As a result, the model favors large-scale and simple source languages. Moreover, the shared semantic space of various languages is difficult to learn due to a lack of restrictions on multilingual pre-training. In this paper, we propose a meta adversarial learning approach to address this problem. The meta-learner will be guided to learn language-independent information by using an adversarial auxiliary objective of language identification, which makes the shared semantic space more compact and improves model generalization. Additionally, we optimize adversarial training using Wasserstein distance and temporal normalization, enabling more stable and simple training. Experiment results on IARPA BABEL and OpenSLR show a significant performance improvement. It also outperforms state-of-the-art results by a large margin in all target languages, and especially in few-shot settings. Finally, we demonstrate how our method is superior by using t-SNE visualization.

Boosting Low-Resource Speech Recognition in Air Traffic Communication via Pretrained Feature Aggregation and Multi-Task Learning

Developing a robust Automatic Speech Recognition (ASR) system usually requires a large amount of well-annotated samples which is extremely hard to build in the Air Traffic Control (ATC) due to domain-specific knowledge. In this work, we present a novel approach to improve ASR performance in the ATC domain by integrating Self-Supervised Learning and Multi-Task learning into a unified framework. Specifically, the proposed framework follows a two-stage training paradigm, i.e., (a) learning universal acoustic representations by employing the wav2vec 2.0 model and (b) jointly finetuning the model by the ASR, Speaker Role Identification, and Language Identification tasks. To capture the task-specific representations, an attention-guided feature aggregation module is dedicatedly designed to disentangle the discriminative representations from the pretrained features. In addition, the uncertainty-based loss combination strategy is employed to balance the loss weights for each task in a learnable manner. Finally, we conduct experiments to validate the technical improvements in a real-world ATC dataset. Experimental results demonstrated that the proposed framework outperforms competitive baselines among all tasks.

JSUM: A Multitask Learning Speech Recognition Model for Jointly Supervised and Unsupervised Learning

In recent years, the end-to-end speech recognition model has emerged as a popular alternative to the traditional Deep Neural Network—Hidden Markov Model (DNN-HMM). This approach maps acoustic features directly onto text sequences via a single network architecture, significantly streamlining the model construction process. However, the training of end-to-end speech recognition models typically necessitates a significant quantity of supervised data to achieve good performance, which poses a challenge in low-resource conditions. The use of unsupervised representation significantly reduces this necessity. Recent research has focused on end-to-end techniques employing joint Connectionist Temporal Classification (CTC) and attention mechanisms, with some also concentrating on unsupervised presentation learning. This paper proposes a joint supervised and unsupervised multi-task learning model (JSUM). Our approach leverages the unsupervised pre-trained wav2vec 2.0 model as a shared encoder that integrates the joint CTC-Attention network and the generative adversarial network into a unified end-to-end architecture. Our method provides a new low-resource language speech recognition solution that optimally utilizes supervised and unsupervised datasets by combining CTC, attention, and generative adversarial losses. Furthermore, our proposed approach is suitable for both monolingual and cross-lingual scenarios.

A Method Improves Speech Recognition with Contrastive Learning in Low-Resource Languages

Building an effective automatic speech recognition system typically requires a large amount of high-quality labeled data; However, this can be challenging for low-resource languages. Currently, self-supervised contrastive learning has shown promising results in low-resource automatic speech recognition, but there is no discussion on the quality of negative sample sets in speech contrastive learning. In this paper, we propose the false negatives impact elimination (FNIE) method to filter false negative samples and improve the quality of the negative sample set in speech. FNIE compares the support vector with the negative sample vector set and optimizes the corresponding loss function, allowing the model to learn better speech representations and achieve superior results in low-resource speech recognition. Experiments demonstrate that FNIE effectively filters negative samples, enhances the quality of the negative sample set, and improves the accuracy of speech recognition. The quality of the negative sample set significantly affects the model’s learning ability, and using too many negative samples can deteriorate it. In a low-resource setting, our FNIE method achieved a relative improvement of 2.98% in WER on the English dataset, 14.3% in WER on the Uyghur dataset, and 4.04% in CER on the Mandarin dataset compared to the baseline model.

Matrix Quantization and LPC Vocoder Based Linear Predictive for Low-Resource Speech Recognition system

Over the last ten years, there has been significant progress in the use of low-rate speech coders in voice applications for computers, military communications, and civil communications. This advancement has been made possible by the development of new speech coders that can generate high-quality speech at low data rates. The majority of existing coders include spectral representation of speech, speech waveform matching, and ”optimization” of the coder’s performance for human hearing. The goal of this paper is to provide a thorough evaluation of voice coding methods for educational purposes, with a particular emphasis on the algorithms used in low-rate cellular communication standards. The algorithm we developed using a voice-excited LPC vocoder produces clear, low-distortion results. Ordinary LPCs, on the other hand, fall short of vocoders because they can handle signals other than speech, such as music. To improve quality, additional bandwidth is used to reduce the bit rate. To improve the quality, we tried two approaches. The first was to increase the number of bits required to quantize the DCT coefficients. This coefficient would outperform the inverse DCT in closer error rearrangements. The second possibility is to increase the total number of quantized coefficients. As a result, error array rearrangements would be more accurate. The goal is to identify the point at which a method improvement outperforms the previous, better result. Other coding methods become more complex, but this vocoder suffices.

Low Resource Kannada Speech Recognition using Lattice Rescoring and Speech Synthesis

Objectives: Improving the accuracy of low resource speech recognition in a model trained on only 4 hours of transcribed continuous speech in Kannada language, using data augmentation. Methods: Baseline language model is augmented with unigram counts of words, that are present in the Wikipedia text corpus but absent in the baseline, for initial decoding. Lattice rescoring is then applied using the language model augmented with Wikipedia text. Speech synthesis-based augmentation with multi-speaker syllable-based synthesis, using voices in Kannada and cross-lingual Telugu languages, is employed. We synthesize basic syllables, syllables with consonant conjuncts, and words that contain syllables that are absent in the training speech, for Kannada language. Findings: An overall word error rate (WER) of 9.04% is achieved over a baseline WER of 40.93%. Language model augmentation and lattice rescoring gives an absolute improvement of 16.68%. Applying our method of syllable-based speech synthesis over language model augmentation and rescoring yields a total reduction of 31.89% in WER. The proposed approach of language model augmentation is memory efficient and consumes only 1/8th the memory required for decoding with Wikipedia augmented language model (2 gigabytes versus 18 gigabytes) while giving comparable WER (22.95% for Wikipedia versus 24.25% for our method). Augmentation with synthesized syllables enhances the ability of the speech recognition model to recognize basic sounds thus improving recognition of out-of-vocabulary words to 90% and in-vocabulary words to 97%. Novelty: We propose novel methods of language model augmentation and synthesis-based augmentation to achieve low WER for a speech recognition model trained on only 4 hours of continuous speech. Obtaining high recognition accuracy (or low WER) for very small speech corpus is a challenge. In this paper, we demonstrate that high accuracy can be achieved using data augmentation for a small corpus-based speech recognition. Keywords: Low resource; Speech synthesis; Data augmentation; Language model; Lattice rescoring

A Study of Speech Recognition for Kazakh Based on Unsupervised Pre-Training

Building a good speech recognition system usually requires a lot of pairing data, which poses a big challenge for low-resource languages, such as Kazakh. In recent years, unsupervised pre-training has achieved good performance in low-resource speech recognition, but it is rarely used in Kazakh and other Central and West Asian languages. In this paper, wav2vec2.0 is improved by integrating a Factorized TDNN layer to better preserve the relationship between the voice and the time step before and after the quantization, which is called wav2vec-F. The unsupervised pre-training strategy was used to learn the potential speech representation from a large number of unlabeled audio data and was applied to the cross-language ASR task, which was optimized using the noise contrast binary classification task. At the same time, speech synthesis is used to promote the performance of speech recognition. The experiment shows that wav2vec-F can effectively utilize the unlabeled data from non-target languages, and the multi-language pre-training is obviously better than the single-language pre-training. The data enhancement method using speech synthesis can bring huge benefits. Compared with the baseline model, Librispeech's test-clean dataset has an average reduction of 1.9% in the word error rate. On the Kazakh KSC test set, the pre-training using only Kazakh reduced the word error rate by 3.8%. The pre-training of a small amount of Kazakh speech data synthesized by multi-language combined with TTS achieved a word error rate of 8.6% on the KSC test set when the labeled data were only 10 h, which was comparable to the results of the previous end-to-end model when the labeled data were 30 times less.

A Semi-Supervised Complementary Joint Training Approach for Low-Resource Speech Recognition

A Semi-Supervised Complementary Joint Training Approach for Low-Resource Speech Recognition

Cross-Lingual Self-training to Learn Multilingual Representation for Low-Resource Speech Recognition

Cross-Lingual Self-training to Learn Multilingual Representation for Low-Resource Speech Recognition

Low-resource automatic speech recognition and error analyses of oral cancer speech

In this paper, we introduce a new corpus of oral cancer speech and present our study on the automatic recognition and analysis of oral cancer speech. A two-hour English oral cancer speech dataset is collected from YouTube. Formulated as a low-resource oral cancer ASR task, we investigate three acoustic modelling approaches that previously have worked well with low-resource scenarios using two different architectures; a hybrid architecture and a transformer-based end-to-end (E2E) model: (1) a retraining approach; (2) a speaker adaptation approach; and (3) a disentangled representation learning approach (only using the hybrid architecture). The approaches achieve a (1) 4.7% (hybrid) and 7.5% (E2E); (2) 7.7%; and (3) 2.0% absolute word error rate reduction, respectively, compared to a baseline system which is not trained on oral cancer speech. A detailed analysis of the speech recognition results shows that (1) plosives and certain vowels are the most difficult sounds to recognise in oral cancer speech — this problem is successfully alleviated by our proposed approaches; (3) however these sounds are also relatively poorly recognised in the case of healthy speech with the exception of/p/. (2) recognition performance of certain phonemes is strongly data-dependent; (4) In terms of the manner of articulation, E2E performs better with the exception of vowels — however, vowels have a large contribution to overall performance. As for the place of articulation, vowels, labiodentals, dentals and glottals are better captured by hybrid models, E2E is better on bilabial, alveolar, postalveolar, palatal and velar information. (5) Finally, our analysis provides some guidelines for selecting words that can be used as voice commands for ASR systems for oral cancer speakers.