Multilingual Speech Research Articles

Spoken Language Identification Using Prosody, Phonotactics, and Acoustics: A Review

Spoken language identification (LID) is the identification of language present in a speech segment despite its size (duration and speed), ambiance (topic and emotion), and moderator (gender, age, demographic region). Information Technology has touched new vistas for a couple of decades mostly to simplify the day-to-day life of humans. One of the key contributions of Information Technology is the application of Artificial Intelligence to achieve better results. The advent of artificial intelligence has given rise to a new branch of Natural Language Processing (NLP) called Computational Linguistics, which generates frameworks for intelligently manipulating spoken language knowledge and has brought human–machine into a new stage. In this context, speech has arisen to be one of the imperative forms of interfaces, which is the basic mode of communication for us, and generally the most preferred one. Recognition of the spoken language is a frontend for several technologies, like multiple languages conversation systems, expressed translation software, multilingual speech recognition, spoken word extraction, speech production systems. This paper reviews and summarises the different levels of information that can be used for language identification. A broad study of acoustic, phonetic, and prosody features has been provided and various classifiers have been used for spoken language identification specifically for Indian languages. This paper has investigated various existing spoken language identification models implemented using prosodic, phonotactic, acoustic, and deep learning approaches, the datasets used, and performance measures utilized for their analysis. It also highlights the main features and challenges faced by these models. Moreover, this review analyses the efficiency of the spoken language models that can help the researchers to propose new language identification models for speech signals.

Multilingual hope speech detection in English and Dravidian languages

Recent work on language technology has aimed to identify negative language such as hate speech and cyberbullying as well as improve offensive language detection to mediate social media platforms. Most of these systems rely on using machine learning models along with the labelled dataset. Such models have succeeded in identifying negativity and removing it from the platform deleting it. However, recently, more research has been conducted on the improvement of freedom of speech on social media. Instead of deleting supposedly offensive speech, we developed a multilingual dataset to identify hope speech in the comments and promote positivity. This paper presents a multilingual hope speech dataset that promotes equality, diversity and inclusion (EDI) in English, Tamil, Malayalam and Kannada. It was collected to promote positivity and ensure EDI in language technology. Our dataset is unique, as it contains data collected from the LGBTQIA+ community, persons with disabilities and women in science, engineering, technology and management (STEM). We also report our benchmark system results in various machine learning models. We experimented on the Hope Speech dataset for Equality, Diversity and Inclusion (HopeEDI) using different state-of-the-art machine learning models and deep learning models to create benchmark systems.

Multilingual Speech to Text using Deep Learning based on MFCC Features

The proposed methodology presented in the paper deals with solving the problem of multilingual speech recognition. Current text and speech recognition and translation methods have a very low accuracy in translating sentences which contain a mixture of two or more different languages. The paper proposes a novel approach to tackling this problem and highlights some of the drawbacks of current recognition and translation methods. The proposed approach deals with recognition of audio queries which contain a mixture of words in two different languages - Kannada and English. The novelty in the approach presented, is the use of a next Word Prediction model in combination with a Deep Learning speech recognition model to accurately recognise and convert the input audio query to text. Another method proposed to solve the problem of multilingual speech recognition and translation is the use of cosine similarity between the audio features of words for fast and accurate recognition. The dataset used for training and testing the models was generated manually by the authors as there was no pre-existing audio and text dataset which contained sentences in a mixture of both Kannada and English. The DL speech recognition model in combination with the Word Prediction model gives an accuracy of 71% when tested on the in-house multilingual dataset. This method outperforms other existing translation and recognition solutions for the same test set. Multilingual translation and recognition is an important problem to tackle as there is a tendency for people to speak in a mixture of languages. By solving this problem, the barrier of language and communication can be lifted and thus can help people connect better and more comfortably with each other.

Transcribing multilingual children’s and adults’ speech

ABSTRACT Speech-language pathologists (SLPs) face challenges in transcription and diagnosis of speech sound disorders (SSD) in multilingual children due to ambient language influences and cross-linguistic transfer. The VietSpeech Multilingual Transcription Protocol, a 4-step process to undertake impressionistic transcription of multilingual speech was tested using data from Vietnamese-Australian children (n = 69) and adult family members (n = 85). The transcription team included an English-speaking SLP, a Vietnamese-English-speaking linguist and accredited interpreter, and two Vietnamese-English-speaking SLPs. (1) Training: The team completed training together in Vietnamese and English phonology. (2) Speech assessment: The participants were assessed using the Diagnostic Evaluation of Articulation and Phonology (DEAP) in English and the Vietnamese Speech Assessment (VSA). (3) Transcription comparison: Inter-rater reliability for 10 children and 12 adults was calculated using consonant-by-consonant agreement. For English the 3-way inter-rater agreement was 92.62% for children and 88.69% for adults. For Vietnamese the 4-way inter-rater agreement was 86.57% for children and 96.05% for adults. There was a significant correlation between speech accuracy and inter-rater reliability for children’s consonants in English (r = 0.95) and Vietnamese (r = 0.91), and adults’ consonants in English (r = 0.90), but not for Vietnamese (r = 0.49). Reliability was influenced by phoneme class and whether the target consonant was shared between languages. (4) Rule generation and consensus: Rules based on near functional equivalence were implemented to maintain consistency and reach consensus. SLPs who do not speak clients’ home languages can be supported to transcribe multilingual speech by working with multilingual teams, and understanding personal limitations during multilingual speech assessments.

A Multi-Lingual Speech Recognition-Based Framework to Human-Drone Interaction

In recent years, human–drone interaction has received increasing interest from the scientific community. When interacting with a drone, humans assume a variety of roles, the nature of which are determined by the drone’s application and degree of autonomy. Common methods of controlling drone movements include by RF remote control and ground control station. These devices are often difficult to manipulate and may even require some training. An alternative is to use innovative methods called natural user interfaces that allow users to interact with drones in an intuitive manner using speech. However, using only one language of interacting may limit the number of users, especially if different languages are spoken in the same region. Moreover, environmental and propellers noise make speech recognition a complicated task. The goal of this work is to use a multilingual speech recognition system that includes English, Arabic, and Amazigh to control the movement of drones. The reason for selecting these languages is that they are widely spoken in many regions, particularly in the Middle East and North Africa (MENA) zone. To achieve this goal, a two-stage approach is proposed. During the first stage, a deep learning based model for multilingual speech recognition is designed. Then, the developed model is deployed in real settings using a quadrotor UAV. The network was trained using 38,850 records including commands and unknown words mixed with noise to improve robustness. An average class accuracy of more than 93% has been achieved. After that, experiments were conducted involving 16 participants giving voice commands in order to test the efficiency of the designed system. The achieved accuracy is about 93.76% for English recognition and 88.55%, 82.31% for Arabic and Amazigh, respectively. Finally, hardware implementation of the designed system on a quadrotor UAV was made. Real time tests have shown that the approach is very promising as an alternative form of human–drone interaction while offering the benefit of control simplicity.

A hybrid CTC+Attention model based on end-to-end framework for multilingual speech recognition

Speech recognition is an important field in natural language processing. In this paper, the end-to-end framework for speech recognition with multilingual datasets is proposed. The end-to-end methods do not require complicated alignment and construction of the pronunciation dictionary, which show a promising prospect. In this paper, we implement a hybrid model of CTC and attention (CTC+Attention) model based on PyTorch. In order to compare speech recognition methods for multiple languages, we design and create three datasets: Chinese, English, and Code-Switch. We evaluate the proposed hybrid CTC+Attention model in multilingual environment. Throughout our experiments, we find that the proposed hybrid CTC+Attention model based on end-to-end framework achieves better performance compared with the HMM-DNN model in a single language and Code-Switch speaking environment. Moreover, the results of speech recognition with regard to different languages are compared in this paper. The CER(i.e., Character Error Rate) of the proposed hybrid CTC+Attention model based on the Chinese dataset defeated the traditional model and reached 10.22%.

Multilingual speech recognition for GlobalPhone languages

In this paper, we present our investigations towards the development of multilingual automatic speech recognition (ML ASR) systems using the GlobalPhone database. In addition to GlobalPhone, we have included 4 Ethiopian languages (Amharic, Oromo, Tigrigna and Wolaytta), as well as Uyghur and English in our investigation. In order to see the impact of language relatedness in ML ASR training, we have analyzed both phonetic overlap and morphological complexity of the languages. Deep Neural Network based ML ASR systems have been developed using ML mix, transfer and multitask learning approaches. Relative word error rate (WER) reductions up to 33.21% have been achieved as a result of using resources of other languages in ML acoustic model training. Our experimental results show that languages with small amounts of monolingual training data benefit a lot from ML training. Moreover, using phonetically related languages in ML training is more beneficiary than using phonetically less related languages. It seems that the nature of the corpus (single or mixed domain, noisy or noise free, etc.) has also an impact in ML training although it is not as important as the phonetic relatedness of the languages.

An optimized machine translation technique for multi-lingual speech to sign language notation

An optimized machine translation technique for multi-lingual speech to sign language notation

SuperSpeech: Multilingual Speech and Language Maintenance Intervention for Vietnamese-Australian Children and Families via Telepractice.

The aim of this pilot feasibility study was to evaluate the effectiveness of the group VietSpeech SuperSpeech program targeting speech skills and home language maintenance via telepractice. In Stage 1, using a case-control design, 30 Vietnamese-English-speaking children were assessed in English and Vietnamese, and parents completed questionnaires about speech and language competency and practices. During Stage 2, children were allocated to intervention (n = 14) or control (n = 16) conditions. COVID-19 restrictions resulted in changes including nonrandom allocation. Online group intervention targeting speech, home language maintenance, and multilingualism as a superpower was delivered 1 hr/week for 8 weeks. For Stage 3, assessments were undertaken approximately 10 weeks after the pre-intervention assessment. Parents in the intervention group significantly increased encouragement of their children to speak Vietnamese. The intervention group significantly increased intelligibility in English. Growth of Vietnamese vocabulary was faster for the control group. There was a moderate effect of intervention for children's perception of being happy talking in Vietnamese and English. There was no significant mean change from pre- to post-intervention compared with the control group for measures of speech sound accuracy in Vietnamese or English, Vietnamese intelligibility, English vocabulary, or hours of Vietnamese spoken each week. This study presents preliminary evidence that this 8-hr online group program targeting speech skills and home language maintenance had some impact on Vietnamese-Australian children's speech and home language maintenance. Further research involving a randomized trial is warranted.

Improving low-resource Tibetan end-to-end ASR by multilingual and multilevel unit modeling

Conventional automatic speech recognition (ASR) and emerging end-to-end (E2E) speech recognition have achieved promising results after being provided with sufficient resources. However, for low-resource language, the current ASR is still challenging. The Lhasa dialect is the most widespread Tibetan dialect and has a wealth of speakers and transcriptions. Hence, it is meaningful to apply the ASR technique to the Lhasa dialect for historical heritage protection and cultural exchange. Previous work on Tibetan speech recognition focused on selecting phone-level acoustic modeling units and incorporating tonal information but underestimated the influence of limited data. The purpose of this paper is to improve the speech recognition performance of the low-resource Lhasa dialect by adopting multilingual speech recognition technology on the E2E structure based on the transfer learning framework. Using transfer learning, we first establish a monolingual E2E ASR system for the Lhasa dialect with different source languages to initialize the ASR model to compare the positive effects of source languages on the Tibetan ASR model. We further propose a multilingual E2E ASR system by utilizing initialization strategies with different source languages and multilevel units, which is proposed for the first time. Our experiments show that the performance of the proposed method-based ASR system exceeds that of the E2E baseline ASR system. Our proposed method effectively models the low-resource Lhasa dialect and achieves a relative 14.2% performance improvement in character error rate (CER) compared to DNN-HMM systems. Moreover, from the best monolingual E2E model to the best multilingual E2E model of the Lhasa dialect, the system’s performance increased by 8.4% in CER.

Optimizing Data Usage for Low-Resource Speech Recognition

Automatic speech recognition has made huge progress recently. However, the current modeling strategy still suffers a large performance degradation when facing the low-resource languages with limited training data. In this paper, we propose a series of methods to optimize the data usage for low-resource speech recognition. Multilingual speech recognition helps a lot in low-resource scenarios. The correlation and similarity between languages are further exploited for multilingual pretraining in our work. We utilize the posterior of the target language extracted from a language classifier to perform data weighing on training samples, which assists the model in being more biased towards the target language during pretraining. Furthermore, dynamic curriculum learning for data allocation and length perturbation for data augmentation are also designed. All these three methods form the new strategy on optimized data usage for low-resource languages. We evaluate the proposed method using rich resource languages for pretraining (PT) and finetuning (FT) the model on the target language with limited data. Experimental results show that the proposed data usage method obtains a 15 to 25&#x0025; relative word error rate reduction for different target languages compared with the commonly adopted multilingual PT+FT method on CommonVoice dataset. The same improvement and conclusion are also observed on Babel dataset with conversational telephone speech, and <inline-formula><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula>40&#x0025; relative character error rate reduction can be obtained for the target low-resource language.

Is Attention Always Needed? A Case Study on Language Identification from Speech

Language Identification (LID), a recommended initial step to Automatic Speech Recognition (ASR), is used to detect a spoken language from audio specimens. In state-of-the-art systems capable of multilingual speech processing, however, users have to explicitly set one or more languages before using them. LID, therefore, plays a very important role in situations where ASR based systems cannot parse the uttered language in multilingual contexts causing failure in speech recognition. We propose an attention based convolutional recurrent neural network (CRNN with Attention) that works on Mel-frequency Cepstral Coefficient (MFCC) features of audio specimens. Additionally, we reproduce some state-of-the-art approaches, namely Convolutional Neural Network (CNN) and Convolutional Recurrent Neural Network (CRNN), and compare them to our proposed method. We performed extensive evaluation on thirteen different Indian languages and our model achieves classification accuracy over 98%. Our LID model is robust to noise and provides 91.2% accuracy in a noisy scenario. The proposed model is easily extensible to new languages.

An automatic machine translation system for multi-lingual speech to Indian sign language

An automatic machine translation system for multi-lingual speech to Indian sign language

Creating a Corpus of Multilingual Parent-Child Speech Remotely: Lessons Learned in a Large-Scale Onscreen Picturebook Sharing Task.

With lockdowns and social distancing measures in place, research teams looking to collect naturalistic parent-child speech interactions have to develop alternatives to in-lab recordings and observational studies with long-stretch recordings. We designed a novel micro-longitudinal study, the Talk Together Study, which allowed us to create a rich corpus of parent-child speech interactions in a fully online environment (N participants = 142, N recordings = 410). In this paper, we discuss the methods we used, and the lessons learned during adapting and running the study. These lessons learned cover nine domains of research design, monitoring and feedback: Recruitment strategies, Surveys and Questionnaires, Video-call scheduling, Speech elicitation tools, Videocall protocols, Participant remuneration strategies, Project monitoring, Participant retention, and Data Quality, and may be used as a primer for teams planning to conduct remote studies in the future.

Investigating into the varieties of language spoken at Benue State University, Makurdi

In a multilingual speech community like Benue State University, Makurdi, where the members thereof have different native language backgrounds and English is chosen to be the official language, members of the university community speak different varieties of language in different domains. The paper entitled ‘Investigating into the varieties of language spoken at Benue State University (BSU), Makurdi’ is set to examine the varieties of language spoken on the BSU’s campus. The survey design was adopted, and questionnaires were used as the instruments for data collection for the paper. The analysis of the varieties of language spoken at the university was based on William Labov’s Variationist Theory which claims that differences are featured in the use of language among individuals based on the contexts. Through the analysis of the data, the researchers discovered that English, Pidgin, and native languages such as Tiv, Idoma, Igede, Etulo, Igbo, Hausa, among others, are spoken at the university. The study also discovered that Pidgin is predominantly used at the university despite the fact that English is the official language because academics and lecture rooms may be the only domains and settings that compulsorily enforce the use of English. The researchers, therefore, suggest that more efforts should be made to ensure a predominant use of English on the BSU’s campus since it is the official and prestigious language of Nigeria.

Multilingual speech annotation of landmarks and other acoustic cues to distinctive features

A speech annotation system developed for English that identifies landmarks and other acoustic cues to distinctive features (Stevens, 2002; Huilgol et al., 2019) has been extended to Spanish and Korean. This process includes retrieving the (allo)phone sequence for the words of a target utterance from a standard lexicon, converting this sequence into an underlying phoneme sequence, and then generating citation-form landmark and other acoustic cue sequences from the phoneme sequence and its features. This standard set is presented to labelers who examine the waveform of the target utterance and mark which of the predicted cues have been realized, or deleted, and where unpredicted cues have been inserted. For each language, we draw from a superset of acoustic cues that function in all three languages, but each language also makes use of an additional set of acoustic cues specific to that language.

ATCSpeechNet: A multilingual end-to-end speech recognition framework for air traffic control systems

In this paper, a multilingual end-to-end framework, called ATCSpeechNet, is proposed to tackle the issue of translating communication speech into human-readable text in air traffic control (ATC) systems. In the proposed framework, we focus on integrating multilingual automatic speech recognition (ASR) into one model, in which an end-to-end paradigm is developed to convert speech waveforms into text directly, without any feature engineering or lexicon. To compensate the deficiency of handcrafted feature engineering caused by ATC challenges, including multilingual, multispeaker dialog and unstable speech rates, a speech representation learning (SRL) network is proposed to capture robust and discriminative speech representations from raw waves. The self-supervised training strategy is adopted to optimize the SRL network from unlabeled data, and to further predict the speech features, i.e., wave-to-feature. An end-to-end architecture is improved to complete the ASR task, in which a grapheme-based modeling unit is applied to address the multilingual ASR issue. Facing the problem of small transcribed samples in the ATC domain, an unsupervised approach with mask prediction is applied to pretrain the backbone network of the ASR model on unlabeled data by a feature-to-feature process. Finally, by integrating the SRL with ASR, an end-to-end multilingual ASR framework is formulated in a supervised manner, which is able to translate the raw wave into text in one model, i.e., wave-to-text. Experimental results on the ATCSpeech corpus demonstrate that the proposed approach achieves high performance with a very small labeled corpus and less resource consumption, only a 4.20% label error rate on the 58-hour transcribed corpus. Compared to the baseline model, the proposed approach obtains over 100% relative performance improvement which can be further enhanced with increasing size of the transcribed samples. It is also confirmed that the proposed SRL and training strategies make significant contributions to improving the final performance. In addition, the effectiveness of the proposed framework is also validated on common corpora (AISHELL, LibriSpeech, and cv-fr). More importantly, the proposed multilingual framework not only reduces the system complexity but also obtains higher accuracy compared to that of the independent monolingual ASR models. The proposed approach can also greatly reduce the cost of annotating samples, which benefits to advance the ASR technique to industrial applications.

Patterns and perspectives shape perceptions: Epistemological and methodological reflections on the study of small-scale multilingualism

Aims and objectives:This paper captures social dimensions of language in highly diverse small-scale multilingual contexts that appear to pose challenges for (socio)linguistic description and documentation. I focus on the seeming contradiction of monolingual imaginations of places with heterogeneous and multilingual inhabitants, on great fluidity and variability of language use and the concomitant limits of reification-based identification of codes, and on personalised repertoires shaped by individual trajectories and relational, rather than categorical, stances.Approach:I propose patterns and perspectives as two interrelated dimensions to guide research in configurations of this kind, illustrating epistemological and methodological points through data from multilingual settings in Casamance, Senegal.Data and analysis:I focus on data collected in the village of Agnack Grand and its surroundings, but also include data from across the Lower Casamance and adjacent regions of Guinea-Bissau, discussing patterns of multilingual organisation and extracts from conversation and how their speech forms are categorised.Findings:The paper brings sociohistorical dimensions of small-scale multilingualism to the fore and identifies their lasting influences on spatial representations of language regimes. Linguistic spaces influence perspectives on speech events taking place in them and circumscribe speech participants’ and observers’ choices in describing repertoires, producing and analysing speech forms. Beyond the selection of language modes, perspective also determines how speech forms are categorised. I demonstrate that the patterns speakers and observers have experienced and the perspectives they assume are decisive in shaping their perception.Originality:My central observation is that there is no objective, neutral viewpoint on (multilingual) speech, but that positionality frames it at all levels. I develop new epistemologies for studying these dimensions.Significance:Putting the categorisation processes employed by speakers and observers and their underlying motivations centre stage and integrating sociolinguistic and anthropological linguistic methods and historical knowledge into linguistic description and documentation constitutes an innovative research programme.

Towards a coherent methodology for the documentation of small-scale multilingualism: Dealing with speech data

Purpose: To contribute to the establishment of a novel approach to language documentation that includes bilingual and multilingual speech data. This approach would open this domain of study to work by specialists of bilingualism and multilingualism. Approach: Within language documentation, the approach adopted in this paper exemplifies the “contemporary communicative ecology” mode of documentation. This radically differs from the “ancestral-code” mode of documentation that characterizes most language documentation corpora. Within the context of multilingualism studies, this paper advocates for the inclusion of a strong ethnographic component to research on multilingualism. Data and Analysis: The data presented comes from a context characterized by small-scale multilingualism, and the analyses provided are by and large focused on uncovering aspects of local metapragmatics. Conclusions: Conducting language documentation in contexts of small-scale multilingualism requires that the adequacy of a corpus is assessed with regard to sociolinguistic, rather than only structural linguistic, requirements. The notion of sociolinguistic adequacy is discussed in detail in analytical terms and illustrated through an example taken from ongoing research led by the authors. Originality: To date, there are no existing publications reviewing in the detail provided here how the documentation of multilingual speech in contexts of small-scale multilingualism should be structured. The contribution is highly original, in particular, for its theoretical grounding of the proposed approach. Significance/Implications: This article can serve as a reference for those interested in methodological and theoretical concerns relating to the practice of language documentation in contexts of small-scale multilingualism across the world. It may also help clarify ways for sociolinguists to engage more closely with work on language documentation, a domain that has thus far remained primarily informed by structural linguistic approaches.

Code-switched automatic speech recognition in five South African languages

Most automatic speech recognition (ASR) systems are optimised for one specific language and their performance consequently deteriorates drastically when confronted with multilingual or code-switched speech. We describe our efforts to improve an ASR system that can process code-switched South African speech that contains English and four indigenous languages: isiZulu, isiXhosa, Sesotho and Setswana. We begin using a newly developed language-balanced corpus of code-switched speech compiled from South African soap operas, which are rich in spontaneous code-switching. The small size of the corpus makes this scenario under-resourced, and hence we explore several ways of addressing this sparsity of data. We consider augmenting the acoustic training sets with in-domain data at the expense of making it unbalanced and dominated by English. We further explore the inclusion of monolingual out-of-domain data in the constituent languages. For language modelling, we investigate the inclusion of out-of-domain text data sources and also the inclusion of synthetically-generated code-switch bigrams. In our experiments, we consider two system architectures. The first considers four bilingual speech recognisers, each allowing code-switching between English and one of the indigenous languages. The second considers a single pentalingual speech recogniser able to process switching between all five languages. We find that the additional inclusion of each acoustic and text data source leads to some improvements. While in-domain data is substantially more effective, performance gains were also achieved using out-of-domain data, which is often much easier to obtain. We also find that improvements are achieved in all five languages, even when the training set becomes unbalanced and heavily skewed in favour of English. Finally, we find the use of TDNN-F architectures for the acoustic model to consistently outperform TDNN–BLSTM models in our data-sparse scenario.