Automatic Language Identification Research Articles

Whisper-SV: Adapting Whisper for low-data-resource speaker verification

Trained on 680,000 h of massive speech data, Whisper is a multitasking, multilingual speech foundation model demonstrating superior performance in automatic speech recognition, translation, and language identification. However, its applicability in speaker verification (SV) tasks remains unexplored, particularly in low-data-resource scenarios where labeled speaker data in specific domains are limited. To fill this gap, we propose a lightweight adaptor framework to boost SV with Whisper, namely Whisper-SV. Given that Whisper is not specifically optimized for SV tasks, we introduce a representation selection module to quantify the speaker-specific characteristics contained in each layer of Whisper and select the top-k layers with prominent discriminative speaker features. To aggregate pivotal speaker-related features while diminishing non-speaker redundancies across the selected top-k distinct layers of Whisper, we design a multi-layer aggregation module in Whisper-SV to integrate multi-layer representations into a singular, compacted representation for SV. In the multi-layer aggregation module, we employ convolutional layers with shortcut connections among different layers to refine speaker characteristics derived from multi-layer representations from Whisper. In addition, an attention aggregation layer is used to reduce non-speaker interference and amplify speaker-specific cues for SV tasks. Finally, a simple classification module is used for speaker classification. Experiments on VoxCeleb1, FFSVC, and IMSV datasets demonstrate that Whisper-SV achieves EER/minDCF of 2.22%/0.307, 6.14%/0.488, and 7.50%/0.582, respectively, showing superior performance in low-data-resource SV scenarios.

Automatic language identification: a case study of Pahari languages

In an attempt to expand the inclusiveness of Natural Language Processing, this paper focuses on developing resources and building machine learning models to identify four languages of the Northern Indo-Aryan family, also known as Pahari languages—Nepali, Garhwali, Kumaoni, and Dogri. This is the first attempt towards building identification models for Pahari languages and developing a plain text corpus for Garhwali and Kumaoni, both of which are lesser-known and under-resourced languages/mother tongues of India. The collected corpus, including data in Nepali and Dogri, is statistically analyzed at the word level. We also trained traditional machine learning models for Pahari language identification on this corpus and found that character n-grams based Linear Support Vector Machines performed best with 99.28% accuracy.

Review of Automatic Language Identification System in Indian languages from the Non-Uniform Region

Abstract—The process of language identification involves au- tomatically detecting which natural language(s) are given in a speech sample. Using automatic language recognition, an audio clip can be recognised as being spoken in a specific language (LID). Identifying the language correctly from a given speech sample is the language identification system’s fundamental purpose. There are two types of systems they are explicit language identification and implicit language identification. The phoneme sequence used in the explicit language recognition system is produced from a speech sample. Language is determined using the phoneme sequence that is retrieved. There is no need to identify the phoneme in the implicit language recognition method, which determines using certain speech properties. The language identification process involves training and a classifier model. the first step is to extract the speech sample using phonemes then it is trained using classifiers. Data are randomly collected from male and female peoples in society. Index Terms—Language identification (LID), Mel-frequency cepstral coefficients (MFCC), broad phoneme classifier (BPC),Neural network, support vector machine (SVM), LDA, CNN,i-vector,GMM Super Vector(GSV),Factor Analysis(FA),Native Language Identification (NLI),Stylistically Related Text Samples (SSTs),Deep Dumb Multi Layer Perceptron (DDMLP), Deep Convolutional Neural Network (DCNN), and Semi-supervised Generative Adversarial Network(SSGAN),Gated Recurrent Units (GRUs),Hidden Markov models (HMMs),phone error rate (PER),Convolutional and Long Short Term Memory Recurrent (CLSTM), Neural Networks ,Singing Language Identification (SLID),Automatic Speech Recognition (ASR).

Grey wolf optimization-extreme learning machine for automatic spoken language identification

Grey wolf optimization-extreme learning machine for automatic spoken language identification

An Overview of Indian Spoken Language Recognition from Machine Learning Perspective

Automatic spoken language identification (LID) is a very important research field in the era of multilingual voice-command-based human-computer interaction. A front-end LID module helps to improve the performance of many speech-based applications in the multilingual scenario. India is a populous country with diverse cultures and languages. The majority of the Indian population needs to use their respective native languages for verbal interaction with machines. Therefore, the development of efficient Indian spoken language recognition systems is useful for adapting smart technologies in every section of Indian society. The field of Indian LID has started gaining momentum since the early 2000s, mainly due to the development of several standard multilingual speech corpora for the Indian languages. Even though significant research progress has already been made in this field, to the best of our knowledge, there are not many attempts to analytically review them collectively. In this work, we have conducted one of the very first attempts to present a comprehensive review of the Indian spoken language recognition research field. In-depth analysis has been presented to emphasize the unique challenges of low-resource and mutual influences for developing LID systems in the Indian contexts. Several essential aspects of the Indian LID research, such as the detailed description of the available speech corpora, the major research contributions, including the earlier attempts based on statistical modeling to the recent approaches based on different neural network architectures, and the future research trends are discussed. This review work will help assess the state of the present Indian LID research by any active researcher or any research enthusiasts from related fields.

Predicting the type and target of offensive social media posts in Marathi

The presence of offensive language on social media is very common motivating platforms to invest in strategies to make communities safer. This includes developing robust machine learning systems capable of recognizing offensive content online. Apart from a few notable exceptions, most research on automatic offensive language identification has dealt with English and a few other high-resource languages such as French, German, and Spanish. In this paper, we address this gap by tackling offensive language identification in Marathi, a low-resource Indo-Aryan language spoken in India. We introduce the Marathi Offensive Language Dataset v.2.0 or MOLD 2.0 and present multiple experiments on this dataset. MOLD 2.0 is a much larger version of MOLD with expanded annotation to the levels B (type) and C (target) of the popular OLID taxonomy. MOLD 2.0 is the first hierarchical offensive language dataset compiled for Marathi, thus opening new avenues for research in low-resource Indo-Aryan languages. Finally, we also introduce SeMOLD, a larger dataset annotated following the semi-supervised methods presented in SOLID (Rosenthal et al. in SOLID: a large-scale semi-supervised dataset for offensive language identification. In: Findings of ACL, 2021).

Automatic spoken language identification using MFCC based time series features

Automatic spoken language identification using MFCC based time series features

Automatic Database Segmentation using Hybrid Spectrum -Visual Approach

Nowadays automated segmentation of speech signals has been attracted many of researchers all-over the world, Many speech processing systems require segmentation of speech waveform into principal acoustic units. In this research, TIMIT DataBase (DB) is utilized to carry on this process and justify its operation or results. Thus, this paper presents a novel method of segmentation of speech phonemes, where the proposed strategy helps in the selection of appropriate feature extraction technique for speech segmentation. There are three main techniques of feature extraction used in our research; the first technique is the Mel Frequency Cepstral Coefficient (MFCC), the second technique is known by Best Tree Encoding (BTE), while the third is Image Normalized Encoder (INE), which is a hybrid technique between the Best Tree Image (BTI), and the Convolution Neural Network (CNN) ResNet-50. Then, data are trained using a hybrid model that consists of Hidden Markov Model (HMM), and Gaussian Mixture Model (GMM) to improve the performance of automatic speech recognition. The proposed model is tested and verified against the most widely used feature Mel Frequency Cepstral Coefficient (MFCC) plus delta and delta-delta coefficients (39 parameters) to evaluate its performance. This approach has the potential to be used in applications such as automatic speech recognition and automatic language identification. The experimental results show that BTE technique achieved the highest success rate (𝜂) (92.64%) than using the (INE) technique. However, the INE technique gives confusion success rate for Tr and NTr of values 97.1% and 99.1%, respectively.

A CNN-BiLSTM based hybrid model for Indian language identification

Automatic language identification (LID) is the practice of detecting the language through linguistic content of speech spoken by certain anonymous speaker. In multi-lingual based Indian society, the capability to identify and classify a spoken language is an imperative task. In this work, convolutional neural network (CNN) based bidirectional long short-term memory (BiLSTM) model has been proposed for Indian language identification with special emphasis on Northeastern languages. Initially, speech samples are transformed into spectrogram images, which comprises the multi-frequency band signals with respect to time. Deep linguistic features for the said task have been extracted from the spectrogram images using CNNs. Furthermore, BiLSTM network is used to extract more temporal information for language identification. On top of it, attention mechanism has been used to find the context for each frame. At the end, softmax function is applied to calculate language score for each language. The simulation is conducted on recorded database consists of four Indian languages. The languages are Assamese, Bengali, Indian English, and Hindi respectively. The performance of proposed LID model is tested on two different CNN architectures ResNet-50 and VGG-16. Simulation results show that the ResNet-50 based model has achieved accuracy up to 98.10% as compare to 97.70% for VGG-16 based model.

A unified system for multilingual speech recognition and language identification

In this paper, a multilingual automatic speech recognition (ASR) and language identification (LID) system is designed. In contrast to conventional multilingual ASR systems, this paper takes advantage of the complementarity of the ASR and LID modules. First, the LID module contributes to the language adaptive training of the multilingual acoustic model. Then, the ASR decoding information acts as the confidence metric to balance the LID results. To simulate complex multilingual speech recognition situations, two types of LID strategies are conducted. For a multilingual speech recognition task in which only one language is contained in the speech stream, the language information can be directly determined based on utterance-level judgment. Under this condition, a segment-level statistical component and a two-stage update strategy are designed to assist in the utterance-level language classification. In another multilingual speech recognition task, where the speech stream contains multiple languages simultaneously, the Viterbi language state retrieval method based on neural network (NN) classification is used to perform dynamic detection of the language state. In both cases, the ASR decoding information is used to adjust the language classification results. Without prior knowledge of language identity information, the enhanced LID module achieves an accuracy of 99.3% for utterance-level language judgment and 92.4% for dynamic language detection, and the multilingual ASR system also provides performance comparable to that of monolingual ASR systems.

FuzzyGCP: A deep learning architecture for automatic spoken language identification from speech signals

In this modern era, language has no geographic boundary. Therefore, for developing an automated system for search engines using audio, tele-medicine, emergency service via phone etc., the first and foremost requirement is to identify the language. The fundamental difficulty of automatic speech recognition is that the speech signals vary significantly due to different speakers, speech variation, language variation, age and sex wise voice modulation variation, contents and acoustic conditions and so on. In this paper, we have proposed a deep learning based ensemble architecture, called FuzzyGCP, for spoken language identification from speech signals. This architecture combines the classification principles of a Deep Dumb Multi Layer Perceptron (DDMLP), Deep Convolutional Neural Network (DCNN) and Semi-supervised Generative Adversarial Network (SSGAN) to increase the precision to maximum and finally applies Ensemble learning using Choquet integral to predict the final output, i.e., the language class. We have evaluated our model on four standard benchmark datasets comprising of two Indic language datasets and two foreign language datasets. Irrespective of the languages, the F1-score of the proposed language identification model is as high as 98% in MaSS dataset and worst performance is that of 67% on the VoxForge dataset which is much better compared to maximum of 44% by state-of-the-art models on multi-class classification. The link to the source code of our model is available here.

Mapping the languages of Twitter in Finland:

Twitter is a popular social media platform for scholarly research, because the user-generated content on the platform can also include geographic and temporal information. We collect a corpus of 38 million Twitter messages with two million geographical coordinates to map the languages used across Finland at the level of regions and municipalities. To cope with the high volume of social media data, we use automatic language identification and place of residence detection. We estimate the linguistic richness and diversity of users and locations using measures developed within ecology and information sciences. The analyses reveal a rich, multilingual environment that varies geographically and temporally, particularly between coastal, rural and urban areas. The results, which underline the mutual benefits of collaboration between linguists and geographers, provide a more fine-grained, accurate and comprehensive view of the languages used on Twitter in Finland than previously available.

Towards a scientific workflow featuring Natural Language Processing for the digitisation of natural history collections

We describe an effective approach to automated text digitisation with respect to natural history specimen labels. These labels contain much useful data about the specimen including its collector, country of origin, and collection date. Our approach to automatically extracting these data takes the form of a pipeline. Recommendations are made for the pipeline's component parts based on state-of-the-art technologies.Optical Character Recognition (OCR) can be used to digitise text on images of specimens. However, recognising text quickly and accurately from these images can be a challenge for OCR. We show that OCR performance can be improved by prior segmentation of specimen images into their component parts. This ensures that only text-bearing labels are submitted for OCR processing as opposed to whole specimen images, which inevitably contain non-textual information that may lead to false positive readings. In our testing Tesseract OCR version 4.0.0 offers promising text recognition accuracy with segmented images.Not all the text on specimen labels is printed. Handwritten text varies much more and does not conform to standard shapes and sizes of individual characters, which poses an additional challenge for OCR. Recently, deep learning has allowed for significant advances in this area. Google's Cloud Vision, which is based on deep learning, is trained on large-scale datasets, and is shown to be quite adept at this task. This may take us some way towards negating the need for humans to routinely transcribe handwritten text.Determining the countries and collectors of specimens has been the goal of previous automated text digitisation research activities. Our approach also focuses on these two pieces of information. An area of Natural Language Processing (NLP) known as Named Entity Recognition (NER) has matured enough to semi-automate this task. Our experiments demonstrated that existing approaches can accurately recognise location and person names within the text extracted from segmented images via Tesseract version 4.0.0.We have highlighted the main recommendations for potential pipeline components. The paper also provides guidance on selecting appropriate software solutions. These include automatic language identification, terminology extraction, and integrating all pipeline components into a scientific workflow to automate the overall digitisation process.

Towards a scientific workflow featuring Natural Language Processing for the digitisation of natural history collections

We describe an effective approach to automated text digitisation with respect to natural history specimen labels. These labels contain much useful data about the specimen including its collector, country of origin, and collection date. Our approach to automatically extracting these data takes the form of a pipeline. Recommendations are made for the pipeline's component parts based on some of the state-of-the-art technologies.Optical Character Recognition (OCR) can be used to digitise text on images of specimens. However, recognising text quickly and accurately from these images can be a challenge for OCR. We show that OCR performance can be improved by prior segmentation of specimen images into their component parts. This ensures that only text-bearing labels are submitted for OCR processing as opposed to whole specimen images, which inevitably contain non-textual information that may lead to false positive readings. In our testing Tesseract OCR version 4.0.0 offers promising text recognition accuracy with segmented images.Not all the text on specimen labels is printed. Handwritten text varies much more and does not conform to standard shapes and sizes of individual characters, which poses an additional challenge for OCR. Recently, deep learning has allowed for significant advances in this area. Google's Cloud Vision, which is based on deep learning, is trained on large-scale datasets, and is shown to be quite adept at this task. This may take us some way towards negating the need for humans to routinely transcribe handwritten text.Determining the countries and collectors of specimens has been the goal of previous automated text digitisation research activities. Our approach also focuses on these two pieces of information. An area of Natural Language Processing (NLP) known as Named Entity Recognition (NER) has matured enough to semi-automate this task. Our experiments demonstrated that existing approaches can accurately recognise location and person names within the text extracted from segmented images via Tesseract version 4.0.0. Potentially, NER could be used in conjunction with other online services, such as those of the Biodiversity Heritage Library to map the named entities to entities in the biodiversity literature (https://www.biodiversitylibrary.org/docs/api3.html).We have highlighted the main recommendations for potential pipeline components. The document also provides guidance on selecting appropriate software solutions. These include automatic language identification, terminology extraction, and integrating all pipeline components into a scientific workflow to automate the overall digitisation process.

A comparison between phonetic engine and GMM–UBM classifier for language identification tasks

In this paper, two automatic language identification (LID) systems are compared. One of the systems is the Hidden Markov Model (HMM) based phonetic engine (PE), and the other is the Gaussian Mixture Model based Universal Background Model (GMM–UBM) classifier. The PE belongs to the category of explicit LID systems while the GMM–UBM classifier falls into the category of implicit LID systems. Ideally, explicit LID requires a segmented and phonetically labelled speech corpus, while the implicit LID systems do not require any phonetic labelling of the data. Both systems are tested here in identifying a set of data belonging to three Indian languages, Manipuri, Assamese and Bengali. The selection of these languages is made due to their wide range of usages in North Eastern India, while at the same time; no proper identification task has been reported so far for a database containing these languages together. The purpose of this comparison is to check the LID efficiency of the relatively new concept of PE with a prevalent identification technique GMM–UBM. In the experiments, it is found that the identification rate (IDR) is more with the PE than that of GMM–UBM system. The average IDR reported with PE is 99% while for the GMM–UBM system it is found to be 96.94% with the same speech corpus being in use. However, the data preparation task is a little more cumbersome and expensive in PE than that of GMM–UBM system. Thus, the compensation for accuracy may be paid with the cost incurred in data preparation.

Broad Phonetic Classification of ASR using Visual Based Features

Abstract: This paper presents a novel method of classifying speech phonemes. Four hybrid techniques based on the acoustic-phonetic approach and pattern recognition approach are used to emphasize the principle idea of this research. The first hybrid model is constructed of fixed state, structured Hidden Markov Model, Gaussian Mixture, Mel scaled Best Tree Image, Convolution Neural network, Vector Quantization (FS-HMM-GM-MBTI-CNN-VQ). The second hybrid model is constructed of variable state, dynamically structured Hidden Markov Model, Gaussian Mixture, Mel scaled Best Tree Image, Convolution Neural network, Vector Quantization (VS-HMM-GM-MBTI-CNN-VQ). The third hybrid model is constructed of fixed state, structured Hidden Markov Model, Gaussian Mixture, Mel scaled Best Tree Image, Convolution Neural network (FS-HMM-GM-MBTI-CNN). The fourth hybrid model is constructed of variable state, dynamically structured Hidden Markov Model, Gaussian Mixture, Mel scaled Best Tree Image, Convolution Neural network (VS-HMM-GM-MBTI-CNN). TIMIT database is used in this paper. All phones are classified into five classes and segregated into Vowels, Plosives, Fricatives, Nasals, and Silences. The results show that using (VS-HMM-GM-MBTI-CNN-VQ) is an available method for classification of phonemes, with the potential for use in applications such as automatic speech recognition and automatic language identification. Competitive results are achieved especially in nasals, plosives, and silence high successive rates than others.

Bottleneck Feature-Based Hybrid Deep Autoencoder Approach for Indian Language Identification

Latest and emerging approaches are essential to resolve the communication barrier among different languages in speech processing. The automatic language identification system is developed to identify the spoken language from speech utterances. Feature selection is a very challenging task in language identification. In this paper, bottleneck feature-based hybrid deep autoencoder approach is proposed to identify the given speech signal with corresponding language features. In the proposed approach, initially Mel-frequency cepstral coefficients, linear prediction coefficients, and shifted delta coefficients features are directly extracted from multilingual speech utterances. Further, we extracted bottleneck feature from the bottleneck layer of the bottleneck deep neural network. Initially, recognition rate has been evaluated for each feature set to find out the best feature. Finally, the best feature along with other features is used as the input for deep autoencoder with softmax regression to identify the language based on class labels. The deep autoencoder is fine-tuned to reach the global optimum through Jaya optimization algorithm. To carry out the experiments, the recorded database is used for four Indian languages with special emphasis on northeastern languages. The experimental results demonstrate that the proposed hybrid approach using bottleneck feature with shifted delta coefficients is performing well with 97.10% accuracy. Moreover, experimental results also show that proposed hybrid approach gives superior outcome when compared with the baseline deep neural network-based approaches.

Evaluating the Effect of Multiple Filters in Automatic Language Identification without Lexical Knowledge

The classical language identification architecture would require a collection of languages independent text and speech information for training by the system before it can identify the languages correctly. This paper also address language identification framework however with data has been downsized considerably from general language identification architecture. The system goal is to identify the type of language being spoken to the system based on a series of trained speech with sound file features and without any language text data or lexical knowledge of the spoken language. The system is also expected to be able to be deployed in mobile platform in future. This paper is specifically about measuring the performance optimisation of audio filters on a CNN model integration for the language identification system. There are several metric to gauge the performance identification system for a classification problem. Precision, recall and F1 Scores is presented for the performance evaluation with different combination of filters together with CNN model as the framework of the language identification system. The goal is not to get the best filter for noise, instead to identify the filter that is a good fit to develop language model with environmental noise for a robust language identification system. Our experiments manage to identify the best combination of filters to increase the accuracy of language identification using short speech. This resulting us to modify our pre-processing phase in the overall language identification system.

Towards Relevance and Sequence Modeling in Language Recognition

The task of automatic language identification (LID) involving multiple dialects of the same language family in the presence of noise is a challenging problem. In these scenarios, the identity of the language/dialect may be reliably present only in parts of the temporal sequence of the speech signal. The conventional approaches to LID (and for speaker recognition) ignore the sequence information by extracting long-term statistical summary of the recording assuming an independence of the feature frames. In this paper, we propose a neural network framework utilizing short-sequence information in language recognition. In particular, a new model is proposed for incorporating relevance in language recognition, where parts of speech data are weighted more based on their relevance for the language recognition task. This relevance weighting is achieved using the bidirectional long short-term memory (BLSTM) network with attention modeling. We explore two approaches, the first approach uses segment level i-vector/x-vector representations that are aggregated in the neural model and the second approach where the acoustic features are directly modeled in an end-to-end neural model. Experiments are performed using the language recognition task in NIST LRE 2017 Challenge using clean, noisy and multi-speaker speech data as well as in the RATS language recognition corpus. In these experiments on noisy LRE tasks as well as the RATS dataset, the proposed approach yields significant improvements over the conventional i-vector/x-vector based language recognition approaches as well as with other previous models incorporating sequence information.

Native Language Identification from Spoken Indian English

Automatic speech recognition (ASR) systems that facilitate voice based search and information retrieval have gained importance recently. While the performance of ASR systems for Indian languages have improved in the recent past. They have yet to gain wide acceptability as much as the ASR systems for English spoken by Indians. Almost all Indians learn English as a second or third language. So, the phoneme set and the prosody of native language of Indians influences the acoustic characteristics of spoken English. Since Indians speak a wide variety of languages, the acoustic characteristics of English spoken by Indians vary a lot. Thus, the recognition accuracy of Indian English could be improved by employing native language dependent English ASR systems. This approach requires automatic identification of the native language of the speaker. Here, we report the performance of an automatic Native Language Identification (NLI) system that recognises the native language of the speaker as Assamese or Bengali or Bodo after analysis of an English sentence spoken by the speaker. Training and performance evaluation of a NLI system needs appropriate linguistic resources. These include (a) speech data, in each of the 3 languages from several speakers, (b) corresponding word level transcriptions and (c) a pronunciation dictionary. While pronunciation dictionaries for English language are freely available, spoken English by speakers of the above-mentioned three languages and transcriptions are not publicly available. So, we created a relevant speech database. We recorded English spoken by native speakers, both male and female, of these three scheduled languages. Each speaker read 100 sentences out of a set of 700 English sentences; these were either proverbs or digit sequences. Each sentence contained 5 to 8 words. The digitised speech, recorded under ambient conditions using a laptop, had the following characteristics: 16000 Hz, 16 bit, mono. The database contains spoken English from 35 native Assamese speakers, 33 Bengali and 30 Bodo speakers. In order to carry out a threefold evaluation of the performance of the system, the speakers from each language were grouped into 3 subsets such that each subset contains nearly equal number of speakers. In each fold, one subset was designated as test data, and the remaining two subsets were used to train the system. We used Kaldi, an open source ASR toolkit, for implementation of the NLI system. As a first step in the development of NLI system, we implemented three English ASR systems, each trained using training data from one of the three languages: Assamese, Bengali and Bodo. A three-state Hidden Markov Model (HMM) represented a phone. Each state of HMM was associated with a Gaussian mixture model. We used Mel frequency cepstral coefficients and their temporal derivatives as features, and bigram as the language model. In order to identify the native language of a speaker, the test speech file was fed to each of the three ASR systems. An ASR system not only generates the decoded word sequence, but also the corresponding log likelihood. The NLI system follows the maximum likelihood criterion. The language corresponding to the ASR system that yielded the highest likelihood for the test speech was declared as the native language of the speaker. The overall accuracy of the NLI system was computed as the unweighted average recall, computed from the confusion matrix. The NLI accuracy of the system, averaged over threefold cross evaluations, was 59% for test speech of just 3 seconds. The confusion was largest among Assamese and Bengali languages as both are close members of Indo-Aryan language family. In contrast, Bodo belongs to the Sino-Tibetan language family. We discuss the performance of the NLI system using different models such as context-dependent and context independent HMMs, employing Gaussian mixture model or deep neural network to estimate the likelihood of a feature vector emitted from a state of HMM. Keywords: Automatic identification, automatic speech recognition, native language identification, voice-based search, information retrieval