Computer Assisted Pronunciation Training Research Articles

After Self-Imitation Prosodic Training L2 Learners Converge Prosodically to the Native Speakers

Little attention is paid to prosody in second language (L2) instruction, but computer-assisted pronunciation training (CAPT) offers learners solutions to improve the perception and production of L2 suprasegmentals. In this study, we extend with acoustic analysis a previous research showing the effectiveness of self-imitation training on prosodic improvements of Japanese learners of Italian. In light of the increased degree of correct match between intended and perceived pragmatic functions (e.g., speech acts), in this study, we aimed at quantifying the degree of prosodic convergence towards L1 Italian speakers used as a model for self-imitation training. To measure convergence, we calculated the difference in duration, F0 mean, and F0 max syllable-wise between L1 utterances and the corresponding L2 utterances produced before and after training. The results showed that after self-imitation training, L2 learners converged to the L1 speakers. The extent of the effect, however, varied based on the speech act, the acoustic measure, and the distance between L1 and L2 speakers before the training. The findings from perceptual and acoustic investigations, taken together, show the potential of self-imitation prosodic training as a valuable tool to help L2 learners communicate more effectively.

An Open CAPT System for Prosody Practice: Practical Steps towards Multilingual Setup

This paper discusses the challenges posed in creating a Computer-Assisted Pronunciation Training (CAPT) environment for multiple languages. By selecting one language from each of three different language families, we show that a single environment may be tailored to cater for different target languages. We detail the challenges faced during the development of a multimodal CAPT environment comprising a toolkit that manages mobile applications using speech signal processing, visualization, and estimation algorithms. Since the applied underlying mathematical and phonological models, as well as the feedback production algorithms, are based on sound signal processing and modeling rather than on particular languages, the system is language-agnostic and serves as an open toolkit for developing phrasal intonation training exercises for an open selection of languages. However, it was necessary to tailor the CAPT environment to the language-specific particularities in the multilingual setups, especially the additional requirements for adequate and consistent speech evaluation and feedback production. In our work, we describe our response to the challenges in visualizing and segmenting recorded pitch signals and modeling the language melody and rhythm necessary for such a multilingual adaptation, particularly for tonal syllable-timed and mora-timed languages.

The pedagogy for teaching suprasegmental features

Abstract Suprasegmental (prosody) deficiencies seriously affect English as a foreign language (EFL) learners’ comprehensibility and intelligibility. Among suprasegmental features, the most problematic for Iranian learners is rhythm, due to fundamental differences between English and Farsi language timing patterns. To improve learners’ pronunciation, computer-assisted pronunciation training (CAPT) is commonly applied to provide a private, individualized learning environment with unlimited input and visual, instant, and precise feedback. In our previous study, a specific kind of CAPT application, (MyET™) was used to enhance suprasegmental features of five Iranian EFL learners. The results showed that CAPT helped them improve their pronunciation skills. They also had positive attitudes toward taking this approach. In this short paper, we describe how prosody teaching actually occurred in the classroom, how we used the material, and how we reported the results. Suggestions and recommendations for further research are also presented.

A Bi-directional Attention Based End to End Mispronunciation Detection and Diagnosis of Mandarin

An increasing number of individuals are acquiring proficiency in Mandarin, signifying the growing significance of employing computer-assisted pronunciation training systems for Mandarin learners. One pivotal component within these systems is the technique for identifying and addressing mispronunciations, known as Mispronunciation Detection and Diagnosis (MDD). Recently, certain end-to-end techniques have tried to fuse features of prompt text and acoustic features into the model for training and have shown good results. However, previous approaches have fused acoustic features with prompt text features by a simple attention mechanism. In this paper, we posit that the impact of text features varies significantly when mapped to distinct acoustic characteristics. Furthermore, we propose that the prompt text can lead the model towards achieving an integrated text-audio representation, thereby enhancing the inference quality. Hence, this article presents a model aimed at detecting and diagnosing mispronunciations. The model utilizes a bidirectional attention mechanism to integrate acoustic and prompt text features. Good results were achieved by conducting experiments on a self-built dataset of short Mandarin read-aloud texts.

The Effect of Computer-Assisted Pronunciation Training on EFL Learners’ Use of Suprasegmental Features and Foreign Language Speaking Anxiety

The Effect of Computer-Assisted Pronunciation Training on EFL Learners’ Use of Suprasegmental Features and Foreign Language Speaking Anxiety

INTEGRATING HTML5-BASED SPEECH RECOGNITION WITH LEARNING MANAGEMENT SYSTEM TO ENHANCE EFL LEARNERS’ PRONUNCIATION SKILLS

HTML5 (HyperText Markup Language version 5) is currently integrated with almost all web-based applications due to its compatibility with numerous types of multimedia content including music, video, and animation. This exploratory study focused on the application and assessment of an HTML5-based plugin called H5P as a Computer Assisted Pronunciation Training (CAPT) tool in developing the pronunciation skills of intermediate-level learners of English as a foreign language (EFL). The target words were initially recorded with their correct pronunciations and then the audio files were imported into H5P to create interactive exercises. The H5P was then integrated with the institutional Learning Management System (LMS). The practice was implemented on n=188 EFL learners enrolled in the General Foundation (GFP) of a university college in the Sultanate of Oman. The efficacy of the approach was assessed through the perspectives of students by using student questionnaires and a student focus group discussion (FGD). The study yielded encouraging findings since most of the students agreed that the use of HTML5 as a CAPT tool was helpful in enhancing their pronunciation skills as it provided automatically generated instant feedback, was easy to use, and could be used by students autonomously at their own pace. Although the impact of software on student achievement in terms of pronunciation skills could not be gauged and the sample was limited to one institution, the study has useful implications for EFL teachers and curriculum designers on using HTML5 for developing pronunciation skills. Further studies are necessary to investigate the pedagogic potential of CAPT resources to develop a high level of competence in EFL learners’ pronunciation.

A RTL Implementation of Heterogeneous Machine Learning Network for French Computer Assisted Pronunciation Training

Computer-assisted pronunciation training (CAPT) is a helpful method for self-directed or long-distance foreign language learning. It greatly benefits from the progress, and of acoustic signal processing and artificial intelligence techniques. However, in real-life applications, embedded solutions are usually desired. This paper conceives a register-transfer level (RTL) core to facilitate the pronunciation diagnostic tasks by suppressing the mulitcollinearity of the speech waveforms. A recently proposed heterogeneous machine learning framework is selected as the French phoneme pronunciation diagnostic algorithm. This RTL core is implemented and optimized within a very-high-level synthesis method for fast prototyping. An original French phoneme data set containing 4830 samples is used for the evaluation experiments. The experiment results demonstrate that the proposed implementation reduces the diagnostic error rate by 0.79–1.33% compared to the state-of-the-art and achieves a speedup of 10.89× relative to its CPU implementation at the same abstract level of programming languages.

Computer-Assisted Pronunciation Teaching: An Analysis of Empirical Research

Computer-assisted pronunciation teaching has been shown to have favourable learning outcomes for language learners, which brought about the emergence of increasing body of research on this topic. Despite some attempted review studies, no systematic review of the line of the relevant literature has hitherto been observed. This study seeks to fill in this gap by reviewing the research on computer-assisted pronunciation teaching in L2 classes published between 2010 and 2021 May. 26 studies were scrutinized in terms of their contexts and participants, technology used, pronunciation aspect analysed, research design and focus, data collection tools and theoretical framework. The results showed that most of the studies were conducted with EFL/ESL learners at university level. Computer-assisted pronunciation training (CAPT) and Automatic Speech Recognition (ASR) programs were utilized more than other technological tools. Quantitative design was the dominant research method, which was followed by mixed method design while few studies were grounded in a theoretical framework. Segmental features of pronunciation received more attention. Pronunciation achievement, learner perceptions, relationship between some variables related to pronunciation learning and student interaction/ participation were the main topics of investigation. The findings suggested that teachers had a key role in promoting the effectiveness of computer-assisted pronunciation teaching by scaffolding, motivating, and guiding the learners as they use CAPT or technological tools. To this end teacher training that supports teachers in developing these competencies can be beneficial. Further research can focus on languages other than English, be conducted in primary and secondary school level and have longitudinal designs.

Factors predicting human performance in error annotation for non-native speech corpus

With low inter-annotator agreement and large time consumption, phonetic annotation of non-native speech is complicated. However, having a large-scale non-native speech corpus with error annotation is important for second language phonology and computer-assisted pronunciation training research. To achieve large-scale annotation, we must clarify the causes of difficulty in error annotation. This study identified the factors that can predict annotation performance of non-native Mandarin speech by considering two perspectives.1) We used a decomposed method (that involved dividing a traditional error annotation task into two different subtasks, namely the error location task and error description task) to determine whether human performance, from the cognitive viewpoint, can be predicted using the decomposed annotation method (Experiment 1). 2) We determined whether the features of annotation task materials (e.g., error severity) can be employed to predict annotation performance (Experiment 2). The results of Experiment 1 revealed that the decomposed method leads to a shorter annotation time and a higher consistent hit rate than does the traditional method. The regressions of Experiment 2 revealed that the features of error severity and sentence length accounted for 96% of the hit rate and the features of sentence length and number of errors accounted for 95.5% of the annotation time. These results suggest that the performance of manual annotation can be predicted through both cognitive factors and material features.

SELECTION OF DISCRIMINATIVE FEATURES FOR ARABIC PHONEME’S MISPRONUNCIATION DETECTION

Pronunciation training is an important part of Computer Assisted Pronunciation Training (CAPT) systems. Mispronunciation detection systems recognized pronunciation mistakes from user’s speech and provided them feedback about their pronunciation. Acoustic phonetic features plays a vital role in speech classification based applications. This research work investigated the suitability of various acoustic features: pitch, energy, spectrum flux, zero-crossing, Entropy and MelFrequency Cepstral Coefficients (MFCCs). Sequential Forward Selection (SFS) was used to find out most suitable acoustic features from the computed feature set. This study used K-Nearest Neighbors (K-NN) classifier was used to detect the pronunciation mistakes from Arabic phonemes. This research selected the set of most discriminative acoustic features for each phoneme. K-NN achieved accuracy of 92.15% for mispronunciation detection of Arabic Phonemes.

SELECTION OF DISCRIMINATIVE FEATURES FOR ARABIC PHONEME’S MISPRONUNCIATION DETECTION

Pronunciation training is an important part of Computer Assisted Pronunciation Training (CAPT) systems. Mispronunciation detection systems recognized pronunciation mistakes from user’s speech and provided them feedback about their pronunciation. Acoustic phonetic features plays a vital role in speech classification based applications. This research work investigated the suitability of various acoustic features: pitch, energy, spectrum flux, zero-crossing, Entropy and MelFrequency Cepstral Coefficients (MFCCs). Sequential Forward Selection (SFS) was used to find out most suitable acoustic features from the computed feature set. This study used K-Nearest Neighbors (K-NN) classifier was used to detect the pronunciation mistakes from Arabic phonemes. This research selected the set of most discriminative acoustic features for each phoneme. K-NN achieved accuracy of 92.15% for mispronunciation detection of Arabic Phonemes

Synchronous computer-mediated communication in pronunciation teaching: A survey study

Remote learning has been in the spotlight since the emergence of the coronavirus pandemic, notably boosting the development of distance learning in the last two years; as universities were obliged to close worldwide and lecturers had to teach online during lockdown, the use of Synchronous Computer-Mediated Communication (SCMC) started to prevail. Pronunciation instruction was not an exception. Indeed, in-person lectures had used technology to support face-to-face (F2F) education by using videos and articles in class. However, educators had to adapt rapidly to teach through video-based platforms. Although much investigation has been conducted on Computer-Assisted Pronunciation Training (CAPT), and researchers currently show more and more interest in the use of new technologies within the pronunciation classroom, little research has been done regarding the teaching methodologies applied in teacher-guided online pronunciation instruction.

Use of ASR-Equipped Software in the Teaching of Suprasegmental Features of Pronunciation

Technology has paved the way for new modalities in language learning, teaching, and assessment. However, there is still a great deal of work to be done to develop such tools for oral communication, specifically tools that address suprasegmental features in pronunciation instruction. Therefore, this critical literature review examines how researchers have tried to create computer-assisted pronunciation training tools using automatic speech recognition (ASR) systems to aid language learners in the perception and production of suprasegmental features. We used 30 texts from 1990 to 2020 to explore how technologies have been and are currently being used to help learners develop their proficiency with suprasegmental features. Based on our thematic analysis, a persistent gap still exists between ASR-equipped software available to participants in research studies and what is available to university and classroom teachers and students. Additionally, there seems to be more development in the production of speech software for language assessment. In contrast, the translation of these tools into instructional tools for individualized learning seems to be almost non-existent. Moving forward, we recommend that more commercialized pronunciation systems utilizing ASR should be made publicly available using the technologies that are currently developed or are in development for the purposes of oral proficiency judgments.

Mispronunciation Detection and Diagnosis with Articulatory-Level Feedback Generation for Non-Native Arabic Speech

A high-performance versatile computer-assisted pronunciation training (CAPT) system that provides the learner immediate feedback as to whether their pronunciation is correct is very helpful in learning correct pronunciation and allows learners to practice this at any time and with unlimited repetitions, without the presence of an instructor. In this paper, we propose deep learning-based techniques to build a high-performance versatile CAPT system for mispronunciation detection and diagnosis (MDD) and articulatory feedback generation for non-native Arabic learners. The proposed system can locate the error in pronunciation, recognize the mispronounced phonemes, and detect the corresponding articulatory features (AFs), not only in words but even in sentences. We formulate the recognition of phonemes and corresponding AFs as a multi-label object recognition problem, where the objects are the phonemes and their AFs in a spectral image. Moreover, we investigate the use of cutting-edge neural text-to-speech (TTS) technology to generate a new corpus of high-quality speech from predefined text that has the most common substitution errors among Arabic learners. The proposed model and its various enhanced versions achieved excellent results. We compared the performance of the different proposed models with the state-of-the-art end-to-end technique of MDD, and our system had a better performance. In addition, we proposed using fusion between the proposed model and the end-to-end model and obtained a better performance. Our best model achieved a 3.83% phoneme error rate (PER) in the phoneme recognition task, a 70.53% F1-score in the MDD task, and a detection error rate (DER) of 2.6% for the AF detection task.

An Acoustic Feature-Based Deep Learning Model for Automatic Thai Vowel Pronunciation Recognition

For Thai vowel pronunciation, it is very important to know that when mispronunciation occurs, the meanings of words change completely. Thus, effective and standardized practice is essential to pronouncing words correctly as a native speaker. Since the COVID-19 pandemic, online learning has become increasingly popular. For example, an online pronunciation application system was introduced that has virtual teachers and an intelligent process of evaluating students that is similar to standardized training by a teacher in a real classroom. This research presents an online automatic computer-assisted pronunciation training (CAPT) using deep learning to recognize Thai vowels in speech. The automatic CAPT is developed to solve the inadequacy of instruction specialists and the complex vowel teaching process. It is a unique system that develops computer techniques integrated with linguistic theory. The deep learning model is the most significant part of recognizing vowels pronounced for the automatic CAPT. The major challenge in Thai vowel recognition is the correct identification of Thai vowels when spoken in real-world situations. A convolutional neural network (CNN), a deep learning model, is applied and developed in the classification of pronounced Thai vowels. A new dataset for Thai vowels was designed, collected, and examined by linguists. The result of an optimal CNN model with Mel spectrogram (MS) achieves the highest accuracy of 98.61%, compared with Mel frequency cepstral coefficients (MFCC) with the baseline long short-term memory (LSTM) model and MS with the baseline LSTM model have an accuracy of 94.44% and 90.00% respectively.

Computer-assisted pronunciation training—Speech synthesis is almost all you need

The research community has long studied computer-assisted pronunciation training (CAPT) methods in non-native speech. Researchers focused on studying various model architectures, such as Bayesian networks and deep learning methods, as well as on the analysis of different representations of the speech signal. Despite significant progress in recent years, existing CAPT methods are not able to detect pronunciation errors with high accuracy (only 60% precision at 40%–80% recall). One of the key problems is the low availability of mispronounced speech that is needed for the reliable training of pronunciation error detection models. If we had a generative model that could mimic non-native speech and produce any amount of training data, then the task of detecting pronunciation errors would be much easier. We present three innovative techniques based on phoneme-to-phoneme (P2P), text-to-speech (T2S) and speech-to-speech (S2S) conversion to generate correctly pronounced and mispronounced synthetic speech. We show that these techniques not only improve the accuracy of three machine learning models for detecting pronunciation errors, but also help establish a new state-of-the-art in the field. Earlier studies have used simple speech generation techniques such as P2P conversion, but only as an additional mechanism to improve the accuracy of pronunciation error detection. We, on the other hand, consider speech generation to be the first-class method of detecting pronunciation errors. The effectiveness of these techniques is assessed in the tasks of detecting pronunciation and lexical stress errors. Non-native English speech corpora of German, Italian, and Polish speakers are used in the evaluations. The best proposed S2S technique improves the accuracy of detecting pronunciation errors in AUC metric by 41% from 0.528 to 0.749 compared to the state-of-the-art approach.

Speakometer: English Pronunciation Coach

Consonant and vowel sounds of English (segmentals) carry a significant weight in communication. Pronunciation instruction focusing on segmental features has been found to be highly effective (e.g., Thomson & Derwing, 2015). However, students with different first languages (L1) or even students from the same L1 backgrounds, have different pronunciation needs. With limited class time, teachers cannot be expected to cater to the pronunciation needs of every student. This has made individualized pronunciation instruction, which enables pronunciation instruction tailored for the needs of each second language (L2) learner, a requirement in today’s language classrooms (Levis, 2007). The growing number of computer-assisted pronunciation training (CAPT) tools have been responding to this need, making individualized pronunciation instruction, as well as individualized feedback more feasible and available for L2 speakers. Chun (2012) asserts that in order for a CAPT tool to be effective, it must contain “auditory and visualization features, automatic speech recognition (ASR), and appropriate and accurate feedback” (p. 8). Speakometer, an online application that provides segmental practice for its users, was built around Chun’s (2012) three pillars, with a strong auditory feature combined with an ASR to provide learners with relevant pronunciation feedback. The application uses an artificial intelligence (AI) algorithm and ASR to rate the user’s spoken English pronunciation. It is targeted for all users who aim to improve their English pronunciation. The users are provided with immediate feedback, which appears on the screen as verbal (e.g., “Very good”), along with the image of a ‘speakometer’ displaying four colors for the rating: red, orange, yellow and green.

Automatic identification of synthetically generated interlanguage transfer phenomena between brazilian portuguese (L1) and english (L2)

Transfer phenomena between Portuguese (L1) and English (L2) produced by Brazilian learners are well documented in the literature. However, the identification and classification of these processes are made mainly through transcriptions, a slow and laborious process done by specialized linguists. The rapid identification of these phenomena would be of great value for software doing proficiency placement tests and could be used in language schools, distance education, computer-assisted pronunciation training (CAPT) or by autodidacts and researchers. The present work analyzed possible techniques and tools that can be used in the automatic identification of some transfer processes. The data for the grapho-phonic-phonological transfer were synthetically generated in the Google Translate™ TTS system. Then we tested three classification algorithms to perform the identification: k-Nearest Neighbor, Centroid Minimum Distance, and Artificial Neural Networks. The results indicate that these techniques are of great value for Linguistics and for new software applications in language learning.

The Use of Computer Assisted Pronunciation Training in Teaching English Pronunciation for First-Year EFL Students at Saida University

The present research work investigates the use of Computer Assisted Pronunciation Training in teaching English pronunciation for first-year EFL students at the department of English language and literature, Saida University, Algeria. It also aims to provide a cursory account of the vital place of Information and Communication Technology in developing the linguistic skills of English foreign language learners in particular and the learning process in Algerian universities in general. The topic is an original contribution in the Algerian context since there is a scarcity of studies related to the teaching of English pronunciation through the use of ICT tools. The researchers relied on “Pronunciation Coach Software,” which focuses on developing the learner’s correct pronunciation of English sounds, including consonants and diphthongs, through an experiment with two groups of students. The investigation took six months. To evaluate students’ oral production, the researchers conducted a Pretest and a Posttest. The test proved that students’ negative attitudes towards learning English pronunciation changed after integrating the Pronunciation Coach. The results also revealed that there is an improvement in pronouncing certain sounds mainly, /S,tS/, dZ, Z/ and /T, D/ and, some vowels, including /Q, ℘/, /U,, /↔U, aU/ and /eI, aI/. The findings further demonstrated that students became more aware of the main differences between English and Arabic pronunciations in terms of place and manner of articulation of sounds.

Impacts of multicollinearity on CAPT modalities: An heterogeneous machine learning framework for computer-assisted French phoneme pronunciation training.

Phoneme pronunciations are usually considered as basic skills for learning a foreign language. Practicing the pronunciations in a computer-assisted way is helpful in a self-directed or long-distance learning environment. Recent researches indicate that machine learning is a promising method to build high-performance computer-assisted pronunciation training modalities. Many data-driven classifying models, such as support vector machines, back-propagation networks, deep neural networks and convolutional neural networks, are increasingly widely used for it. Yet, the acoustic waveforms of phoneme are essentially modulated from the base vibrations of vocal cords, and this fact somehow makes the predictors collinear, distorting the classifying models. A commonly-used solution to address this issue is to suppressing the collinearity of predictors via partial least square regressing algorithm. It allows to obtain high-quality predictor weighting results via predictor relationship analysis. However, as a linear regressor, the classifiers of this type possess very simple topology structures, constraining the universality of the regressors. For this issue, this paper presents an heterogeneous phoneme recognition framework which can further benefit the phoneme pronunciation diagnostic tasks by combining the partial least square with support vector machines. A French phoneme data set containing 4830 samples is established for the evaluation experiments. The experiments of this paper demonstrates that the new method improves the accuracy performance of the phoneme classifiers by 0.21 − 8.47% comparing to state-of-the-arts with different data training data density.