Computer Assisted Pronunciation Training Research Articles

Quantifying the impact of ASR-based instruction: What does the iSpraak platform learner data show?

Computer-assisted Pronunciation Training (CAPT) tools have become increasingly dependent on Automatic Speech Recognition (ASR) technology to provide automated corrective pronunciation feedback to learners. The extent to which ASR-based tools measurably improve second language (L2) pronunciation is of great interest to language educators globally, and Computer-assisted Language Learning (CALL) researchers. Studies to date have largely been conducted by research practitioners with small-to-medium sized samples at single institutions. The findings and conclusions drawn from such small-scale data collection might be significantly bolstered by analysing the vast stores of learner data from large CAPT platforms. This study is informed by a sizable eight-year dataset from iSpraak, an open-source pronunciation tool designed to model and evaluate L2 speech. Quantitative analysis of anonymised learner interactions with this application reveals significant gains in intelligibility measures across multiple languages. Results also suggest that the extent of ASR’s ability to improve learner pronunciation may be L2 dependent.

Computer-assisted pronunciation training: A systematic review

Abstract This systematic review maps the trends of computer-assisted pronunciation training (CAPT) research based on the pedagogy of second language (L2) pronunciation instruction and assessment. The review was limited to empirical studies investigating the effects of CAPT on healthy L2 learners’ pronunciation. Thirty peer-reviewed journal articles published between 1999 and 2022 were selected based on specific inclusion and exclusion criteria. Data were collected about the studies’ contexts, participants, experimental designs, CAPT systems, pronunciation training scopes and approaches, pronunciation assessment practices, and learning measures. Using a pedagogically informed codebook, the pronunciation training and assessment practices were classified and evaluated based on established L2 pronunciation teaching guidelines. The findings indicated that most of the studies focused on the pronunciation training of adult English learners with an emphasis on the production of segmental features (i.e. vowels and consonants) rather than suprasegmental features (i.e. stress, intonation, and rhythm). Despite the innovation promised by CAPT technology, pronunciation practice in the studies reviewed was characterized by the predominant use of drilling through listen-and-repeat and read-aloud activities. As for assessment, most CAPT studies relied on human listeners to measure the accurate production of discrete pronunciation features (i.e. segmental and suprasegmental accuracy). Meanwhile, few studies employed global pronunciation learning measures such as intelligibility and comprehensibility. Recommendations for future research are provided based on the discussion of these results.

2D Spectrogram analysis using vision transformer to detect mispronounced Arabic utterances for children

Pronunciation feedback is essential for teaching languages to children, urging the need to create computer-assisted pronunciation training (CAPT) systems to automate this process. Most of the current CAPT systems for Arabic Language examined a wide age range, including children, with a dominant adults’ portion. However, these systems did not consider many challenges facing children’s voice, in terms of the acoustic difference between children and adults. Moreover, due to the lack of publicly Arabic datasets, Arabic-related systems were evaluated on datasets consisting of the sound of Arabic letters, with minor efforts to contain words or sentences. In this paper, we propose the Arabic Utterance Mispronunciation Detector for Children (AUMD-Child) system that detects the mispronunciation of children for Arabic words at real-time. The proposed system is based on the fusion of Vision Transformer (ViT) and transfer learning-based model (AlexNet). A novel public dataset is constructed, in which 16 words have been collected from children aged between 7 and 12 years old to evaluate our proposed system. The experimental results, compared to the state-of-the-art models in literature, showed that the accuracy of our proposed AUMD-Child system exceeded the handcrafted features-based models by an average of 7 %, the CNN features-based models using AlexNet and SVM by 5 %, transfer learning-based model using AlexNet by 4 %, and transformers-based model using ViT by 2 %, achieving an average accuracy of 91.81 % for detecting the mispronunciation patterns in Arabic words but in an average processing time of 33 ms, exceeding the other state-of-the-art models with 13 %, 0.03 %, 0.06 %, and 23 % respectively.

An Examination of Automatic Speech Recognition (ASR)-based Computer-assisted Pronunciation Training (CAPT) for Less-proficient EFL Students Using the Technology Acceptance Model

An Examination of Automatic Speech Recognition (ASR)-based Computer-assisted Pronunciation Training (CAPT) for Less-proficient EFL Students Using the Technology Acceptance Model

After Self-Imitation Prosodic Training L2 Learners Converge Prosodically to the Native Speakers

Little attention is paid to prosody in second language (L2) instruction, but computer-assisted pronunciation training (CAPT) offers learners solutions to improve the perception and production of L2 suprasegmentals. In this study, we extend with acoustic analysis a previous research showing the effectiveness of self-imitation training on prosodic improvements of Japanese learners of Italian. In light of the increased degree of correct match between intended and perceived pragmatic functions (e.g., speech acts), in this study, we aimed at quantifying the degree of prosodic convergence towards L1 Italian speakers used as a model for self-imitation training. To measure convergence, we calculated the difference in duration, F0 mean, and F0 max syllable-wise between L1 utterances and the corresponding L2 utterances produced before and after training. The results showed that after self-imitation training, L2 learners converged to the L1 speakers. The extent of the effect, however, varied based on the speech act, the acoustic measure, and the distance between L1 and L2 speakers before the training. The findings from perceptual and acoustic investigations, taken together, show the potential of self-imitation prosodic training as a valuable tool to help L2 learners communicate more effectively.

An Open CAPT System for Prosody Practice: Practical Steps towards Multilingual Setup

This paper discusses the challenges posed in creating a Computer-Assisted Pronunciation Training (CAPT) environment for multiple languages. By selecting one language from each of three different language families, we show that a single environment may be tailored to cater for different target languages. We detail the challenges faced during the development of a multimodal CAPT environment comprising a toolkit that manages mobile applications using speech signal processing, visualization, and estimation algorithms. Since the applied underlying mathematical and phonological models, as well as the feedback production algorithms, are based on sound signal processing and modeling rather than on particular languages, the system is language-agnostic and serves as an open toolkit for developing phrasal intonation training exercises for an open selection of languages. However, it was necessary to tailor the CAPT environment to the language-specific particularities in the multilingual setups, especially the additional requirements for adequate and consistent speech evaluation and feedback production. In our work, we describe our response to the challenges in visualizing and segmenting recorded pitch signals and modeling the language melody and rhythm necessary for such a multilingual adaptation, particularly for tonal syllable-timed and mora-timed languages.

Moving Beyond the Plateau with Computer-Assisted Pronunciation Mediation

This paper examines the plateau effect and the role of Computer-Assisted Pronunciation Training (CAPT) in enabling learners to move beyond the plateau. Specifically, the samples and empirical evidence for this study is provided by StudyIntonation, a top-down CAPT environment supporting visual display of phrasal intonation in the form of pitch graphs presented for both the model pitch of native speakers and attempts of learners. Utterances of model speakers are grouped by situational context, prosodic forms and mode of feedback. The system largely relies upon concepts of sociocultural theory, such as mediation through the zone of proximal development. With the help of an experimental group of high school students, we use mathematical models to pinpoint the Zone of Proximal Development (ZPD), allowing for delivery of tailored task classification based on syntax and discourse tags. Longitudinal pronunciation assessment supported by dynamic time warping and cross-recurrence quantification analysis is used to locate the ZPD, and provide personalized practice. Preliminary experiments for the current study demonstrate that this approach can help learners in overcoming the plateau effect experienced during the learning process. The reported new findings create further grounds for improving and targeting the CAPT system feedback to learners with respect to their individual achievements and differences in learning styles.

The pedagogy for teaching suprasegmental features

Abstract Suprasegmental (prosody) deficiencies seriously affect English as a foreign language (EFL) learners’ comprehensibility and intelligibility. Among suprasegmental features, the most problematic for Iranian learners is rhythm, due to fundamental differences between English and Farsi language timing patterns. To improve learners’ pronunciation, computer-assisted pronunciation training (CAPT) is commonly applied to provide a private, individualized learning environment with unlimited input and visual, instant, and precise feedback. In our previous study, a specific kind of CAPT application, (MyET™) was used to enhance suprasegmental features of five Iranian EFL learners. The results showed that CAPT helped them improve their pronunciation skills. They also had positive attitudes toward taking this approach. In this short paper, we describe how prosody teaching actually occurred in the classroom, how we used the material, and how we reported the results. Suggestions and recommendations for further research are also presented.

A Bi-directional Attention Based End to End Mispronunciation Detection and Diagnosis of Mandarin

An increasing number of individuals are acquiring proficiency in Mandarin, signifying the growing significance of employing computer-assisted pronunciation training systems for Mandarin learners. One pivotal component within these systems is the technique for identifying and addressing mispronunciations, known as Mispronunciation Detection and Diagnosis (MDD). Recently, certain end-to-end techniques have tried to fuse features of prompt text and acoustic features into the model for training and have shown good results. However, previous approaches have fused acoustic features with prompt text features by a simple attention mechanism. In this paper, we posit that the impact of text features varies significantly when mapped to distinct acoustic characteristics. Furthermore, we propose that the prompt text can lead the model towards achieving an integrated text-audio representation, thereby enhancing the inference quality. Hence, this article presents a model aimed at detecting and diagnosing mispronunciations. The model utilizes a bidirectional attention mechanism to integrate acoustic and prompt text features. Good results were achieved by conducting experiments on a self-built dataset of short Mandarin read-aloud texts.

The Effect of Computer-Assisted Pronunciation Training on EFL Learners’ Use of Suprasegmental Features and Foreign Language Speaking Anxiety

The Effect of Computer-Assisted Pronunciation Training on EFL Learners’ Use of Suprasegmental Features and Foreign Language Speaking Anxiety

INTEGRATING HTML5-BASED SPEECH RECOGNITION WITH LEARNING MANAGEMENT SYSTEM TO ENHANCE EFL LEARNERS’ PRONUNCIATION SKILLS

HTML5 (HyperText Markup Language version 5) is currently integrated with almost all web-based applications due to its compatibility with numerous types of multimedia content including music, video, and animation. This exploratory study focused on the application and assessment of an HTML5-based plugin called H5P as a Computer Assisted Pronunciation Training (CAPT) tool in developing the pronunciation skills of intermediate-level learners of English as a foreign language (EFL). The target words were initially recorded with their correct pronunciations and then the audio files were imported into H5P to create interactive exercises. The H5P was then integrated with the institutional Learning Management System (LMS). The practice was implemented on n=188 EFL learners enrolled in the General Foundation (GFP) of a university college in the Sultanate of Oman. The efficacy of the approach was assessed through the perspectives of students by using student questionnaires and a student focus group discussion (FGD). The study yielded encouraging findings since most of the students agreed that the use of HTML5 as a CAPT tool was helpful in enhancing their pronunciation skills as it provided automatically generated instant feedback, was easy to use, and could be used by students autonomously at their own pace. Although the impact of software on student achievement in terms of pronunciation skills could not be gauged and the sample was limited to one institution, the study has useful implications for EFL teachers and curriculum designers on using HTML5 for developing pronunciation skills. Further studies are necessary to investigate the pedagogic potential of CAPT resources to develop a high level of competence in EFL learners’ pronunciation.

A RTL Implementation of Heterogeneous Machine Learning Network for French Computer Assisted Pronunciation Training

Computer-assisted pronunciation training (CAPT) is a helpful method for self-directed or long-distance foreign language learning. It greatly benefits from the progress, and of acoustic signal processing and artificial intelligence techniques. However, in real-life applications, embedded solutions are usually desired. This paper conceives a register-transfer level (RTL) core to facilitate the pronunciation diagnostic tasks by suppressing the mulitcollinearity of the speech waveforms. A recently proposed heterogeneous machine learning framework is selected as the French phoneme pronunciation diagnostic algorithm. This RTL core is implemented and optimized within a very-high-level synthesis method for fast prototyping. An original French phoneme data set containing 4830 samples is used for the evaluation experiments. The experiment results demonstrate that the proposed implementation reduces the diagnostic error rate by 0.79–1.33% compared to the state-of-the-art and achieves a speedup of 10.89× relative to its CPU implementation at the same abstract level of programming languages.

Computer-Assisted Pronunciation Teaching: An Analysis of Empirical Research

Computer-assisted pronunciation teaching has been shown to have favourable learning outcomes for language learners, which brought about the emergence of increasing body of research on this topic. Despite some attempted review studies, no systematic review of the line of the relevant literature has hitherto been observed. This study seeks to fill in this gap by reviewing the research on computer-assisted pronunciation teaching in L2 classes published between 2010 and 2021 May. 26 studies were scrutinized in terms of their contexts and participants, technology used, pronunciation aspect analysed, research design and focus, data collection tools and theoretical framework. The results showed that most of the studies were conducted with EFL/ESL learners at university level. Computer-assisted pronunciation training (CAPT) and Automatic Speech Recognition (ASR) programs were utilized more than other technological tools. Quantitative design was the dominant research method, which was followed by mixed method design while few studies were grounded in a theoretical framework. Segmental features of pronunciation received more attention. Pronunciation achievement, learner perceptions, relationship between some variables related to pronunciation learning and student interaction/ participation were the main topics of investigation. The findings suggested that teachers had a key role in promoting the effectiveness of computer-assisted pronunciation teaching by scaffolding, motivating, and guiding the learners as they use CAPT or technological tools. To this end teacher training that supports teachers in developing these competencies can be beneficial. Further research can focus on languages other than English, be conducted in primary and secondary school level and have longitudinal designs.

Factors predicting human performance in error annotation for non-native speech corpus

With low inter-annotator agreement and large time consumption, phonetic annotation of non-native speech is complicated. However, having a large-scale non-native speech corpus with error annotation is important for second language phonology and computer-assisted pronunciation training research. To achieve large-scale annotation, we must clarify the causes of difficulty in error annotation. This study identified the factors that can predict annotation performance of non-native Mandarin speech by considering two perspectives.1) We used a decomposed method (that involved dividing a traditional error annotation task into two different subtasks, namely the error location task and error description task) to determine whether human performance, from the cognitive viewpoint, can be predicted using the decomposed annotation method (Experiment 1). 2) We determined whether the features of annotation task materials (e.g., error severity) can be employed to predict annotation performance (Experiment 2). The results of Experiment 1 revealed that the decomposed method leads to a shorter annotation time and a higher consistent hit rate than does the traditional method. The regressions of Experiment 2 revealed that the features of error severity and sentence length accounted for 96% of the hit rate and the features of sentence length and number of errors accounted for 95.5% of the annotation time. These results suggest that the performance of manual annotation can be predicted through both cognitive factors and material features.

SELECTION OF DISCRIMINATIVE FEATURES FOR ARABIC PHONEME’S MISPRONUNCIATION DETECTION

Pronunciation training is an important part of Computer Assisted Pronunciation Training (CAPT) systems. Mispronunciation detection systems recognized pronunciation mistakes from user’s speech and provided them feedback about their pronunciation. Acoustic phonetic features plays a vital role in speech classification based applications. This research work investigated the suitability of various acoustic features: pitch, energy, spectrum flux, zero-crossing, Entropy and MelFrequency Cepstral Coefficients (MFCCs). Sequential Forward Selection (SFS) was used to find out most suitable acoustic features from the computed feature set. This study used K-Nearest Neighbors (K-NN) classifier was used to detect the pronunciation mistakes from Arabic phonemes. This research selected the set of most discriminative acoustic features for each phoneme. K-NN achieved accuracy of 92.15% for mispronunciation detection of Arabic Phonemes.

SELECTION OF DISCRIMINATIVE FEATURES FOR ARABIC PHONEME’S MISPRONUNCIATION DETECTION

Pronunciation training is an important part of Computer Assisted Pronunciation Training (CAPT) systems. Mispronunciation detection systems recognized pronunciation mistakes from user’s speech and provided them feedback about their pronunciation. Acoustic phonetic features plays a vital role in speech classification based applications. This research work investigated the suitability of various acoustic features: pitch, energy, spectrum flux, zero-crossing, Entropy and MelFrequency Cepstral Coefficients (MFCCs). Sequential Forward Selection (SFS) was used to find out most suitable acoustic features from the computed feature set. This study used K-Nearest Neighbors (K-NN) classifier was used to detect the pronunciation mistakes from Arabic phonemes. This research selected the set of most discriminative acoustic features for each phoneme. K-NN achieved accuracy of 92.15% for mispronunciation detection of Arabic Phonemes

Synchronous computer-mediated communication in pronunciation teaching: A survey study

Remote learning has been in the spotlight since the emergence of the coronavirus pandemic, notably boosting the development of distance learning in the last two years; as universities were obliged to close worldwide and lecturers had to teach online during lockdown, the use of Synchronous Computer-Mediated Communication (SCMC) started to prevail. Pronunciation instruction was not an exception. Indeed, in-person lectures had used technology to support face-to-face (F2F) education by using videos and articles in class. However, educators had to adapt rapidly to teach through video-based platforms. Although much investigation has been conducted on Computer-Assisted Pronunciation Training (CAPT), and researchers currently show more and more interest in the use of new technologies within the pronunciation classroom, little research has been done regarding the teaching methodologies applied in teacher-guided online pronunciation instruction.

Use of ASR-Equipped Software in the Teaching of Suprasegmental Features of Pronunciation

Technology has paved the way for new modalities in language learning, teaching, and assessment. However, there is still a great deal of work to be done to develop such tools for oral communication, specifically tools that address suprasegmental features in pronunciation instruction. Therefore, this critical literature review examines how researchers have tried to create computer-assisted pronunciation training tools using automatic speech recognition (ASR) systems to aid language learners in the perception and production of suprasegmental features. We used 30 texts from 1990 to 2020 to explore how technologies have been and are currently being used to help learners develop their proficiency with suprasegmental features. Based on our thematic analysis, a persistent gap still exists between ASR-equipped software available to participants in research studies and what is available to university and classroom teachers and students. Additionally, there seems to be more development in the production of speech software for language assessment. In contrast, the translation of these tools into instructional tools for individualized learning seems to be almost non-existent. Moving forward, we recommend that more commercialized pronunciation systems utilizing ASR should be made publicly available using the technologies that are currently developed or are in development for the purposes of oral proficiency judgments.

Mispronunciation Detection and Diagnosis with Articulatory-Level Feedback Generation for Non-Native Arabic Speech

A high-performance versatile computer-assisted pronunciation training (CAPT) system that provides the learner immediate feedback as to whether their pronunciation is correct is very helpful in learning correct pronunciation and allows learners to practice this at any time and with unlimited repetitions, without the presence of an instructor. In this paper, we propose deep learning-based techniques to build a high-performance versatile CAPT system for mispronunciation detection and diagnosis (MDD) and articulatory feedback generation for non-native Arabic learners. The proposed system can locate the error in pronunciation, recognize the mispronounced phonemes, and detect the corresponding articulatory features (AFs), not only in words but even in sentences. We formulate the recognition of phonemes and corresponding AFs as a multi-label object recognition problem, where the objects are the phonemes and their AFs in a spectral image. Moreover, we investigate the use of cutting-edge neural text-to-speech (TTS) technology to generate a new corpus of high-quality speech from predefined text that has the most common substitution errors among Arabic learners. The proposed model and its various enhanced versions achieved excellent results. We compared the performance of the different proposed models with the state-of-the-art end-to-end technique of MDD, and our system had a better performance. In addition, we proposed using fusion between the proposed model and the end-to-end model and obtained a better performance. Our best model achieved a 3.83% phoneme error rate (PER) in the phoneme recognition task, a 70.53% F1-score in the MDD task, and a detection error rate (DER) of 2.6% for the AF detection task.

An Acoustic Feature-Based Deep Learning Model for Automatic Thai Vowel Pronunciation Recognition

For Thai vowel pronunciation, it is very important to know that when mispronunciation occurs, the meanings of words change completely. Thus, effective and standardized practice is essential to pronouncing words correctly as a native speaker. Since the COVID-19 pandemic, online learning has become increasingly popular. For example, an online pronunciation application system was introduced that has virtual teachers and an intelligent process of evaluating students that is similar to standardized training by a teacher in a real classroom. This research presents an online automatic computer-assisted pronunciation training (CAPT) using deep learning to recognize Thai vowels in speech. The automatic CAPT is developed to solve the inadequacy of instruction specialists and the complex vowel teaching process. It is a unique system that develops computer techniques integrated with linguistic theory. The deep learning model is the most significant part of recognizing vowels pronounced for the automatic CAPT. The major challenge in Thai vowel recognition is the correct identification of Thai vowels when spoken in real-world situations. A convolutional neural network (CNN), a deep learning model, is applied and developed in the classification of pronounced Thai vowels. A new dataset for Thai vowels was designed, collected, and examined by linguists. The result of an optimal CNN model with Mel spectrogram (MS) achieves the highest accuracy of 98.61%, compared with Mel frequency cepstral coefficients (MFCC) with the baseline long short-term memory (LSTM) model and MS with the baseline LSTM model have an accuracy of 94.44% and 90.00% respectively.