Computer Assisted Pronunciation Training System Research Articles

Computer-assisted pronunciation training: A systematic review

Abstract This systematic review maps the trends of computer-assisted pronunciation training (CAPT) research based on the pedagogy of second language (L2) pronunciation instruction and assessment. The review was limited to empirical studies investigating the effects of CAPT on healthy L2 learners’ pronunciation. Thirty peer-reviewed journal articles published between 1999 and 2022 were selected based on specific inclusion and exclusion criteria. Data were collected about the studies’ contexts, participants, experimental designs, CAPT systems, pronunciation training scopes and approaches, pronunciation assessment practices, and learning measures. Using a pedagogically informed codebook, the pronunciation training and assessment practices were classified and evaluated based on established L2 pronunciation teaching guidelines. The findings indicated that most of the studies focused on the pronunciation training of adult English learners with an emphasis on the production of segmental features (i.e. vowels and consonants) rather than suprasegmental features (i.e. stress, intonation, and rhythm). Despite the innovation promised by CAPT technology, pronunciation practice in the studies reviewed was characterized by the predominant use of drilling through listen-and-repeat and read-aloud activities. As for assessment, most CAPT studies relied on human listeners to measure the accurate production of discrete pronunciation features (i.e. segmental and suprasegmental accuracy). Meanwhile, few studies employed global pronunciation learning measures such as intelligibility and comprehensibility. Recommendations for future research are provided based on the discussion of these results.

Moving Beyond the Plateau with Computer-Assisted Pronunciation Mediation

This paper examines the plateau effect and the role of Computer-Assisted Pronunciation Training (CAPT) in enabling learners to move beyond the plateau. Specifically, the samples and empirical evidence for this study is provided by StudyIntonation, a top-down CAPT environment supporting visual display of phrasal intonation in the form of pitch graphs presented for both the model pitch of native speakers and attempts of learners. Utterances of model speakers are grouped by situational context, prosodic forms and mode of feedback. The system largely relies upon concepts of sociocultural theory, such as mediation through the zone of proximal development. With the help of an experimental group of high school students, we use mathematical models to pinpoint the Zone of Proximal Development (ZPD), allowing for delivery of tailored task classification based on syntax and discourse tags. Longitudinal pronunciation assessment supported by dynamic time warping and cross-recurrence quantification analysis is used to locate the ZPD, and provide personalized practice. Preliminary experiments for the current study demonstrate that this approach can help learners in overcoming the plateau effect experienced during the learning process. The reported new findings create further grounds for improving and targeting the CAPT system feedback to learners with respect to their individual achievements and differences in learning styles.

A Bi-directional Attention Based End to End Mispronunciation Detection and Diagnosis of Mandarin

An increasing number of individuals are acquiring proficiency in Mandarin, signifying the growing significance of employing computer-assisted pronunciation training systems for Mandarin learners. One pivotal component within these systems is the technique for identifying and addressing mispronunciations, known as Mispronunciation Detection and Diagnosis (MDD). Recently, certain end-to-end techniques have tried to fuse features of prompt text and acoustic features into the model for training and have shown good results. However, previous approaches have fused acoustic features with prompt text features by a simple attention mechanism. In this paper, we posit that the impact of text features varies significantly when mapped to distinct acoustic characteristics. Furthermore, we propose that the prompt text can lead the model towards achieving an integrated text-audio representation, thereby enhancing the inference quality. Hence, this article presents a model aimed at detecting and diagnosing mispronunciations. The model utilizes a bidirectional attention mechanism to integrate acoustic and prompt text features. Good results were achieved by conducting experiments on a self-built dataset of short Mandarin read-aloud texts.

Mispronunciation Detection and Diagnosis with Articulatory-Level Feedback Generation for Non-Native Arabic Speech

A high-performance versatile computer-assisted pronunciation training (CAPT) system that provides the learner immediate feedback as to whether their pronunciation is correct is very helpful in learning correct pronunciation and allows learners to practice this at any time and with unlimited repetitions, without the presence of an instructor. In this paper, we propose deep learning-based techniques to build a high-performance versatile CAPT system for mispronunciation detection and diagnosis (MDD) and articulatory feedback generation for non-native Arabic learners. The proposed system can locate the error in pronunciation, recognize the mispronounced phonemes, and detect the corresponding articulatory features (AFs), not only in words but even in sentences. We formulate the recognition of phonemes and corresponding AFs as a multi-label object recognition problem, where the objects are the phonemes and their AFs in a spectral image. Moreover, we investigate the use of cutting-edge neural text-to-speech (TTS) technology to generate a new corpus of high-quality speech from predefined text that has the most common substitution errors among Arabic learners. The proposed model and its various enhanced versions achieved excellent results. We compared the performance of the different proposed models with the state-of-the-art end-to-end technique of MDD, and our system had a better performance. In addition, we proposed using fusion between the proposed model and the end-to-end model and obtained a better performance. Our best model achieved a 3.83% phoneme error rate (PER) in the phoneme recognition task, a 70.53% F1-score in the MDD task, and a detection error rate (DER) of 2.6% for the AF detection task.

Web-based intonation training helps improve ESL and EFL Chinese students' oral speech

This paper examines whether a web-based training on English discourse intonation leads to better spontaneous speech quality for Mandarin Chinese speakers who reside in the U.S. and in China. The four-week fully online training consisted of meta-instruction videos as well as listening and speaking activities, including instant visual pitch contour feedback and individualized evaluation. The students gave a one-minute spontaneous speech on a given topic at the beginning and the end of the study via videoconferencing. Four native English speakers judged the students' speech comprehensibility, fluidity, accent, confidence and attractiveness, in addition to their intonation performance. Two-way ANCOVA test results show that the experimental group made statistically significant improvement in their speech comprehensibility and speaking confidence. In contrast, the control group did not show improvement. The participants' residence in the U.S. or in China did not affect the training effects. There was not an interaction between the participants' residence and the training. The web-based training, visualization and CMC technology provided an effective scaffolding experience and benefited both EFL and ESL students equally. This study also explores Chinese students' challenges with specific intonation features based on both the raters' judgments and the learners' self-evaluations. The results suggest they have more difficulties with thought groups and prominence than with tone choices. While the trainees gave high ratings to all the activities, they preferred individualized evaluation from the researcher to self-created visual feedback using Praat. The findings have implications for Chinese L1-specific intonation instruction and developing web-based computer assisted pronunciation training systems.

Computer-Assisted Pronunciation Training (CAPT): Current Issues and Future Directions

This viewpoint essay considers the current status of computer-assisted pronunciation training (CAPT) before examining some of the current issues and future directions in the field. The underlying premise is the pedagogic potential of CAPT systems and resources for teaching and learning, and the need for greater synergy between technological design and functionality on the one hand, and pedagogic purpose on the other. Some of the key issues examined include providing accurate and individualised automated feedback for pronunciation, for both learning and assessment, and evaluating the effectiveness of CAPT tools and systems. When considering future directions, the discussion focuses on what aspects of pedagogy are likely to be at the forefront of developments, including ubiquitous learning; intelligent tutoring and authentic interaction; and goal-oriented, task-based learning.

Speech Processing for Language Learning: A Practical Approach to Computer-Assisted Pronunciation Teaching

This article contributes to the discourse on how contemporary computer and information technology may help in improving foreign language learning not only by supporting better and more flexible workflow and digitizing study materials but also through creating completely new use cases made possible by technological improvements in signal processing algorithms. We discuss an approach and propose a holistic solution to teaching the phonological phenomena which are crucial for correct pronunciation, such as the phonemes; the energy and duration of syllables and pauses, which construct the phrasal rhythm; and the tone movement within an utterance, i.e., the phrasal intonation. The working prototype of StudyIntonation Computer-Assisted Pronunciation Training (CAPT) system is a tool for mobile devices, which offers a set of tasks based on a “listen and repeat” approach and gives the audio-visual feedback in real time. The present work summarizes the efforts taken to enrich the current version of this CAPT tool with two new functions: the phonetic transcription and rhythmic patterns of model and learner speech. Both are designed on a base of a third-party automatic speech recognition (ASR) library Kaldi, which was incorporated inside StudyIntonation signal processing software core. We also examine the scope of automatic speech recognition applicability within the CAPT system workflow and evaluate the Levenstein distance between the transcription made by human experts and that obtained automatically in our code. We developed an algorithm of rhythm reconstruction using acoustic and language ASR models. It is also shown that even having sufficiently correct production of phonemes, the learners do not produce a correct phrasal rhythm and intonation, and therefore, the joint training of sounds, rhythm and intonation within a single learning environment is beneficial. To mitigate the recording imperfections voice activity detection (VAD) is applied to all the speech records processed. The try-outs showed that StudyIntonation can create transcriptions and process rhythmic patterns, but some specific problems with connected speech transcription were detected. The learners feedback in the sense of pronunciation assessment was also updated and a conventional mechanism based on dynamic time warping (DTW) was combined with cross-recurrence quantification analysis (CRQA) approach, which resulted in a better discriminating ability. The CRQA metrics combined with those of DTW were shown to add to the accuracy of learner performance estimation. The major implications for computer-assisted English pronunciation teaching are discussed.

Smart media and application

Smart media and application

Beyond self-directed computer-assisted pronunciation learning: a qualitative investigation of a collaborative approach

In foreign language education, instruction in pronunciation has received less attention than instruction in other language skills. Many teachers do not know how to teach correct pronunciation due to lack of knowledge in pronunciation pedagogy even if they want to or they assume that students pick it up on their own. With the development of technology, automatic speech recognition (ASR) software has been adopted in computer-assisted pronunciation training (CAPT) and has been found to be effective in facilitating pronunciation learning. Nevertheless, CAPT software still has its limitation. Very little research has been done on what difficulties students may encounter during practice with the system, such as how to read the feedback provided by the software. Moreover, very few CAPT studies investigated how collaborative learning can be integrated to augment the advantages of the technology. Efforts also have to be made on how to increase the enjoyment of learners using it when their motivation is curtailed by the unchanging feedback the CAPT system provides. The present study aims to investigate how those software users feel about the mediated assistance the system offers, their difficulty and attitudes toward using it, and, above all, how peers can help to mediate the pronunciation learning of students using MyET, a CAPT system designed in Taiwan. Two groups of junior college students (N = 60) practiced the system for 10 weeks with or without their peers. A qualitative investigation found that the collaborative group wrote down more statements about how they had benefited from the software than about their pronunciation difficulties. They also reported that the feedback helped raise their awareness of pronunciation errors. In the collaborative practice with the CAPT system, they revealed that they could explore more functions the software provided and had fun. This study further examines how peers interacted, analyzed, and discussed the effects of collaboration on computer-assisted pronunciation learning. Finally, suggestions were made for teachers interested in conducting CAPT in a collaborative context.

A Computer-Assisted Pronunciation Training System for Correcting Pronunciation of Adjacent Phonemes

A Computer-Assisted Pronunciation Training System for Correcting Pronunciation of Adjacent Phonemes

Automatic analysis of pronunciations for children with speech sound disorders

Computer-Assisted Pronunciation Training (CAPT) systems aim to help a child learn the correct pronunciations of words. However, while there are many online commercial CAPT apps, there is no consensus among Speech Language Therapists (SLPs) or non-professionals about which CAPT systems, if any, work well. The prevailing assumption is that practicing with such programs is less reliable and thus does not provide the feedback necessary to allow children to improve their performance. The most common method for assessing pronunciation performance is the Goodness of Pronunciation (GOP) technique. Our paper proposes two new GOP techniques. We have found that pronunciation models that use explicit knowledge about error pronunciation patterns can lead to more accurate classification whether a phoneme was correctly pronounced or not. We evaluate the proposed pronunciation assessment methods against a baseline state of the art GOP approach, and show that the proposed techniques lead to classification performance that is more similar to that of a human expert.

Dynamic time warping in phoneme modeling for fast pronunciation error detection

The presented paper describes a novel approach to the detection of pronunciation errors. It makes use of the modeling of well-pronounced and mispronounced phonemes by means of the Dynamic Time Warping (DTW) algorithm. Four approaches that make use of the DTW phoneme modeling were developed to detect pronunciation errors: Variations of the Word Structure (VoWS), Normalized Phoneme Distances Thresholding (NPDT), Furthest Segment Search (FSS) and Normalized Furthest Segment Search (NFSS). The performance evaluation of each module was carried out using a speech database of correctly and incorrectly pronounced words in the Polish language, with up to 10 patterns of every trained word from a set of 12 words having different phonetic structures. The performance of DTW modeling was compared to Hidden Markov Models (HMM) that were used for the same four approaches (VoWS, NPDT, FSS, NFSS). The average error rate (AER) was the lowest for DTW with NPDT (AER=0.287) and scored better than HMM with FSS (AER=0.473), which was the best result for HMM. The DTW modeling was faster than HMM for all four approaches. This technique can be used for computer-assisted pronunciation training systems that can work with a relatively small training speech corpus (less than 20 patterns per word) to support speech therapy at home.

Error Patterns for Automatic Error Detection in Computer Assisted Pronunciation Training Systems

This paper presents error patterns built on the basis of our comparative analysis of American English and Mexican Spanish phonemes and allophones which can be applied in designing the error detection module of a Computer Assisted Pronunciation Training (CAPT) System for teaching American English pronunciation to Mexican Spanish speakers. Error identification is important for an adequate choice of correcting techniques which improves English pronunciation acquisition and helps learners to develop less accented speech. Since automatic individual error detection remains a highly complex computational task, error patterns can enhance the system performance and improve its precision. To the best of our knowledge, error patterns in American English speech generated by Mexican Spanish speakers has not been defined in previous work which was done mainly for Castilian- originated standard Spanish.

A computer assisted pronunciation training system

A computer assisted pronunciation training (CAPT) system is implemented for native Korean speakers who are learning American English. The CAPT system is designed to help a Korean adult learner improve his/her production and perception of American English front vowels (/i, I, ɛ, æ/) since these vowels are the most difficult for Korean learners due to the different phonetic systems of the two languages. The CAPT system provides a learner a learning session mimicking a live interaction between teacher and student as well as a practice session triggering a learner’s interest in continued practice. Pedagogically meaningful activities such as listen-and-repeat, minimal-pair-comparison, target-sound-isolation, and record-and-play are utilized in the learning session. During the learning session, the CAPT system analyzes a monosyllabic word including one of the target front vowels spoken by a learner and gives instantaneous personalized feedback. During the practice session, the CAPT system provides real-time games that are fun but also provide the necessary perception and articulation practice.

Linguistic Support of a CAPT System for Teaching English Pronunciation to Mexican Spanish Speakers

This paper presents an error prevention approach in the development of a Computer Assisted Pronunciation Training System (CAPT System) for teaching American English pronunciation to adult Mexican Spanish speakers developed for the case of vowels. Prior knowledge of the learner’s typical articulatory and auditory perception errors enhances the effectiveness of training. It enables to organize the teaching material in a way that foresees possible errors as well as to select appropriate exercises instead of using general pronunciation drills. Our first contribution is an extended comparative analysis of American English and Mexican Spanish vowels at the level of both phonemes and allophones. To the best of our knowledge, such analysis was not done in previous work. Another contribution of this work is a two-fold application of our analysis results: on the one hand, we use the results in designing error patterns to be employed in the error detection module of the CAPT system, and on the other hand, we employ the results as a basis for creating the linguistic content of the error prevention oriented tutor module of the same system. We illustrate this approach with examples.

Investigation of golden speakers for second language learners from imitation preference perspective by voice modification

This paper investigates what voice features (e.g., speech rate and pitch-formants) make a teacher’s voice preferable for second language learners to imitate, when they practice sentence pronunciation using Computer-Assisted Pronunciation Training (CAPT) systems. The CAPT system employed in our investigation uses a single teacher’s voice as the source to automatically resynthesize several sample voices with different voice features based on the features of a learner’s voice. Our approach is different from that in the study conducted by Probst et al. which uses multiple native speakers’ voices as sample voices [Probst, K., Ke, Y., Eskenazi, M., 2002. Enhancing foreign language tutors—in search of the golden speaker. Speech Communication 37 (3–4), 161–173]. Our approach can reduce the influence of characteristics of teachers’ voices (e.g., voice quality and clarity) on the investigation. Our experimental results show that a teacher’s voice, which has similar speech rate and pitch-formants to a learner’s voice, is not always the learner’s first imitation preference. Many factors can influence learners’ imitation preferences, e.g., background and proficiency of the language that they are learning. Also, a learner’s preferences may change at different learning stages. We thus advocate an automatic voice modification function in CAPT systems to provide speech learning material with a wide variety of voice features, e.g., different speech rates or different pitch-formants. Learners then can control the voice modifications according to their preferences.

The Effects of Computer-Assisted Pronunciation Training in the Production of English Coda Consonants and Consonant Clusters

This study is designed to investigate Korean speakers' error patterns in English coda consonant and consonant cluster production, and the effects of a Computer-Assisted Pronunciation Training (CAPT) system. In addition, this study examines the learners' perception of the CAPT program. For these purposes, this study compares the results of the pretest and those of the posttest after a two-week pronunciation training session using Dr. Speaking. A questionnaire is used to gauge participants' reactions to the CAPT system. The results showed that in both the pretest and posttest the majority of errors were made when consonant clusters occurred in the coda position, and vowel insertion errors were predominant. Further, participants showed considerable progress in the pronunciation of English coda consonants and consonant clusters after a short-term training program. The participants' responses to the CAPT system, with respect to the grades and feedback they received, were very positive. These results suggest that a CAPT system providing automatic corrective feedback can be a useful pedagogical tool. However, more individually tailored feedback is needed in place of repetitive and standardized feedback in order to improve the effectiveness of the system.

The effectiveness of computer assisted pronunciation training for foreign language learning by children

This study investigates whether a computer assisted pronunciation training (CAPT) system can help young learners improve word-level pronunciation skills in English as a foreign language at a level comparable to that achieved through traditional teacher-led training. The pronunciation improvement of a group of learners of 11 years of age receiving teacher-fronted instruction was compared to that of a group receiving computer assisted pronunciation training by means of a system including an automatic speech recognition component. Results show that 1) pronunciation quality of isolated words improved significantly for both groups of subjects, and 2) both groups significantly improved in pronunciation quality of words that were considered particularly difficult to pronounce and that were likely to have been unknown to them prior to the training. Training with a computer-assisted pronunciation training system with a simple automatic speech recognition component can thus lead to short-term improvements in pronunciation that are comparable to those achieved by means of more traditional, teacher-led pronunciation training.

The effectiveness of computer-based speech corrective feedback for improving segmental quality in L2 Dutch

AbstractAlthough the success of automatic speech recognition (ASR)-based Computer Assisted Pronunciation Training (CAPT) systems is increasing, little is known about the pedagogical effectiveness of these systems. This is particularly regrettable because ASR technology still suffers from limitations that may result in the provision of erroneous feedback, possibly leading to learning breakdowns. To study the effectiveness of ASR-based feedback for improving pronunciation, we developed and tested a CAPT system providing automatic feedback on Dutch phonemes that are problematic for adult learners of Dutch. Thirty immigrants who were studying Dutch were assigned to three groups using either the ASR-based CAPT system with automatic feedback, a CAPT system without feedback, or no CAPT system. Pronunciation quality was assessed for each participant before and after the training by human experts who evaluated overall segmental quality and the quality of the phonemes addressed in the training. The participants' impressions of the CAPT system used were also studied through anonymous questionnaires. The results on global segmental quality show that the group receiving ASR-based feedback made the largest mean improvement, but the groups' mean improvements did not differ significantly. The group receiving ASR-based feedback showed a significantly larger improvement than the no-feedback group in the segmental quality of the problematic phonemes targeted.

Pronunciation feedback from real and virtual language teachers

The aim of this paper is to summarise how pronunciation feedback on the phoneme level should be given in computer-assisted pronunciation training (CAPT) in order to be effective. The study contains a literature survey of feedback in the language classroom, interviews with language teachers and their students about their attitudes towards pronunciation feedback, and observations of how feedback is given in their classrooms. The study was carried out using focus group meetings, individual semi-structured interviews and classroom observations. The feedback strategies that were advocated and observed in the study on pronunciation feedback from human teachers were implemented in a computer-animated language tutor giving articulation feedback. The virtual tutor was subsequently tested in a user trial and evaluated with a questionnaire. The article proposes several feedback strategies that would improve the pedagogical soundness of CAPT systems.