Computer Assisted Pronunciation Training Research Articles

Estoñol, a computer-assisted pronunciation training tool for Spanish L1 speakers to improve the pronunciation and perception of Estonian vowels

Over the past few years the number of online language teaching materials for non-native speakers of Estonian has increased. However, they focus mainly on vocabulary and pay little attention to pronunciation. In this study we introduce a computerassisted pronunciation training tool, Estoñol, developed to help native speakers of Spanish to train their perception and production of Estonian vowels. The tool’s training program involves seven vowel contrasts, /i-y/, /u-y/, /ɑ-o/, /ɑ-æ/, /e-æ/, /o-ø/, and /o-ɤ/, which have proven to be difficult for native speakers of Spanish. The training activities include theoretical videos and four training modes (exposure, discrimination, pronunciation, and mixed) in every lesson. The tool is integrated into a pre/post-test design experiment with native speakers of Spanish and Estonian to assess the language learners’ perception and production improvement. It is expected that the tool will have a positive effect on the results, as has been shown in previous studies using similar methodology.
 Kokkuvõte. Katrin Leppik ja Cristian Tejedor-García: Estoñol, mobiilirakendus hispaania emakeelega eesti keele õppijatele vokaalide häälduse ja taju treenimiseks. Eesti keele õppimiseks on loodud mitmeid e-kursusi ja mobiilirakendusi, kuid need keskenduvad peamiselt sõnavara ja gram matika õpetamisele ning pööravad väga vähe tähelepanu hääldusele. Eesti keele häälduse omandamise lihtsustamiseks töötati välja mobiilirakendus Estoñol, mis on mõeldud hispaania emakeelega eesti keele õppijatele. Varasemad uurimused on näidanud, et hispaania emakeelega eesti keele õppijatele valmistab raskusi vokaalide /ɑ, y, ø, æ, ɤ/ hääldamine. Mobiilirakenduse sisu on jagatud seitsmeks peatükiks, kus on võimalik harjutada vokaalipaaride /i-y/, /u-y/, /ɑ-o/, /ɑ-æ/, /e-æ/, /o-ø/, /o-ɤ/ tajumist ja hääldamist. Iga peatükk algab teoreetilise videoga, millele järgnevad taju- ja hääldusharjutused. Mobiilirakenduse mõju hindamiseks keeleõppija hääldusele ja tajule plaanitakse läbi viia eksperiment.
 Märksõnad: CAPT, eesti keel, hispaania keel, L2, hääldus, taju, vokaalid, Estoñol

Cross-Lingual Transfer Learning of Non-Native Acoustic Modeling for Pronunciation Error Detection and Diagnosis

In computer-assisted pronunciation training (CAPT), the scarcity of large-scale non-native corpora and human expert annotations are two fundamental challenges to non-native acoustic modeling. Most existing approaches of acoustic modeling in CAPT are based on non-native corpora while there are so many living languages in the world. It is impractical to collect and annotate every non-native speech corpus considering different language pairs. In this work, we address non-native acoustic modeling (both on phonetic and articulatory level) based on transfer learning. In order to effectively train acoustic models of non-native speech without using such data, we propose to exploit two large native speech corpora of learner's native language (L1) and target language (L2) to model cross-lingual phenomena. This kind of transfer learning can provide a better feature representation of non-native speech. Experimental evaluations are carried out for Japanese speakers learning English. We first demonstrate the proposed acoustic-phone model achieves a lower word error rate in non-native speech recognition. It also improves the pronunciation error detection based on goodness of pronunciation (GOP) score. For diagnosis of pronunciation errors, the proposed acoustic-articulatory modeling method is effective for providing detailed feedback at the articulation level.

Smart media and application

Smart media and application

Beyond self-directed computer-assisted pronunciation learning: a qualitative investigation of a collaborative approach

In foreign language education, instruction in pronunciation has received less attention than instruction in other language skills. Many teachers do not know how to teach correct pronunciation due to lack of knowledge in pronunciation pedagogy even if they want to or they assume that students pick it up on their own. With the development of technology, automatic speech recognition (ASR) software has been adopted in computer-assisted pronunciation training (CAPT) and has been found to be effective in facilitating pronunciation learning. Nevertheless, CAPT software still has its limitation. Very little research has been done on what difficulties students may encounter during practice with the system, such as how to read the feedback provided by the software. Moreover, very few CAPT studies investigated how collaborative learning can be integrated to augment the advantages of the technology. Efforts also have to be made on how to increase the enjoyment of learners using it when their motivation is curtailed by the unchanging feedback the CAPT system provides. The present study aims to investigate how those software users feel about the mediated assistance the system offers, their difficulty and attitudes toward using it, and, above all, how peers can help to mediate the pronunciation learning of students using MyET, a CAPT system designed in Taiwan. Two groups of junior college students (N = 60) practiced the system for 10 weeks with or without their peers. A qualitative investigation found that the collaborative group wrote down more statements about how they had benefited from the software than about their pronunciation difficulties. They also reported that the feedback helped raise their awareness of pronunciation errors. In the collaborative practice with the CAPT system, they revealed that they could explore more functions the software provided and had fun. This study further examines how peers interacted, analyzed, and discussed the effects of collaboration on computer-assisted pronunciation learning. Finally, suggestions were made for teachers interested in conducting CAPT in a collaborative context.

The effectiveness of computer‐assisted pronunciation training: A meta‐analysis

Computers are used in almost all aspects of language learning and have also created numerous opportunities in this area. This study aims to examine the overall effectiveness of using a computer in pronunciation learning and training. A meta‐analysis of studies was conducted, which compared the outcomes of students learning the pronunciation of a foreign language (FL) using a computer with the results from those learning pronunciation based on traditional methods of teaching. This meta‐analysis examined how effect sizes varied depending on the learners’ age, learners’ target language proficiency, features of pronunciation, educational level, settings, treatment duration, and assessment tasks. This meta‐analysis examined the effect of using a computer on pronunciation learning by reviewing 20 studies involving 1014 participants. The findings of the meta‐analysis suggest that computer‐assisted training has a positive effect on FL pronunciation (d = 0.68), and is equally effective for both young and adult learners, but more effective for beginner and intermediate learners than advanced learners. The studies also suggest that computer‐assisted pronunciation training (CAPT) is more beneficial for university students than for school students. However, caution has to be exercised in interpreting the results. The small number of studies, all of very low quality (most with fewer than 100 participants, conducted within the same institution using intervention‐related assessments) means that the evidence of effectiveness can only be indicative and not conclusive. More robust, large‐scale studies involving several institutions and using objective standardised tests would be needed for more conclusive results.

A Computer-Assisted Pronunciation Training System for Correcting Pronunciation of Adjacent Phonemes

A Computer-Assisted Pronunciation Training System for Correcting Pronunciation of Adjacent Phonemes

King's Speech: Pronounce a Foreign Language with Style

Computer assisted pronunciation training requires strategies that capture the attention of the learners and guide them along the learning pathway. In this paper, we introduce an immersive storytelling scenario for creating appropriate learning conditions. The proposed learning interaction is orchestrated by a spoken karaoke. We motivate the concept of the spoken karaoke and describe our design. Driven by the requirements of the proposed scenario, we suggest a modular architecture designed for immersive learning applications. We present our prototype system and our approach for the processing of spoken and visual interaction modalities. Finally, we discuss how technological challenges can be addressed in order to enable the learner's self-evaluation.

Computer-Assisted Visual Articulation Feedback in L2 Pronunciation Instruction: A Review.

Language learning is a multimodal endeavor; to improve their pronunciation in a new language, learners access not only auditory information about speech sounds and patterns, but also visual information about articulatory movements and processes. With the development of new technologies in computer-assisted pronunciation training (CAPT) come new possibilities for delivering feedback in both auditory and visual modalities. The present paper surveys the literature on computer-assisted visual articulation feedback, including direct feedback that provides visual models of articulation and indirect feedback that uses visualized acoustic information as a means to inform articulation instruction. Our focus is explicitly on segmental features rather than suprasegmental ones, with visual feedback conceived of as providing visualizations of articulatory configurations, movements, and processes. In addition to discussing types of visual articulation feedback, we also consider the criteria for effective delivery of feedback, and methods of evaluation.

Automatic analysis of pronunciations for children with speech sound disorders

Computer-Assisted Pronunciation Training (CAPT) systems aim to help a child learn the correct pronunciations of words. However, while there are many online commercial CAPT apps, there is no consensus among Speech Language Therapists (SLPs) or non-professionals about which CAPT systems, if any, work well. The prevailing assumption is that practicing with such programs is less reliable and thus does not provide the feedback necessary to allow children to improve their performance. The most common method for assessing pronunciation performance is the Goodness of Pronunciation (GOP) technique. Our paper proposes two new GOP techniques. We have found that pronunciation models that use explicit knowledge about error pronunciation patterns can lead to more accurate classification whether a phoneme was correctly pronounced or not. We evaluate the proposed pronunciation assessment methods against a baseline state of the art GOP approach, and show that the proposed techniques lead to classification performance that is more similar to that of a human expert.

Comparative analysis of American English and Mexican Spanish consonants for Computer Assisted Pronunciation Training

The objective of this work is two-fold. Firstly, we aim at detecting similarities and differences between the consonant systems of two languages, namely, American English and Mexican Spanish. To achieve this, we perform a theoretic comparative analysis of consonants of the two languages at the level of both phonemes and allophones. Secondly, a possible practical usage of our results is considered; therefore, as an example of an application, we consider computer-assisted pronunciation training (CAPT) for teaching American English pronunciation to Mexican Spanish speakers. In particular, we took advantage of the results of our analysis to define some hypothetic error patterns which can be used as a starting point for diagnosing possible mispronunciations, their posterior verification, and adjustment taking into account the principles of phonotactics and empirical phonetic analysis of the English learners’ speech. The latter will result in error rules to be applied in a CAPT system for error identification and generation of appropriate corrective feedback. An adequate choice of correcting techniques will improve English pronunciation acquisition and help learners to develop less accented speech. Also, similarities found between the two consonant systems make it possible to organize and present the pronunciation teaching material using a stress-free method of helping learners to adjust their speech organs to new sounds building on the phonetic habits of their first language.

Can second language suprasegmentals be learned? A study on Japanese learners of Italian as foreign language

One of the biggest challenges for L2 learners is to develop native-like prosodic competence. There are evidence that computer-assisted pronunciation training helps learners perceive and produce L2 suprasegmentals. In this study we test to the effectiveness of one of the techniques developed in the area of spoken language technology for education and language learning—self-imitation—to facilitate the acquisition of the rhythmic and prosodic characteristics of Italian. 7 Japanese learners of Italian as Foreign language (NNSs) and 2 Italian native speakers (NSs) were asked to read aloud and record two sentences in Italian conveying different pragmatic functions. The utterances of NNSs were manipulated as they receive the segmental durational characteristics as well as the f0 characteristics of the corresponding NSs’ utterances. NNSs imitated their own voice previously modified to match the reference NSs and recorded the new performance. To quantify the degree of approximation to the native prosodic and rhythmic pattern after self-imitation, pre- and post-training utterances were compared to those of the native models by speech rate, f0, and segmental durational characteristics. Preliminary results show that after the training utterance duration and vocalic intervals length better match the target duration.

Smartphone-Assisted Pronunciation Learning Technique for Ambient Intelligence

In an ambient intelligence (AmI) environment, electronic devices that comprise the Internet of things (IoT) network work together seamlessly to provide a wide variety of applications and intelligent services to users. Computer-assisted pronunciation training (CAPT), a widely used application in the traditional Internet environment that corrects user’s pronunciation, is a promising service for transition to the AmI environment. However, the migration of the CAPT to the AmI environment is challenging due to its high computational requirements that is at odds with the low computational capacity of IoT members. In this paper, we propose a smartphone-assisted pronunciation learning technique based on a lightweight word recommendation method that exploits built-in functions supported by IoT members and a computationally moderate word selection method. The experimental evaluation of the proposed method demonstrates that the user pronunciation is significantly improved without incurring unacceptable computational costs for a smartphone platform.

Teaching Pronunciation with Computer Assisted Pronunciation Instruction in a Technological University

The purpose of this study is to evaluate the effectiveness of computer assisted pronunciation instruction in English pronunciation for students in vocational colleges and universities in Taiwan. The participants were fifty-one first-year undergraduate students from a technological university located in central Taiwan. The participants received an eight-week pronunciation instruction, in which the participants were presented model utterances and corresponding pitch contours of target sentences and then practiced pronunciation of target sentences with computer assisted pronunciation training software. Computerized speaking tests were conducted before, during, and after the training to measure the improvement of pronunciation quality, and a questionnaire was administered at the end of the instruction. The results of repeated measures analysis of variance on the scores of the tests indicate that the pronunciation quality of the participants was significantly improved. The results of this study provide empirical evidence in teaching pronunciation to vocational college students with computer technology.

Large-scale characterization of non-native Mandarin Chinese spoken by speakers of European origin: Analysis on iCALL

In this work, we analyze phonetic and prosodic pronunciation patterns from iCALL, a speech corpus designed to evaluate Mandarin mispronunciations by non-native speakers of European origin and to address the lack of large-scale, non-native corpora with comprehensive annotations for applications in CAPT (computer-assisted pronunciation training). iCALL consists of 90,841 utterances from 305 speakers with a total duration of 142 hours. The speakers are from diverse linguistic backgrounds (spanning Germanic, Romance, and Slavic native languages). The read utterances are phonetically balanced with phonetic, tonal, and fluency annotations. Our findings on iCALL reveal that lexical tone errors are over six times more prevalent than phonetic errors, French speakers are twice as likely to mispronounce Tone 2, 3, 4 when compared to English speakers, native Romance language speakers are more likely to make de-aspiration and aspiration mistakes, and fluency scores correlate inversely with tone and phone error rate.

A Longitudinal View of Look at the Effectiveness of Elicited Imitation with Computer Assisted Pronunciation Training (CAPT)

<p>It is the aim for most EFL learners to have a clear pronunciation in English; however, pronunciation training has always been a challenge for EFL teachers and learners alike. The advent of Computer Assisted Pronunciation Training (CAPT) seems to be the solution to this problem. This longitudinal study adopted a latent growth curve to describe 30 EFL learner’s (N = 30) development of imitated pronunciation in English as well as their satisfaction regarding CAPT. These 30 participants were randomly assigned to the experimental group (N = 18) and control group (N = 12). There were no significant differences among the participants in their pronunciation of English or attitudes toward CAPT at the beginning of this study. After 16 months of training, the growth curve analysis showed that participants in the CAPT group made significantly greater development in English pronunciation, and they were more satisfied with the CAPT. The large effect size indicated that the grouping was the major effect that led to a significant difference in satisfaction. Moreover, the qualitative data derived from focus-group interviews confirmed the benefits of CAPT in participant’s pronunciation training. These findings suggest that EFL learner’s opportunities to conduct elicited imitation with the help of ASR-CAPT will be helpful for their development of imitated pronunciation in English and increase their satisfaction of it, which is a key aspect in the effectiveness of pronunciation training. </p>

Dynamic time warping in phoneme modeling for fast pronunciation error detection

The presented paper describes a novel approach to the detection of pronunciation errors. It makes use of the modeling of well-pronounced and mispronounced phonemes by means of the Dynamic Time Warping (DTW) algorithm. Four approaches that make use of the DTW phoneme modeling were developed to detect pronunciation errors: Variations of the Word Structure (VoWS), Normalized Phoneme Distances Thresholding (NPDT), Furthest Segment Search (FSS) and Normalized Furthest Segment Search (NFSS). The performance evaluation of each module was carried out using a speech database of correctly and incorrectly pronounced words in the Polish language, with up to 10 patterns of every trained word from a set of 12 words having different phonetic structures. The performance of DTW modeling was compared to Hidden Markov Models (HMM) that were used for the same four approaches (VoWS, NPDT, FSS, NFSS). The average error rate (AER) was the lowest for DTW with NPDT (AER=0.287) and scored better than HMM with FSS (AER=0.473), which was the best result for HMM. The DTW modeling was faster than HMM for all four approaches. This technique can be used for computer-assisted pronunciation training systems that can work with a relatively small training speech corpus (less than 20 patterns per word) to support speech therapy at home.

An empirical examination of EFL learners' perceptual learning styles and acceptance of ASR-based computer-assisted pronunciation training

This study aims to explore the structural relationships among the variables of EFL (English as a foreign language) learners' perceptual learning styles and Technology Acceptance Model (TAM). Three hundred and forty-one (n = 341) EFL learners were invited to join a self-regulated English pronunciation training program through automatic speech recognition (ASR) computer system. Participants were asked to actively undertake the interactions with ASR-based computer-assisted pronunciation training (CAPT) on a daily basis for three months. They were directed to finish a questionnaire on their perceptual learning style and technology acceptance. The collected data were analysed with descriptive statistics and structural equation model to investigate the structural relationships. Results show that most participants were visual learners; furthermore, no significant relationship between any type of perceptual learning style and perceived usefulness was discovered. Visual style as well as kinaesthetic style was found to be significantly influential to perceived ease of use which further significantly led to perceived usefulness. Perceived usefulness was significantly related to attitude of using the system that was decisive to EFL learners' continuing use of ASR-based CAPT.

Pronunciation analysis for children with speech sound disorders.

Phonological disorders affect 10% of preschool and school-age children, adversely affecting their communication, academic performance, and interaction level. Effective pronunciation training requires prolonged supervised practice and interaction. Unfortunately, many children do not have access or only limited access to a speech-language pathologist. Computer-assisted pronunciation training has the potential for being a highly effective teaching aid; however, to-date such systems remain incapable of identifying pronunciation errors with sufficient accuracy. In this paper, we propose to improve accuracy by (1) learning acoustic models from a large children's speech database, (2) using an explicit model of typical pronunciation errors of children in the target age range, and (3) explicit modeling of the acoustics of distorted phonemes.

Error Patterns for Automatic Error Detection in Computer Assisted Pronunciation Training Systems

This paper presents error patterns built on the basis of our comparative analysis of American English and Mexican Spanish phonemes and allophones which can be applied in designing the error detection module of a Computer Assisted Pronunciation Training (CAPT) System for teaching American English pronunciation to Mexican Spanish speakers. Error identification is important for an adequate choice of correcting techniques which improves English pronunciation acquisition and helps learners to develop less accented speech. Since automatic individual error detection remains a highly complex computational task, error patterns can enhance the system performance and improve its precision. To the best of our knowledge, error patterns in American English speech generated by Mexican Spanish speakers has not been defined in previous work which was done mainly for Castilian- originated standard Spanish.

Evaluating a computer-assisted pronunciation training (CAPT) technique for efficient classroom instruction

This study investigates a computer-assisted pronunciation training (CAPT) technique that combines oral reading with peer review to improve pronunciation of Taiwanese English major students. In addition to traditional in-class instruction, students were given a short passage every week along with a recording of the respective text, read by a native speaker. They practiced at home by listening to the recordings, reading out loud while listening, recording themselves, and comparing their recordings to the native speaker. When satisfied, they posted their own recording to an online discussion board. Every student listened to the recordings of three classmates, supplying feedback through the discussion board. Two control groups received only in-class instruction without the recording and posting assignments. Results show that the integration of the CAPT technique was superior in reducing students’ pronunciation problems compared to only in-class instruction.