Leveraging audible and inaudible signals for pronunciation training by sensing articulation through a smartphone

Abstract

• Under pronunciation principles, this paper presents a smartphone-based pronunciation training system for practicing monophthong regarding vowel articulation. • Speech and ultrasonic signals are captured and analyzed simultaneously to determine the tongue position and shape of the lips respecting uttering vowels. • The proposed system implementing a commercial smartphone is robust with high accuracy for practical scenarios. • The evaluation for its impact on user experience shows that the proposed system has raised learning engagement and enjoyment. Learning a foreign language pronunciation is the most challenging task for non-native speakers. Improving pronunciation based on feedback on pronunciation error scores is also not easy for learners. Our goal is to develop an alternative approach to pronunciation training that can point out articulation errors at the phoneme level to users through a smartphone-based system. Using self-correction as a means of ensuring learners' engagement, we focus on self-correction of pronunciation by adjusting articulation. In order to identify articulation concerning pronunciation, the system evaluates both audible and inaudible acoustic signals to examine fine-grained frequency-shifting direction of mouth movements and tongue position simultaneously. The result shows that the system can reach an average accuracy of 99.09% and is robust in different scenarios and genders. Additionally, the evaluation of the user study reveals that the proposed system provides positive user experiences and allows learners to improve their pronunciation more efficiently.