A Mandarin edutainment system integrated virtual learning environments

Abstract

In this paper, a novel Mandarin edutainment system is developed for learning Mandarin in immersing, interactive virtual learning environments (VLE). Our system is mainly comprised of two parts: speech technology support and virtual 3D game design. First, 3D face recognition technology is introduced to discriminate the different learners and provide the personalized learning services based on the characteristics of the individuals. Then, a Mandarin pronunciation recognition and assessment scheme is constructed by state-of-the-art speech processing technology. According to the distinctive differences of Mandarin rhythm from the Western languages, we integrate the prosodic parameters into the recognition and evaluation model to highlight Mandarin characteristics and improve the evaluation performance. In order to promote the engagement of foreign learners, we embed our technology framework into a Virtual Reality (VR) game environment. The character design reflects the Chinese traditional culture, and the plots effectively give consider to learning pronunciation and learners’ interest, providing the scoring feedback simultaneously. In the experimental design, first, we test the correlation of recognition results and machine scores with the different errors and human scores. Then, we evaluate the usability, likeability, and knowledgeability of the whole VLE system. We divide the learners into three categories in terms of their Mandarin levels, and they provide feedback via a questionnaire. The results show that our system can effectively promote the foreign learners’ engagement and improve their Mandarin level.