How Prior Knowledge Affects Visual Attention of Japanese Mimicry and Onomatopoeia and Learning Outcomes: Evidence from Virtual Reality Eye Tracking

Abstract

According to the United Nations Sustainable Development Goal (SDG) 4, “achieving inclusive and quality education for all”, foreign language learning has come to be seen as a process of integrating sustainable development into the socio-cultural aspects of education and learning. The aim of this study was to employ virtual reality (VR) eye tracker to examine how students with different levels of prior knowledge process visual behaviors for Japanese Mimicry and Onomatopoeia (MIO) while learning Japanese as a second foreign language. A total of 20 students studying at the Department of Applied Japanese at the university of Southern Taiwan were recruited. Based on the Japanese language proficiency test (JLPT) level, 20 participants were divided into high prior knowledge group (levels N1–N3) with 7 participants, and low prior knowledge group (level N4 or below) with 13 participants. The learning stimuli materials were created by Unreal Engine 4 (UE4) development tool to design a 3D virtual MIO paradise, including 5 theme amusement parks. Through a VR eye tracker, participants’ visual behaviors were tracked and recorded based on 24 different regions of interest (ROIs) (i.e., ROI1–ROI24). This was done to discuss the distribution of visual attention in terms of different ROIs of each theme amusement park based on four eye movement indicators, including latency of first fixation (LFF), duration of first fixation (DFF), total fixation durations (TFD), and fixation counts (FC). Each ROI of the two groups were then compared. In addition, a heat zone map was also generated to show the overall visual distribution of each group. After the experiment, based on the eye movement indicators and test scores in the pre-test and post-test phases, statistical analysis was used to examine and evaluate the differences in visual attention and learning outcomes. The results revealed that the gaze sequences of the two prior knowledge groups gazing at the ROIs in theme parks were different, except for the gaze sequence in the circus theme park. Different prior knowledge groups exhibited differences in visual attention in the ROIs fixated on in each amusement park. Additionally, in terms of TFD and FC of different groups in each amusement park, there was no significant difference except in ROI10, ROI16, and ROI18. Moreover, after receiving cognitive comprehension processes introduced in the VR-simulated MIO scenes, students from both groups achieved higher post-test scores compared with pre-test scores, and such differences had statistical significance. In conclusion, the implications of VR eye movement analysis on developing students’ competence related to learning Japanese and cross-cultural aspects, compatible with sustainable development, were presented.