A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

Abstract

Virtual experiments have become an interesting research topic in the field of education. However, we found that there are some limitations in the current virtual experiments: first, the researchers used the virtual effects of the simulation to represent the virtual experiments, which led to decrease the immersion of the user's simulated experiments; second, most of the virtual experiments are only mouse or touch screen interactive mode, which reduces the realism of user simulation experiments; third, students independently explore the experimental operation process and spend too much time simulating the experiment, which leads to problems such as overloading the operation and low interaction efficiency. In order to solve the above problems, we propose and implement a multimodal navigational interaction virtual and real fusion chemistry laboratory (MNIVRFCL). We design a new sensing structure intelligent equipment and propose a multimodal navigational interaction algorithm (MMNI) based on auditory and tactile channel, which are verified and applied in MNIVRFCL. The MMNI algorithm can detect user's specific behaviors to understand their behavioral intentions, and then the system guide and rectify users' current correct or incorrect behaviors in the form of voice navigation broadcasts. Finally, we achieve the purpose that students can use virtual and real fusion interactions through tactile and auditory channels, they can independently complete simulations and learning experiments based on experimental navigation. The experimental statistic results show that system's successful understanding of intention is 91.48%, and prove the MNIVRFCL operational load reduce by 23.81% compared to the pure virtual experiment, it can reduce the time consumption and improves the students' interaction efficiency.