Engagement detection and enhancement for STEM education through computer vision, augmented reality, and haptics

Abstract

STEM (Science, technology, engineering, and mathematics) based education is being revolutionized with the rapid development of the applications of intelligent Haptics. Theoretical knowledge may now be put into practice, and complex ideas can now be presented using three-dimensional models that are enhanced in the actual environment via Augmented Reality (AR) and computer Haptics to enhance the student's interest in concepts and studies. This article presents a novel design and development of an AR and Haptics-based STEM product comprising of World Map with Haptic feedback using Vuforia, Unity 3D, and Open-Haptics to enhance student engagement. To detect students' engagement in online learning, a computer vision-based system is also designed and deployed in the real-time website that uses the features, such as facial emotions, pose estimation, and head rotation. Marker-based augmentation is employed, and a sense of touch is included for an immersive experience for the students with the use of the haptic Touch Omni device. Hence, the proposed approach can improve classroom learning activities by using Augmented reality and Haptics-based STEM product and by its evaluation using a computer vision system. To validate the improvement in engagement, a comprehensive user study is performed and analyzed on the proposed setup.