Spontaneous facial expression database of learners’ academic emotions in online learning with hand occlusion

Abstract

Academic emotions refer to various emotional experiences in connection with learners’ academic activities while learning, and these emotions are vital to the development of learners’ physiology and mentality. Facial expression recognition (FER) technology has been widely used in online learning to identify learners’ academic emotions. However, learners often inadvertently cover part of their face with their hands during online learning, which affects the accuracy of the technology's recognition of academic emotions. Most existing databases lack facial expression data with hand occlusion, which makes it difficult for researchers to further improve recognition accuracy. Therefore, this research established an online learners’ facial expression database with hand occlusion (OLFED-HO) to solve the above problem. This database has a total of 92,947 facial expression images of online learners, including four different hand occlusion situations (no occlusion, left occlusion, middle occlusion, and right occlusion) and seven academic emotions (confusion, curiosity, distraction, enjoyment, fatigue, depression, and neutrality). Then, to indicate the high reliability of the database established in this study, we analyzed the confusion matrix and concluded that the expression labels marked by different external coders have a high internal consistency. The database is expected to further promote the application of expression recognition technology in the field of education and provide online learners’ facial expressions with hand occlusion for the academic emotion database. In addition, an automatic facial expression recognition method with transfer learning based on region attention networks (RAN) is proposed in this paper, which efficiently reduces the impact of hand occlusion. The proposed architecture achieves an accuracy of 89% on the test set of our database.