Facial Emotion Detection to Assess Learner's State of Mind in an Online Learning System

Abstract

Despite the success and the popularity of the online learning system, it still lacks in dynamically adapting suitable pedagogical methods according to the changing emotions and behaviour of the learner, as can be done in the face-to-face mode of learning. This makes the learning process mechanized, which significantly affects the learning outcome. To resolve this, the first and necessary step is to assess the emotion of a learner and identify the change of emotions during a learning session. Usually, images of facial expressions are analysed to assess one's state of mind. However, human emotions are far more complex, and these psychological states may not be reflected only through the basic emotion of a learner (i.e. analysing a single image), but a combination of two or more emotions which may be reflected on the face over a period of time. From a real survey, we derived four complex emotions that are a combination of basic human emotions often experienced by a learner, in concert, during a learning session. To capture these combined emotions correctly, we considered a fixed set of continuous image frames, instead of discrete images. We built a CNN model to classify the basic emotions and then identify the states of mind of the learners. The outcome is verified mathematically as well as surveying the learners. The results show a 65% and 62% accuracy respectively, for emotion classification and state of mind identification.