Abstract
Students will experience a complex mixture of mental states during discussion, including concentration, confusion, frustration, and boredom, which have been widely acknowledged as crucial components for revealing a student's learning states. In this study, we propose using multimodal data to design an intelligent monitoring agent that can assist teachers in effectively monitoring the multiple mental states of students during discussion. We firstly developed an advanced multi-sensor-based system and applied it in a real university's research lab to collect a multimodal “in-the-wild” teacher-student conversation dataset. Then, we derived a set of proxy features from facial, heart rate, and acoustic modalities and used them to train several supervised learning classifiers with different multimodal fusion approaches single-channel-level, feature-level, and decision-level fusion to recognize students' multiple mental states in conversations. We explored how to design multimodal analytics to augment the ability to recognize different mental states and found that fusing heart rate and acoustic modalities yields better recognize the states of concentration (AUC = 0.842) and confusion (AUC = 0.695), while fusing three modalities yield the best performance in recognizing the states of frustration (AUC = 0.737) and boredom (AUC = 0.810). Our results also explored the possibility of leveraging the advantages of the replacement capabilities between different modalities to provide human teachers with solutions for addressing the challenges with monitoring students in different real-world education environments.
Highlights
Conversation-based discussion is one form of typical complex learning activities held in higher education today in which students are required to complete a series of complex learning tasks including answer questions, generate explanations, express opinions, and transfer acquired knowledge
The random forest (RF) classifier based on the heart rate (HR) modality achieved a mean area under curve (AUC) of 0.704, which was slighter better than the multi-layer perceptron (MLP) classifier with a mean AUC of 0.690
The RF classifier showed outstanding performance in using acoustic cues to recognize all of the mental state classes, which yielded a mean AUC of 0.728, stronger than the support vector machine (SVM) classifier with a mean AUC of 0.694 and better than the MLP classifier with a mean AUC score of 0.721; the first 20 top-ranked acoustic features we used to achieve the best performance
Summary
Conversation-based discussion is one form of typical complex learning activities held in higher education today in which students are required to complete a series of complex learning tasks including answer questions, generate explanations, express opinions, and transfer acquired knowledge. A broad range of remarkable research has validated the idea that students’ may consistently experience a mixture of multiple mental states, such as concentration/engagement, anxiety, delight, satisfaction, confusion, frustration, boredom etc., in complex cognitive learning [1]–[6]. D’Mello and Graesser [9] explored the dynamic changes in a student’s learning-centered mental states, concentration, confusion, frustration, and boredom, when they complete complex learning activities such as conversation-based discussion. They suggest that a student commonly enters learning activities with a state of engaged concentration, and this state will remain until they reach a difficult impasse, which may result in their state transitioning to confusion. One is that they go back to being engaged if the impasse has been resolved, which can be due to positive accomplishments brought about by solving problems
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
