Multimodal Learning Analytics Research Articles

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA.

Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

Designing Analytics for Collaboration Literacy and Student Empowerment

Collaboration has garnered global attention as an important skill for the 21st century. While researchers have been doing work on collaboration for nearly a century, many of the questions that the field is investigating overlook the need for students to learn how to read and respond to different collaborative settings. Existing research focuses on chronicling the various factors that predict the effectiveness of a collaborative experience, or on changing user behaviour in the moment. These are worthwhile research endeavours for developing our theoretical understanding of collaboration. However, there is also a need to centre student perceptions and experiences with collaboration as an important area of inquiry. Based on a survey of 131 university students, we find that student collaboration-related concerns can be represented across seven different categories or dimensions: Climate, Compatibility, Communication, Conflict, Context, Contribution, and Constructive. These categories extend prior research on collaboration and can help the field ensure that future collaboration analytics tools are designed to support the ways that students think about and utilize collaboration. Finally, we describe our instantiation of many of these dimensions in our collaborative analytics tool, BLINC, and suggest that these seven dimensions can be instructive for re-orienting the Multimodal Learning Analytics (MMLA) and collaboration analytics communities.

Challenges and opportunities of multimodal data in human learning: The computer science students' perspective

AbstractMultimodal data have the potential to explore emerging learning practices that extend human cognitive capacities. A critical issue stretching in many multimodal learning analytics (MLA) systems and studies is the current focus aimed at supporting researchers to model learner behaviours, rather than directly supporting learners. Moreover, many MLA systems are designed and deployed without learners' involvement. We argue that in order to create MLA interfaces that directly support learning, we need to gain an expanded understanding of how multimodal data can support learners' authentic needs. We present a qualitative study in which 40 computer science students were tracked in an authentic learning activity using wearable and static sensors. Our findings outline learners' curated representations about multimodal data and the non‐technical challenges in using these data in their learning practice. The paper discusses 10 dimensions that can serve as guidelines for researchers and designers to create effective and ethically aware student‐facing MLA innovations.

Detecting Drowsy Learners at the Wheel of e-Learning Platforms With Multimodal Learning Analytics

Learners are expected to stay wakeful and focused while interacting with e-learning platforms. Although wakefulness of learners strongly relates to educational outcomes, detecting drowsy learning behaviors only from log data is not an easy task. In this study, we describe the results of our research to model learners’ wakefulness based on multimodal data generated from heart rate, seat pressure, and face recognition. We collected multimodal data from learners in a blended course of informatics and conducted two types of analysis on them. First, we clustered features based on learners’ wakefulness labels as generated by human raters and ran a statistical analysis. This analysis helped us generate insights from multimodal data that can be used to inform learner and teacher feedback in multimodal learning analytics. Second, we trained machine learning models with multiclass-Support Vector Machine (SVM), Random Forest (RF) and CatBoost Classifier (CatBoost) algorithms to recognize learners’ wakefulness states automatically. We achieved an average macro-F1 score of 0.82 in automated user-dependent models with CatBoost. We also showed that compared to unimodal data from each sensor, the multimodal sensor data can improve the accuracy of models predicting the wakefulness states of learners while they are interacting with e-learning platforms.

Towards Collaborative and Intelligent Learning Environments Based on Eye Tracking Data and Learning Analytics: A Survey

The current pandemic has significantly impacted educational practices, modifying many aspects of how and when we learn. In particular, remote learning and the use of digital platforms have greatly increased in importance. Online teaching and e-learning provide many benefits for information retention and schedule flexibility in our on-demand world while breaking down barriers caused by geographic location, physical facilities, transportation issues, or physical impediments. However, educators and researchers have noticed that students face a learning and performance decline as a result of this sudden shift to online teaching and e-learning from classrooms around the world. In this paper, we focus on reviewing eye-tracking techniques and systems, data collection and management methods, datasets, and multi-modal learning data analytics for promoting pervasive and proactive learning in educational environments. We then describe and discuss the crucial challenges and open issues of current learning environments and data learning methods. The review and discussion show the potential of transforming traditional ways of teaching and learning in the classroom, and the feasibility of adaptively driving learning processes using eye-tracking, data science, multimodal learning analytics, and artificial intelligence. These findings call for further attention and research on collaborative and intelligent learning systems, plug-and-play devices and software modules, data science, and learning analytics methods for promoting the evolution of face-to-face learning and e-learning environments and enhancing student collaboration, engagement, and success.

Multimodal Learning Analytics to Inform Learning Design: Lessons Learned from Computing Education

Programming is a complex learning activity that involves coordination of cognitive processes and affective states. These aspects are often considered individually in computing education research, demonstrating limited understanding of how and when students learn best. This issue confines researchers to contextualize evidence-driven outcomes when learning behaviour deviates from pedagogical intentions. Multimodal learning analytics (MMLA) captures data essential for measuring constructs (e.g., cognitive load, confusion) that are posited in the learning sciences as important for learning, and cannot effectively be measured solely with the use of programming process data (IDE-log data). Thus, we augmented IDE-log data with physiological data (e.g., gaze data) and participants’ facial expressions, collected during a debugging learning activity. The findings emphasize the need for learning analytics that are consequential for learning, rather than easy and convenient to collect. In that regard, our paper aims to provoke productive reflections and conversations about the potential of MMLA to expand and advance the synergy of learning analytics and learning design among the community of educators from a post-evaluation design-aware process to a permanent monitoring process of adaptation.

Multimodal Data Fusion in Learning Analytics: A Systematic Review.

Multimodal learning analytics (MMLA), which has become increasingly popular, can help provide an accurate understanding of learning processes. However, it is still unclear how multimodal data is integrated into MMLA. By following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, this paper systematically surveys 346 articles on MMLA published during the past three years. For this purpose, we first present a conceptual model for reviewing these articles from three dimensions: data types, learning indicators, and data fusion. Based on this model, we then answer the following questions: 1. What types of data and learning indicators are used in MMLA, together with their relationships; and 2. What are the classifications of the data fusion methods in MMLA. Finally, we point out the key stages in data fusion and the future research direction in MMLA. Our main findings from this review are (a) The data in MMLA are classified into digital data, physical data, physiological data, psychometric data, and environment data; (b) The learning indicators are behavior, cognition, emotion, collaboration, and engagement; (c) The relationships between multimodal data and learning indicators are one-to-one, one-to-any, and many-to-one. The complex relationships between multimodal data and learning indicators are the key for data fusion; (d) The main data fusion methods in MMLA are many-to-one, many-to-many and multiple validations among multimodal data; and (e) Multimodal data fusion can be characterized by the multimodality of data, multi-dimension of indicators, and diversity of methods.

A Multimodal Real-Time Feedback Platform Based on Spoken Interactions for Remote Active Learning Support.

While technology has helped improve process efficiency in several domains, it still has an outstanding debt to education. In this article, we introduce NAIRA, a Multimodal Learning Analytics platform that provides Real-Time Feedback to foster collaborative learning activities’ efficiency. NAIRA provides real-time visualizations for students’ verbal interactions when working in groups, allowing teachers to perform precise interventions to ensure learning activities’ correct execution. We present a case study with 24 undergraduate subjects performing a remote collaborative learning activity based on the Jigsaw learning technique within the COVID-19 pandemic context. The main goals of the study are (1) to qualitatively describe how the teacher used NAIRA’s visualizations to perform interventions and (2) to identify quantitative differences in the number and time between students’ spoken interactions among two different stages of the activity, one of them supported by NAIRA’s visualizations. The case study showed that NAIRA allowed the teacher to monitor and facilitate the learning activity’s supervised stage execution, even in a remote learning context, with students working in separate virtual classrooms with their video cameras off. The quantitative comparison of spoken interactions suggests the existence of differences in the distribution between the monitored and unmonitored stages of the activity, with a more homogeneous speaking time distribution in the NAIRA supported stage.

The Future of Embodied Design for Mathematics Teaching and Learning

A rising epistemological paradigm in the cognitive sciences—embodied cognition—has been stimulating innovative approaches, among educational researchers, to the design and analysis of STEM teaching and learning. The paradigm promotes theorizations of cognitive activity as grounded, or even constituted, in goal-oriented multimodal sensorimotor phenomenology. Conceptual learning, per these theories, could emanate from, or be triggered by, experiences of enacting or witnessing particular movement forms, even before these movements are explicitly signified as illustrating target content. Putting these theories to practice, new types of learning environments are being explored that utilize interactive technologies to initially foster student enactment of conceptually oriented movement forms and only then formalize these gestures and actions in disciplinary formats and language. In turn, new research instruments, such as multimodal learning analytics, now enable researchers to aggregate, integrate, model, and represent students’ physical movements, eye-gaze paths, and verbal–gestural utterance so as to track and evaluate emerging conceptual capacity. We—a cohort of cognitive scientists and design-based researchers of embodied mathematics—survey a set of empirically validated frameworks and principles for enhancing mathematics teaching and learning as dialogic multimodal activity, and we synthetize a set of principles for educational practice.

The promise and challenges of multimodal learning analytics

The promise and challenges of multimodal learning analytics

Enhancing consent forms to support participant decision making in multimodal learning data research

AbstractAdvances in the field of multimodal learning analytics (MMLA) research is often accomplished by actively exploring new technologies and techniques related to the collection and analysis of data. Exploration of ethical principles and procedures for governing the use of new technologies and techniques, however, is not as readily pursued. As collected data grow in complexity and invasiveness, potentially, a growing need is arising to scrutinize ethical aspects of MMLA research. In our study, we introduce an informed consent comprehension test for educational technology research and assess the effects of enhancing MMLA consent forms on comprehension of informed consent and on rates of enrollment in a MMLA study. One form is written from a researcher perspective and the other from a participant perspective. Results of the study involving first‐year undergraduate students suggest that the overall level of comprehension did not differ between conditions. Yet, the participant‐oriented consent form resulted in significantly lower rates of enrollment. Implications for MMLA researchers are discussed.

Multimodal learning analytics for game‐based learning

AbstractA distinctive feature of game‐based learning environments is their capacity to create learning experiences that are both effective and engaging. Recent advances in sensor‐based technologies such as facial expression analysis and gaze tracking have introduced the opportunity to leverage multimodal data streams for learning analytics. Learning analytics informed by multimodal data captured during students’ interactions with game‐based learning environments hold significant promise for developing a deeper understanding of game‐based learning, designing game‐based learning environments to detect maladaptive behaviors and informing adaptive scaffolding to support individualized learning. This paper introduces a multimodal learning analytics approach that incorporates student gameplay, eye tracking and facial expression data to predict student posttest performance and interest after interacting with a game‐based learning environment, Crystal Island. We investigated the degree to which separate and combined modalities (ie, gameplay, facial expressions of emotions and eye gaze) captured from students (n = 65) were predictive of student posttest performance and interest after interacting with Crystal Island. Results indicate that when predicting student posttest performance and interest, models utilizing multimodal data either perform equally well or outperform models utilizing unimodal data. We discuss the synergistic effects of combining modalities for predicting both student interest and posttest performance. The findings suggest that multimodal learning analytics can accurately predict students’ posttest performance and interest during game‐based learning and hold significant potential for guiding real‐time adaptive scaffolding.

Round or rectangular tables for collaborative problem solving? A multimodal learning analytics study

AbstractThe current knowledge of the effects of the physical environment on learners’ behaviour in collaborative problem‐solving tasks is underexplored. This paper aims to critically examine the potential of multimodal learning analytics, using new data sets, in studying how the shapes of shared tables affect the learners’ behaviour when collaborating in terms of patterns of participation and indicators related to physical social interactions. The research presented in this paper investigates this question considering the potential interplay with contextual aspects (level of education) and learning design decisions (group size). Three dependent variables (distance between students, range of movement and level of participation) are tested using quantitative and qualitative analyses of data collected using a motion capture system and video recordings. Results show that the use of round tables (vs rectangular tables) leads to higher levels of on‐task participation in the case of elementary school students. For university students, different table shapes seem to have a limited impact on their levels of participation in collaborative problem solving. The analysis shows significant differences regarding the relationship between group size and the distance between students, but there is no substantial evidence that group size affects the level of participation. The findings support previous research highlighting the importance of studying the role of the physical environment as an element of learning design and the potential of multimodal learning analytics in approaching these studies.

Data Capture and Multimodal Learning Analytics Focused on Engagement With a New Wearable IoT Approach

Increasing school dropout rates are a problem in many educational systems, with student disengagement being one significant factor. Learning analytics is a new field with a key role in educational institutions in the coming years. It may help make strategic decisions to reduce student disengagement. The use of technology in educational environments has grown significantly and, with it, awareness of the importance of student engagement. We exploit tracking and wearable technologies to increase user engagement in learning processes, exploring also the area of multimodal learning analytics (MMLA). We use wearables and Internet of Things for education, an interactive and collaborative system designed to improve motivation and learning. This article presents the results obtained in different experiments conducted in a secondary school in a long-term participatory learning context. The captured data were analyzed and used to identify different students’ behavior patterns, showing their progress and motivation. Subsequently, from the captured data and aiming at a decision-making phase, we used machine learning techniques and MMLA methodologies to construct models able to “explain” when student engagement is present, so this knowledge can later be exploited. In particular, we chose decision trees and rule systems based on a set of variables with proven relevance to the problem. The evaluation of this novel engagement classification system confirms the high performance of these variables. The rules obtained, which can be easily interpreted by a nonexpert, help the teacher to observe, analyze, and make decisions with the purpose of fostering engagement.

Multimodal Learning Analytics research with young children: A systematic review

AbstractLearning Analytics and Multimodal Learning Analytics are changing the way of analysing the learning process while students interact with an educational content. This paper presents a systematic literature review aimed at describing practices in recent Multimodal Learning Analytics and Learning Analytics research literature in order to identify tools and strategies useful for the assessment of the progress and behaviour of children under 6 years old in respect of their learning. The purpose is to provide guidance for Multimodal Learning Analytics research with children under 6 years old to assess their engagement in a task, their emotions, attention, understanding and achievement of a goal. The current state of knowledge on Multimodal Learning Analytics research suggests how performance analytics, face and speech recognition systems, eye tracking, Kinect analytics and wristbands could be used with children. The results show the complexity of collecting data using non‐invasive methodologies with children under 6 years old. Ethical implications related to multimodal data from audio, visual, biometric and quantitative measures of child behaviour are discussed.

Multimodal learning analytics to investigate cognitive load during online problem solving

AbstractTo have insight into cognitive load (CL) during online complex problem solving, this study aimed at measuring CL through physiological data. This study experimentally manipulated intrinsic and extraneous load of exercises in the domain of statistics, resulting in four conditions: high complex with hints, low complex with hints, high complex without hints and low complex without hints. The study had a within‐subject‐design in which 67 students solved the exercises in a randomized order. Self‐reported CL was combined with physiological data, namely, galvanic skin response (GSR), skin temperature (ST), heart rate (HR) and heart rate variability (HRV). Multiple imputation was used for handling missing data from resp. 16 and 19 students for GSR/ST and HR/HRV. First, differences between conditions in view of physiological data were examined. Second, we investigated how much variance of self‐reported CL and task performance was explained by physiological data. Finally, we investigated which features can be used to assess (objective) CL. Results revealed no significant differences between the manipulated conditions in terms of physiological data. Nonetheless, HR and ST were significantly related to self‐reported CL, whereas ST to task performance. Additionally, this study revealed the potential of ST and HR to assess high CL.

A Scalable Architecture for the Dynamic Deployment of Multimodal Learning Analytics Applications in Smart Classrooms.

The smart classrooms of the future will use different software, devices and wearables as an integral part of the learning process. These educational applications generate a large amount of data from different sources. The area of Multimodal Learning Analytics (MMLA) explores the affordances of processing these heterogeneous data to understand and improve both learning and the context where it occurs. However, a review of different MMLA studies highlighted that ad-hoc and rigid architectures cannot be scaled up to real contexts. In this work, we propose a novel MMLA architecture that builds on software-defined networks and network function virtualization principles. We exemplify how this architecture can solve some of the detected challenges to deploy, dismantle and reconfigure the MMLA applications in a scalable way. Additionally, through some experiments, we demonstrate the feasibility and performance of our architecture when different classroom devices are reconfigured with diverse learning tools. These findings and the proposed architecture can be useful for other researchers in the area of MMLA and educational technologies envisioning the future of smart classrooms. Future work should aim to deploy this architecture in real educational scenarios with MMLA applications.

Contextualising Learning Analytics with Classroom Observations: A Case study

Educational processes take place in physical and digital places. To analyse educational processes, Learning Analytics (LA) enable data collection from the digital learning context. At the same time, to gain more insights, the LA data can be complemented with the data coming from physical spaces enabling Multimodal Learning Analytics (MMLA). To interpret this data, theoretical grounding or contextual information is needed. Learning designs (LDs) can be used for contextualisation, however, in authentic scenarios the availability of machine-readable LD is scarce. We argue that Classroom Observations (COs), traditionally used to understand educational processes taking place in physical space, can provide the missing context and complement the data from the co-located classrooms. This paper reports on a co-design case study from an authentic scenario that used CO to make sense of the digital traces. In this paper we posit that the development of MMLA approaches can benefit from co-design methodologies; through the involvement of the end-users (project managers) in the loop, we illustrate how these data sources can be systematically integrated and analysed to better understand the use of digital resources. Results indicate that CO can drive sense-making of LA data where predefined LD is not available. Furthermore, CO can support layered contextualisation depending on research design, rigour and systematic documentation/data collection efforts.Also, co-designing the MMLA solution with the end-users proved to be a useful approach.

Multimodal Data Value Chain (M-DVC): A Conceptual Tool to Support the Development of Multimodal Learning Analytics Solutions

Multimodal Learning Analytics (MMLA) systems, understood as those that exploit multimodal evidence of learning to better model a learning situation, have not yet spread widely in educational practice. Their inherent technical complexity, and the lack of educational stakeholder involvement in their design, are among the hypothesized reasons for the slow uptake of this emergent field. To aid in the process of stakeholder communication and systematization leading to the specification of MMLA systems, this paper proposes a Multimodal Data Value Chain (M-DVC). This conceptual tool, derived from both the field of Big Data and the needs of MMLA scenarios, has been evaluated in terms of its usefulness for stakeholders, in three authentic case studies of MMLA systems currently under development. The results of our mixed-methods evaluation highlight the usefulness of the M-DVC to elicit unspoken assumptions or unclear data processing steps in the initial stages of development. The evaluation also revealed limitations of the M-DVC in terms of the technical terminology employed, and the need for more detailed contextual information to be included. These limitations also prompt potential improvements for the M-DVC, on the path towards clearer specification and communication within the multi-disciplinary teams needed to build educationally-meaningful MMLA solutions.

Lowering Barriers for Accessing Sensor Data in Education: Lessons Learned from Teaching Multimodal Learning Analytics to Educators

In an increasingly data-driven world, large volumes of fine-grained data are infiltrating all aspects of our lives. The world of education is no exception to this phenomenon: in classrooms, we are witnessing an increasing amount of information being collected on learners and teachers. Because educational practitioners have so much contextual and practical knowledge about classroom management, we argue that data-mining workflows should be co-designed with them. This paper describes a class on Multimodal Learning Analytics taught to graduate students in education who used to be (or are planning to become) educators, teachers, school administrators and have little to no technical background. The course was designed to provide novices with career-relevant hands-on activities and facilitate personal engagement with data collection and analysis. We provide examples of student-created data mining workflows and the trajectory they followed to get a foundational understanding of data mining. Finally, we present survey data illustrating the strengths and weaknesses of the assignments and projects used in the class. We conclude with lessons learned and recommendations for implementing such a course at other institutions.