A momentary view of engagement in collaborative learning: Triangulation through multimodal data

Over the last decade, there has been a renewed interest in capturing twenty-first century skills using new data collection tools. In this article, we leverage an existing dataset where electrodermal activity (EDA) was used to identify markers of productive collaboration. The data came from 42 pairs of participants (N = 84) who had no coding experience and were asked to program a robot to solve a variety of mazes. Because little is known on how physiological synchrony relates to collaborative learning, we explored four different measures of synchrony: Signal Matching (SM), Instantaneous Derivative Matching (IDM), Directional Agreement (DA) and Pearson’s Correlation (PC). Overall, we found PC to be positively associated with learning gains (r = 0.35) and DA with collaboration quality (r = 0.3). To gain further insights into these results, we also qualitatively analyzed two groups and identified situations with high or low physiological synchrony. We observed higher synchrony values when members of a productive group reacted to an external event (e.g., following instructions, receiving a hint), oscillations when they were watching a video or interacting with each other, and lower values when they were programming and / or seem to be confused. Based on these results, we developed a new measure of collaboration using electrodermal data: we computed the number of cycles between low and high synchronization. We found this measure to be significantly correlated with collaboration quality (r = 0.57) and learning gains (r = 0.47). This measure was not significantly correlated with the measures of physiological synchrony mentioned above, suggesting that it is capturing a different construct. We compare those results with prior studies and discuss implications for measuring collaborative process through physiological sensors.

Monitoring in collaborative learning: Co-occurrence of observed behavior and physiological synchrony explored

The coordination of cognitive and non-cognitive interactive processes contributes to successful collaboration in groups, but it is hard to evidence in computer-supported collaborative learning (CSCL). Monitoring is a metacognitive process that can be an indicator of a student’s ability to recognize success or failure in collaboration. This study focuses on how monitoring occurs in CSCL during a collaborative exam situation by examining how individual student contributions to monitoring processes are related to physiological synchrony and physiological arousal in groups. The participants were organized in four groups of three members each, and they wore sensors that measured their physiological activity. The data consist of video recordings from collaborative exam sessions lasting 90 minutes and physiological data captured from each student with Empatica 4.0 sensors. The video data were analyzed using qualitative content analysis to identify monitoring events. Students’ physiological arousal was determined through peak detection, and physiological concordance was used as an index for the students’ physiological synchrony. The individual and group level analysis investigated arousal and physiological synchrony in concordance with monitoring during the collaborative exam. The results showed that, in each group, each student contributed to joint monitoring. In addition, the monitoring activities exhibited a significant correlation with the arousal, indicating that monitoring events are reflected in physiological arousal. Physiological synchrony occurred within two groups, which experienced difficulties during the collaborative exam, whereas the two groups who had no physiological synchrony did not experience difficulties. It is concluded that physiological synchrony may be a new indicator for recognizing meaningful events in CSCL

Examining shared monitoring in collaborative learning: A case of a recurrence quantification analysis approach

Interpersonal physiological synchrony has been consistently found during collaborative tasks. However, few studies have applied synchrony to predict collaborative learning quality in real classroom. To explore the relationship between interpersonal physiological synchrony and collaborative learning activities, this study collected electrodermal activity (EDA) and heart rate (HR) during naturalistic class sessions and compared the physiological synchrony between independent task and group discussion task. The students were recruited from a renowned university in China. Since each student learn differently and not everyone prefers collaborative learning, participants were sorted into collaboration and independent dyads based on their collaborative behaviors before data analysis. The result showed that, during group discussions, high collaboration pairs produced significantly higher synchrony than low collaboration dyads (p = 0.010). Given the equivalent engagement level during independent and collaborative tasks, the difference of physiological synchrony between high and low collaboration dyads was triggered by collaboration quality. Building upon this result, the classification analysis was conducted, indicating that EDA synchrony can identify different levels of collaboration quality (AUC = 0.767 and p = 0.015).

Interpersonal physiological synchrony (PS), or the similarity of physiological signals between individuals over time, may be used to detect attentionally engaging moments in time. We here investigated whether PS in the electroencephalogram (EEG), electrodermal activity (EDA), heart rate and a multimodal metric signals the occurrence of attentionally relevant events in time in two groups of participants. Both groups were presented with the same auditory stimulus, but were instructed to attend either to the narrative of an audiobook (audiobook-attending: AA group) or to interspersed emotional sounds and beeps (stimulus-attending: SA group). We hypothesized that emotional sounds could be detected in both groups as they are expected to draw attention involuntarily, in a bottom-up fashion. Indeed, we found this to be the case for PS in EDA or the multimodal metric. Beeps, that are expected to be only relevant due to specific “top-down” attentional instructions, could indeed only be detected using PS among SA participants, for EDA, EEG and the multimodal metric. We further hypothesized that moments in the audiobook accompanied by high PS in either EEG, EDA, heart rate or the multimodal metric for AA participants would be rated as more engaging by an independent group of participants compared to moments corresponding to low PS. This hypothesis was not supported. Our results show that PS can support the detection of attentionally engaging events over time. Currently, the relation between PS and engagement is only established for well-defined, interspersed stimuli, whereas the relation between PS and a more abstract self-reported metric of engagement over time has not been established. As the relation between PS and engagement is dependent on event type and physiological measure, we suggest to choose a measure matching with the stimulus of interest. When the stimulus type is unknown, a multimodal metric is most robust.

Is an audience captured by a speech or lecture? At what times especially? Do different groups in an audience experience the same speech in different ways? Insight into attentional engagement of individuals can be valuable but difficult to quantify using self-report. Physiological synchrony, the degree to which physiological measurements such as electrodermal activity (EDA) of multiple people uniformly change, has been shown to covary with attentional engagement in lab settings. In this study, we moved out of the lab and monitored EDA of 30 individuals attending a real-life inaugural lecture. These individuals were labeled as belonging to either the personal or professional group, based on their relation with the speaker. We expected these groups to differ in their attentional engagement. We computed physiological synchrony between the participants and investigated how well this metric distinguished between the professional and personal groups, how well it marked predefined engaging events in the lecture, and its relation with levels of engagement as self-reported afterwards. Where possible, we compared physiological synchrony results to results based on individuals' EDA. We found that physiological synchrony in EDA can distinguish between the two groups. Individuals' EDA can also distinguish between the groups, if the occurrence and timing is known of an event that is expected to elicit different levels of engagement for the two groups. We further found that both synchrony and individuals' EDA measures mark predefined engaging events with above-chance accuracies. Neither was reliably related to self-reported levels of attentional engagement, highlighting the complementary value of EDA. Our work shows the sensitivity of EDA measures in real-life conditions, where low-level sensory effects, movement and speech cannot be the explanatory factor. Ultimate applications may be in educational and entertainment domains, exploring potential differences in attentional engagement patterns between experts and novices, or different target groups in entertainment.

There is a growing body of research on physiological synchrony (PS) in Collaborative Problem Solving (CPS). However, the current literature presents inconclusive findings about the way in which PS is reflected in cognitive and affective group processes and performance. In light of this, this study investigates the relationship between PS and metacognitive experiences (ie, judgement of confidence, task interest, task difficulty, mental effort and emotional valence) that are manifested during CPS. In addition, the study explores the association between PS and group performance. The participants were 77 university students who worked together on a computer‐based CPS simulation in groups of three. Participants’ electrodermal activity (EDA) was recorded as they worked on the simulation and metacognitive experiences were measured with situated self‐reports. A Multidimensional Recurrence Quantification Analysis was used to calculate the PS among the collaborators. The results show a positive relationship between continuous PS episodes and groups’ collective mental effort. No relationship was found between PS and judgement of confidence, task interest, task difficulty or emotional valence. The relationship between PS and group performance was also non‐significant. The current work addresses several challenges in utilising multimodal data analytics in CPS research and discusses future research directions.

Individuals that pay attention to narrative stimuli show synchronized heart rate (HR) and electrodermal activity (EDA) responses. The degree to which this physiological synchrony occurs is related to attentional engagement. Factors that can influence attention, such as instructions, salience of the narrative stimulus and characteristics of the individual, affect physiological synchrony. The demonstrability of synchrony depends on the amount of data used in the analysis. We investigated how demonstrability of physiological synchrony varies with varying group size and stimulus duration. Thirty participants watched six 10 min movie clips while their HR and EDA were monitored using wearable sensors (Movisens EdaMove 4 and Wahoo Tickr, respectively). We calculated inter-subject correlations as a measure of synchrony. Group size and stimulus duration were varied by using data from subsets of the participants and movie clips in the analysis. We found that for HR, higher synchrony correlated significantly with the number of answers correct for questions about the movie, confirming that physiological synchrony is associated with attention. For both HR and EDA, with increasing amounts of data used, the percentage of participants with significant synchrony increased. Importantly, we found that it did not matter how the amount of data was increased. Increasing the group size or increasing the stimulus duration led to the same results. Initial comparisons with results from other studies suggest that our results do not only apply to our specific set of stimuli and participants. All in all, the current work can act as a guideline for future research, indicating the amount of data minimally needed for robust analysis of synchrony based on inter-subject correlations.

When multiple individuals are presented with narrative movie or audio clips, their electrodermal activity (EDA) and heart rate show significant similarities. Higher levels of such inter-subject physiological synchrony are related with higher levels of attention toward the narrative, as for instance expressed by more correctly answered questions about the narrative. We here investigate whether physiological synchrony in EDA and heart rate during watching of movie clips predicts performance on a subsequent vigilant attention task among participants exposed to a night of total sleep deprivation. We recorded EDA and heart rate of 54 participants during a night of total sleep deprivation. Every hour from 22:00 to 07:00 participants watched a 10-min movie clip during which we computed inter-subject physiological synchrony. Afterwards, they answered questions about the movie and performed the psychomotor vigilance task (PVT) to capture attentional performance. We replicated findings that inter-subject correlations in EDA and heart rate predicted the number of correct answers on questions about the movie clips. Furthermore, we found that inter-subject correlations in EDA, but not in heart rate, predicted PVT performance. Individuals' mean EDA and heart rate also predicted their PVT performance. For EDA, inter-subject correlations explained more variance of PVT performance than individuals' mean EDA. Together, these findings confirm the association between physiological synchrony and attention. Physiological synchrony in EDA does not only capture the attentional processing during the time that it is determined, but also proves valuable for capturing more general changes in the attentional state of monitored individuals.

Is an audience captured by a speech or lecture? At what times especially? Do different groups in an audience experience the same speech in different ways? Insight into attentional engagement of individuals can be valuable but difficult to quantify using self-report. Physiological synchrony, the degree to which physiological measurements such as electrodermal activity (EDA) of multiple people uniformly change, has been shown to covary with attentional engagement in lab settings. In this study, we moved out of the lab and monitored EDA of 30 individuals attending a real-life inaugural lecture. These individuals were labeled as belonging to either the personal or professional group, based on their relation with the speaker. We expected these groups to differ in their attentional engagement. We computed physiological synchrony between the participants and investigated how well this metric distinguished between the professional and personal groups, how well it marked predefined engaging events in the lecture, and its relation with levels of engagement as self-reported afterwards. Where possible, we compared physiological synchrony results to results based on individuals’ EDA. We found that physiological synchrony in EDA can distinguish between the two groups. Individuals’ EDA can also distinguish between the groups, if the occurrence and timing is known of an event that is expected to elicit different levels of engagement for the two groups. We further found that both synchrony and individuals’ EDA measures mark predefined engaging events with above-chance accuracies. Neither was reliably related to self-reported levels of attentional engagement, highlighting the complementary value of EDA. Our work shows the sensitivity of EDA measures in real-life conditions, where low-level sensory effects, movement and speech cannot be the explanatory factor. Ultimate applications may be in educational and entertainment domains, exploring potential differences in attentional engagement patterns between experts and novices, or different target groups in entertainment.

In this chapter, we outline how modes of interaction, such as cognitive and socio-emotional, and the regulation of learning provide support for collaborative engagement and examine how it changes over time. We start by framing how regulated learning is embedded in the cognitive and socio-emotional interaction between the group members from both a theoretical and a methodological perspective. We then move to illustrate, with an empirical case example, how multimodal data (i.e., video) and physiological signals, such as electrodermal activity indicating physiological synchrony between the group members, can be used to capture varying levels of collaborative engagement. The empirical example provides a complementary view on group interaction and collaborative engagement. We conclude by discussing how investigating group interaction that targets regulation can reveal how collaborative engagement is built and maintained. Additionally, we discuss future possibilities to harness multimodal data in practice to support collaborative engagement.

Socially shared regulation contributes to the success of collaborative learning. However, the assessment of socially shared regulation of learning (SSRL) faces several challenges in the effort to increase the understanding of collaborative learning and support outcomes due to the unobservability of the related cognitive and emotional processes. The recent development of trace‐based assessment has enabled innovative opportunities to overcome the problem. Despite the potential of a trace‐based approach to study SSRL, there remains a paucity of evidence on how trace‐based evidence could be captured and utilised to assess and promote SSRL. This study aims to investigate the assessment of electrodermal activities (EDA) data to understand and support SSRL in collaborative learning, hence enhancing learning outcomes. The data collection involves secondary school students ( N = 94) working collaboratively in groups through five science lessons. A multimodal data set of EDA and video data were examined to assess the relationship among shared arousals and interactions for SSRL. The results of this study inform the patterns among students' physiological activities and their SSRL interactions to provide trace‐based evidence for an adaptive and maladaptive pattern of collaborative learning. Furthermore, our findings provide evidence about how trace‐based data could be utilised to predict learning outcomes in collaborative learning. Practitioner notes What is already known about this topic Socially shared regulation has been recognised as an essential aspect of collaborative learning success. It is challenging to make the processes of learning regulation ‘visible’ to better understand and support student learning, especially in dynamic collaborative settings. Multimodal learning analytics are showing promise for being a powerful tool to reveal new insights into the temporal and sequential aspects of regulation in collaborative learning. What this paper adds Utilising multimodal big data analytics to reveal the regulatory patterns of shared physiological arousal events (SPAEs) and regulatory activities in collaborative learning. Providing evidence of using multimodal data including physiological signals to indicate trigger events in socially shared regulation. Examining the differences of regulatory patterns between successful and less successful collaborative learning sessions. Demonstrating the potential use of artificial intelligence (AI) techniques to predict collaborative learning success by examining regulatory patterns. Implications for practice and/or policy Our findings offer insights into how students regulate their learning during collaborative learning, which can be used to design adaptive supports that can foster students' learning regulation. This study could encourage researchers and practitioners to consider the methodological development incorporating advanced techniques such as AI machine learning for capturing, processing and analysing multimodal data to examine and support learning regulation.

Learning analytics in collaborative learning supported by Slack: From the perspective of engagement

When interested in monitoring attentional engagement, physiological signals can be of great value. A popular approach is to uncover the complex patterns between physiological signals and attentional engagement using supervised learning models, but it is often unclear which physiological measures can best be used in such models and collecting enough training data with a reliable ground-truth to train such model is very challenging. Rather than using physiological responses of individual participants and specific events in a trained model, one can also continuously determine the degree to which physiological measures of multiple individuals uniformly change, often referred to as physiological synchrony. As a directly proportional relation between physiological synchrony in brain activity and attentional engagement has been pointed out in the literature, no trained model is needed to link the two. I aim to create a more robust measure of attentional engagement among groups of individuals by combining electroencephalography (EEG), electrodermal activity (EDA) and heart rate into a multimodal metric of physiological synchrony. I formulate three main research questions in the current research proposal: 1) How do physiological synchrony in physiological measures from the central and peripheral nervous system relate to attentional engagement? 2) Does physiological synchrony reliably reflect shared attentional engagement in real-world use-cases? 3) How can these physiological measures be fused to obtain a multimodal metric of physiological synchrony that outperforms unimodal synchrony?