Accuracy and effectiveness of an orchestration tool on instructors’ interventions and groups’ collaboration

An analysis of student collaborative problem solving activities mediated by collaborative simulations

As been highlighted by many, for instance by PISA, Collaborative Problem Solving (CPS) is a critical and necessary 21st century skill across educational settings. While many initiatives have been launched to investigate the nature of these skills, fewer are the attempts to understand how they should be assessed. However, in 2015, the PISA organization presented a framework for assessing CPS skills. This paper reports on a study investigating the predictive validity of the PISA assessment framework and if and how modes of communication influence the assessment of 24 students’ collaborative problem solving activities when using a computer-based assessment task system. The findings presented demonstrate that the PISA CPS assessment framework have a weak predictive validity, does not count for quality or productivity in communication, and that the mode of communication indeed influence CPS processes and in turn what is possible to assess.

The objective of this study is to explore new ways of assessing collaborative problem solving (CPS) processes based on different modalities of audio data and their combination. The data collection took place in an educational lab setting during an experiment with adult teams from professional contexts who collaboratively solved multiple problems as part of a CPS training. From audio data, both verbal (ie, speech) and non‐verbal (ie, pitch) aspects were extracted. Four analysis methods were used, including (a) content analysis; (b) linguistic inquiry and word count; (c) verbal entrainment analysis; and (d) acoustic–prosodic entrainment based on pitch data. Insights are given into the CPS processes of the participating groups using these measures and relevant relationships between some of these measures are further investigated. Based on content analysis, it was found that most of the interactions during the CPS process are task oriented, whereas team‐oriented interactions are less present. Second, three measures of proportion of contribution in CPS were investigated and clear differences in participation patterns between and within teams were found. We suggest that a combination of utterance count and words per sentence could provide valuable insights for quantity and equality of participation. Third, the study explored pronoun use and found that the most frequently used personal pronouns were first‐person singular. Next, the results indicated a relationship between pronoun use and the relative frequency of interactions. Fourth, a rather weak relationship between lexical entrainment measures and the acoustic–prosodic measures were found, suggesting that these measures are indicative of separate communicative aspects in CPS. This study contributes to a better understanding of which type of audio‐based data is most informative to teachers and students as a feedback or assessment tool. This study complements previous research as it focuses on spoken human‐to‐human communication collected in an authentic context. Practitioner notesWhat is already known about this topic Support and guidance systems for learning coaches, teachers and learners are needed to foster the educational quality of collaborative problem solving (CPS) activities. CPS is a complex process and measuring the quality of CPS processes remains challenging. Multimodal learning analytics, focusing on verbal and non‐verbal data sources and using content analysis, linguistic inquiry and word count and verbal and acoustic entrainment measures could be valuable to measure the quality of CPS. What this paper adds The majority of interactions during CPS processes are task oriented or cognitive of nature, whereas team‐oriented interactions are less present. Utterance count and words per sentence should be used in combination, as they are indicative of different aspects. Pronoun use in learners' discourse is related to the types of CPS interactions. Lexical entrainment measures and acoustic–prosodic are indicative of distinctive communicative aspects in CPS. Implications for practice and/or policy Quality indicators of CPS processes should include both verbal and non‐verbal measures of students' interactions. Educational researchers and the (Edtech) industry should further leverage their forces to foster the development of (semi‐)automated systems for measuring the quality of CPS processes. It should be further investigated how quality indicators of CPS processes can be most meaningful to trainers, teachers and learners, for example, through the use of dashboards.

The objective of this study is to identify the instructional design requirements of Educational Robotics (ER) activities for sustaining collaborative problem solving (CPS) skills. In CPS activities, learners work in teams towards the analysis, the identification and the solution of a problem. PISA and most of the 21st century skills frameworks have promoted the importance of CPS for 21st century education. In this study, we aim to analyze ER opportunities for 21st century education, in general, and CPS skills, according to the PISA CPS framework. To achieve this, we first review the design requirements of the learning activities that have been identified in the literature as sustaining CPS. Afterwards, we identify the design requirements of ER activities that can sustain the development of CPS. Activities engaging learners in authentic ill-defined problems, with a certain level of complexity are identified as design requirements that have the possibility to promote CPS in ER activities.

ABSTRACTThe development of students’ practical operation and their collaborative problem solving (CPS) ability are emphasized in higher education nowadays. Providing students a simulation learning tool and an social networking environment can improve the students’ learning in CPS activities. This can be a potential learning environment for the students when the automatic scaffolding guidance tools are appropriately given to them. Therefore, the present study developed a scaffolding-based CPS learning environment. This empirical study explored learners’ learning performance. The students’ online discussion messages were also encoded for the later quantitative content analysis to explore their cognitive process in CPS activities. The study included 94 participants, who were divided into the study sheet group (use the simulation tool and paper study sheets for discussions) and the scaffolding mind tool group. The results showed that learning performance of the scaffolding mind tool group was significantly better than that of the study sheet group. Moreover, learners in the scaffolding mind tool group showed more diverse levels in the cognitive process. The researcher further conducted a Lag sequential analysis based on the coding results. The results also suggested that the scaffolding mind tool group showed more diverse cognitive process transitions in their behavioral patterns.

This paper discusses an alternative approach for developing problem solving experiences for students. The major argument is that students can develop their creativity by engaging in collaborative problem solving activities in which they apply a variety of mathematical methods creatively to solve problems. The argument is supported by: considering why developing creativity is a worthwhile adventure for students; describing why collaborative problem solving is a unique endeavor in which to engage students more fluidly and less compartmentally; and discussing how students can become skilled at thinking logically and creatively about mathematics content through their engagement in collaborative problem solving tasks.

Identifying emergent roles and their relationship with learning outcomes and collaborative problem-solving skills

Collaborative Problem Solving (CPS) is a multifaceted process where students with strong social and cognitive skills tend to excel. However, improving these skills requires an in-depth understanding of the relationship between skills and students' internal state of mind. It is difficult to assess a student's state of mind based on external cues like facial expressions, speech and behaviour, as these can be easily disguised. This study used a portable EEG device to measure brainwave signals during online CPS tasks, focusing on variations in brain activity using the Approximate Entropy (ApEn) metric to understand why students with high CPS skills perform better. Seventy-two undergraduates participated, with their brainwaves continuously recorded during CPS activities. The findings showed distinct brainwave patterns among students with different social and cognitive skills. Notably, brain activity was heightened during the problem-solving stage compared to the conceptualisation stage. Students with higher social skills showed significant activity in the alpha (inspiration), beta (decision-making) and gamma (creative thinking) bands, while those with stronger cognitive skills primarily activated the alpha band. These results highlight the need to tailor online CPS activities to effectively support students' development in both social and cognitive domains.

Collaborative problem-solving (CPS) is an essential competency in the 21st century. However, this skill is quite complex due to the two-dimensional domain having to measure at once. This study is a literature review aimed to presents various CPS measurement methods from empirical research for the past 17 years. Hence, teachers can get insight to implement and to develop CPS assessment mainly for training students to solve complex problems. Based on this review, there are two broad methods to measure CPS including using computer environment and real-life context. The former allows measurement to be more accurate and quicker, while the latter enables students to experience face-to-face interaction during problem solving. However, the development of CPS assessment using a computer needs professional ICT and an adequate number of computers as well as a high internet connection. In Indonesia, the implementation of computer-based assessment faces many challenges, and only some schools in the city centre can meet that requirement. Hence, it can be an alternative to measure the interaction of students during the CPS activity by putting high consideration of the using pedagogical content to facilitate CPS environment.

Laboratory experience is critical to foster college students’ collaborative problem-solving (CPS) abilities, but whether students stay cognitively engaged in CPS tasks during online laboratory sessions remains unknown. This study applied multimodal data analysis to examine college students’ (N = 36) cognitive engagement in CPS during their online experimentation experience. Groups of three collaborated on CPS tasks via shared worksheets and computer-based simulations on videoconferences. Portable electroencephalogram instruments were used to determine students’ levels of cognitive engagement in CPS activities. The multimodal data analysis (e.g., electroencephalogram, surveys, and artifacts) results showed a significant difference in students’ cognitive engagement between different phases of CPS. The students’ cognitive engagement significantly differed between groups who did and did not complete the task. Additionally, intrinsic motivation predicted students’ cognitive engagement in the completion groups while self-efficacy was the primary predictor of cognitive engagement for the groups who did not complete the task.

Collaborative learning plays an important role in improving individuals' skills. While various studies have proposed the integration of adaptive mechanisms in collaborative learning environments, only a few studies considered how these mechanisms can be used to improve students' collaboration skills. Furthermore, there is a lack of studies that address key challenges in collaborative problem solving (CPS). Most studies rely on guidance based on post study coding of student problem solving skills. In this paper, the authors suggest adaptive collaborative learning environments, exploiting Epistemic Network Analysis, to dynamically assess students' CPS skills. The adaptive learning system suggested multi-level feedback tailored according to students' CPS skills and the quality of their collaboration. The objective is to improve students' skills and their reflection. Students were divided into an experimental and a control group and engaged in CPS activities. The findings reveal the positive effect of the adaptive system on student participation and the development of their CPS skills.

Collaborative problem solving (CPS) is a critical competency in the Artificial intelligence (AI) era, requiring the integration of cognitive and social skills through real-time dialogue and coordination. While prior studies have explored CPS behaviours using human-coded text from online platforms, limited research has examined how machine learning (ML) and deep learning (DL) models perform on spoken peer dialogue in face-to-face (F2F) classroom settings. This study investigates the automatic classification of CPS phases using a validated coding framework applied to two classroom tasks—one supported by a GenAI assistant and one not. A total of 7,744 utterances were manually labelled across nine CPS subskills and three broader facets. Six ML and five DL models were evaluated, including lightweight BERT variants combined with various classifiers. Results show that BERT-based models significantly outperform traditional ML approaches. Specifically, BERT+ANN achieved better overall performance in smaller, imbalanced datasets, while BERT+CNN performed better in larger datasets. Reducing label granularity from nine subskills to three facets consistently improved classification accuracy and F1 scores. Both models achieved AUROC scores around 0.90, indicating strong discriminative capability. Several key insights emerged from the findings: Model architecture matters: Simpler classifiers like ANN preserve BERT’s semantic representations and offer stable performance, especially in smaller or imbalanced datasets. Task context influences CPS behaviour: Different tasks elicit distinct CPS skill distributions, with task regulation dominating in technical tasks and communicative participation more prevalent in reflective tasks. Label granularity affects performance: Reducing the number of classification labels (e.g., from 9 subskills to 3 facets) significantly improves model accuracy and generalizability. Lightweight models are viable: Even with a reduced-capacity BERT model, competitive performance was achieved, suggesting potential for real-time, resource-efficient deployment in educational settings. This study contributes to educational AI by introducing a novel oral CPS dataset, benchmarking multiple models, and demonstrating the feasibility of lightweight architectures for real-time deployment. Limitations include the small sample size and single-modality input. Future work should explore multimodal features, larger and more diverse classrooms, and teacher-facing dashboards for actionable feedback. The findings support the development of scalable, ethical, and human-centered learning analytics tools that enhance collaborative learning in AI-enhanced education.

Collaborative problem-solving (CPS) engages students in solving ill-structured problems, creating group knowledge, and developing self-regulation and collaboration skills. Different scaffoldings, such as minimal-guided, task-oriented, and idea-oriented, can be used to facilitate students’ CPS activities, but their effects have not been comprehensively explored. In this research, we use minimally-guided, task-oriented, and idea-oriented scaffoldings to promote Chinese university students’ online CPS activities and use a multi-method approach to analyze the effects of three scaffolding on collaboration. The results indicate relatively complicated collaborative processes and outcomes supported by three scaffoldings. It is initially shown that the idea-centered scaffolding strengthens students’ connections between idea contribution, metacognitive regulation, and knowledge artifact behaviors, which are critical factors for improving the CPS quality. Based on the empirical research results, we conclude that future instructional design should carefully consider the educational culture, time constraint, and student regulation to better facilitate CPS practices.

Collaborative problem solving is a core ability that has been highly valued in recent years. Collaborative problem solving activities allow learners develop collaboration skills. In science education, collaborative learning with simulations enables learners to manipulate a science problem to explore scientific concepts. However, the collaboration during such a learning context is a complicated process and researchers face difficulties in understanding learners’ mental effort in using the simulations. The use of dual eye-tracking techniques is helpful to uncover learners’ visual attention, and thus to better analyze student collaboration in activities. In this paper, the research focus on learners’ difficulties when they learn together with the simulation in different places. The results show that the techniques are helpful to identify the subtle interaction problem including the problem of lacking coordination, the process misunderstanding problem, and misunderstanding in partners’ attention. Educators may need to address these problems when simulations are applied to support remote collaborative science learning.

BackgroundCommunity health nurses collaborate in teams to address client care, organise services, and conduct preventive activities. Hence, it is crucial for students in this work context, to practise and enhance their collaborative problem-solving (CPS) skills. To facilitate this, a hybrid serious game called "Carion" was developed, immersing second-year nursing students in a semester-long experience of working as district nurses in self-managed teams. The game challenges them to collectively tackle authentic and fictional problems. However, the extent and manner in which this hybrid serious game contributes to students' CPS skill development is unknown. AimThe aim of this study is to explore how students perceive the development of CPS by doing assignments carried out in the learning context of the hybrid serious game Carion. ParticipantsTwo teams each consisting of five second-year nursing students, were selected for the qualitative case study. Two lecturers acting as team coaches also participated in this study. 110 (response rate 61 %) second-year students participated in the quantitative survey at the beginning of the semester, 109 students at the end of the semester. MethodsThe study is a mixed method design. The qualitative component involves a case study while the quantitative component consists of an online survey. Qualitative and quantitative data regarding collaborative problem solving was collected at different time points using focus group interviews, one-on-one interviews, digital logbooks and an online self-assessment tool. ResultsQualitative analysis revealed insights into three key themes: (1) Collaboration, encompassing consulting with each other, and capitalising on each other’s qualities and strengths; (2) Problem solving, encompassing quicker and easier problem solving, allocating assignments, coming to a joint solution, and problem-solving method; and (3) Learning from each other, encompassing learning from each other’s knowledge and experience, and from each other’s ideas. No growth in collaborative problem-solving development was statistically shown in the group. ConclusionsThis study gives an ambiguous view on students' perception of their CPS development through involvement in various Carion assignments. Based on this exploratory investigation, prematurely labelling Carion as a potential educational environment for developing CPS among undergraduate nursing students in community healthcare would be unwarranted. Further research is needed, particularly in scrutinising the interplay between CPS and diverse learning activities, assignments and learning outcomes within this specific hybrid serious game context.