Accuracy and effectiveness of an orchestration tool on instructors’ interventions and groups’ collaboration

Abstract

This paper presents the development of a novel orchestration tool that predicts collaborative problem-solving (CPS) behaviors of undergraduate engineering groups and investigates the use of that tool by instructors. We explore the impact of receiving real-time, machine-learning, model-based prompts on 1) instructors’ orchestration strategies, which are strategies instructors use to manage and facilitate collaborative activities, and 2) groups’ participation, including how groups are engaged in CPS activities. The orchestration tool is a dashboard that notifies instructors of—and advises them on—monitoring and intervening with groups who may need collaborative support and guidance. We describe the accuracy of the models in predicting CPS behaviors and of instructors in identifying these behaviors in the classroom. We then describe how real-time prompts from models can affect instructors’ orchestration strategies and students’ participation. Our findings show that there is variability in the accuracy of our machine learning models and that instructors are better at identifying predictive behaviors as compared to the models. Instructors in this context engaged in orchestration strategies, like monitoring and probing when using the orchestration tool, and groups of students were largely talking while on-task across classes. We triangulate across data sources to examine the effectiveness of the orchestration tool in the classroom and share pedagogical and technical implications for the field.