From Explaining How Random Forest Classifier Predicts Learning of Software Engineering Teamwork to Guidance for Educators

Abstract

This Research Full Paper builds on our previous research in Software Engineering (SE) Teamwork Assessment and Prediction project (SETAP) where we used Random Forest (RF) classifier to predict with over 70% accuracy the student learning effectiveness in software engineering teamwork based on 115 objective and quantitative Team Activity Measures (TAM). These TAM measures have been obtained from monitoring and measuring activities of 74 student teams during the creation of their final class project in a joint software engineering classes which ran concurrently at three universities (San Francisco State University, Fulda University and Florida Atlantic University) over the period of four years and, together with team outcomes, have been collected in publicly available SETAP database. In this paper, we provide much deeper analysis of how and why RF made its decisions, namely we address the explainability of our RF classification. We also provide in-depth analysis of differences in grading and behavior of local and global student teams (composed of students from multiple schools). We then use these insights to provide concrete and practical guidance to educators teaching SE teamwork in local and global classroom setting.