Peer Instruction in Online Synchronous Software Engineering - Findings from fine-grained clicker data

Abstract

In this Research Full paper, we present the results of a replication study in a semester-long, sophomore-level software engineering course utilizing Peer Instruction (PI). PI is an active learning pedagogy with roots in STEM Education. In this study, we examine the relationship between student response data from in-class PI correctness and students' performance on quizzes and exams. We worked with a fully remote, synchronous course offered over Zoom. The study we replicated was with an honors cohort of students with a diversity of undergraduate majors, while we focused on a non-honors course containing computing-related majors. Our intervention design included a flipped-classroom approach for each class session with required readings, reading quizzes, followed by PI in class using online breakout rooms for peer discussion. Our course modules were heavily based on industry practices and knowledge from the workforce, across several varied modules that encompass the complete software development lifecycle, and were as follows: Software Process Models (SPM), Software Architecture (SA), Databases (DB), User Interface/user Experience (UI/UX), Software Testing (ST), and Continuous Integration (CI). Our data points for analysis with fine-grained PI student response data were two-fold: scores from weekly online quizzes, and a summative final exam, administered online through a course management system (CMS), at different points during the semester after the PI sessions. The online quizzes and the online exam were timed, closed book/notes, and conducted during class periods. We analyzed and classified individual student responses before and after each question in each module and attempted to create response patterns for each module. We correlated these response patterns with exam and quiz scores using ANOVA techniques, on a variety of questions including Parson's problems. We report overall correctness on each type of vote, track student response patterns from in-class to quizzes and the exam, and quantify absolute percentages of students that demonstrate longer-term learning from the PI process. Our results show that 58% of students exhibited cognitive gains across all modules during PI sessions. Students who learn in class from PI perform well on the quizzes and the final exam, indicating persistence of the knowledge gained during PI several weeks after the actual sessions. We also found that those who fail to learn from the PI process in the class perform worse on quizzes and the final exam. Our results were consistent across all modules. More significantly, we found PI to be an effective way to teach our software engineering course based on student learning before and after PI, in a completely virtual environment, a result unique to our study. Based on our results, we discuss the implications for software engineering education, both in-person and virtual.