Using Failing Test Cases to Semi-automate Feedback for Beginner Programmers

Abstract

Ninas Pro, which runs C++ workshops for beginner programmers, uses automated feedback tools to give students quick feedback about their code in class. However, many students do not complete exercises with the tools used currently, as they do not know how to fix their code after getting a 'wrong answer' message. In the literature, binary feedback like this has been shown to have negative effects on student engagement, and tutors end up repeating the same feedback to many students for the same exercise. In this work, we have built a tool that groups incorrect solutions by the output they produce, to help tutors identify common programming and logic errors in student code during class. Tutors annotate incorrect outputs with suggestions, which will then be shown to students whenever their code produces the same wrong output. We carried out an exploratory case study to validate our approach, where students were able to fix both logical and presentation errors using the suggestions provided by our tool. Not all errors can be annotated, these must still be reviewed in person by a tutor. We also carried out a usability study where tutors successfully annotated solution groups. As such, the main contribution of this work is a tool that supports the work that tutors do during programming classes, letting them give richer feedback in a more scalable manner.