Semantic Approach for Increasing Test Case Coverage in Automated Grading of Programming Exercise

Abstract

The widely popular approach for automatic grading in computer science is to run black-box testing against the student’s implementation. This kind of autograder evaluate programs solely based on their outputs given a set of inputs. However, manually writing a set of test cases with high coverage is laborious and inefficient. Hence, we explore another alternative approach in building test cases, specifically white-box testing. In theory, by knowing the internal workings of implementation, we can evaluate all possible execution paths, producing better test cases coverage, ultimately producing a complete grading. In this paper, we present research on using semantic analysis to generate test cases to determine the correctness of a student’s implementation. Instead of writing test cases, the evaluator will write a reference code, a correct implementation based on the programming problem specification. We implement a system that records execution paths, detects path deviation, and checks path equivalence to analyze the semantic difference of the reference code and student’s implementation. The system is built utilizing a concolic execution method for exploration and an SMT solver to solve formulas. Our experiments reveal that it is possible to automatically generate test cases and grade programming assignments by analyzing the semantic difference between reference and student implementation. Compared with grading using a random test case generator, it is evident that the system can provide better test case coverage for automatic grading in many occurrences.