Investigating the Impact of Test Case Density and Execution Variety on Fault Localization for Novice Programs

Abstract

Programming Online Verification Exam (OE) system has been widely used in algorithm education and practice since it can automatically analyze program results (e.g., correct or incorrect) after executing the submitted programs with corresponding test cases. OE systems can provide both execution results and error information so that novice programmers can get feedback quickly. If the submitted program cannot pass all the test cases, the novice programmers will get wrong-answer feedback, and they have to find and fix the errors in the program. Automated program fault localization techniques, which can locate the errors in programs under test automatically, thus help novice programmers fix the errors quickly. However, the performance of current automated fault localization techniques is limited due to the high-density test cases in novice programs of OE system. In this paper, we analyze the impact of test case density (TCD) and execution variety on fault localization performance and propose a method to reduce TCD to improve fault localization precision for novice programs. To evaluate the performance of our method, we conduct a number of empirical studies on 1199 real fault diagnosis algorithm related novice programs, and the experimental results show that using improved test cases through our method for fault localization in OE system can enhance the precision of fault localization for novice programs. Specifically, after decreasing the test cases’ density, the improvement of fault localization accuracy ranges from 0.6% to 17.34% in terms of the Expense metric, and from the Accuracy@N metrics, the number of faulty statements that can be found increases in most cases.