Checked Coverage and Object Branch Coverage

Abstract

Many educators currently use code coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or branch coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage metrics assess what percentage of code has been exercised, but do not depend on whether a test suite adequately checks that the expected behavior is achieved. This paper evaluates checked coverage, an alternative measure of test thoroughness aimed at overcoming this limitation, along with object branch coverage, a structure code coverage metric that has received little discussion in educational assessment. Checked coverage works backwards from behavioral assertions in test cases, measuring the dynamic slice of the executed code that actually influences the outcome of each assertion. Object branch coverage (OBC) is a stronger coverage criterion similar to weak variants of modified condition/decision coverage. We experimentally compare checked coverage and OBC against statement coverage, branch coverage, mutation analysis, and all-pairs testing to evaluate which is the best predictor of how likely a test suite is to detect naturally occurring defects. While checked coverage outperformed other coverage measures in our experiment, followed closely by OBC, both were only weakly correlated with a test suite's ability to detect naturally occurring defects produced by students in the final versions of their programs. Still, OBC appears to be an improved and practical alternative to existing statement and branch coverage measures, while achieving nearly the same benefits as checked coverage.