Investigating unexpected outcomes through the application of statistical debuggers

Abstract

Predictions from simulations with inherent uncertainty have entered the mainstream of public policy decision-making practices. Unfortunately, methods for gaining insight into unexpected simulation outcomes have not kept pace. Subject matter experts (SMEs) need to understand if the unexpected outcomes reflect a fault in the simulation or new knowledge. Recent work has adapted statistical debuggers, used in software engineering, to automatically identify simulation faults via extensive profiling of executions. The adapted debuggers have been shown to be effective, but have only been applied to simulations with large test suites and known faults. Here we employ these debuggers in a different manner. We investigate how they facilitate a SME exploring an unexpected outcome that reflects new knowledge. We also evaluate the debuggers in the face of smaller test suites and sparse execution profiling. These novel applications and evaluations show that these debuggers are more effective and robust than previously realized.