A Statistical Review of the Cyber Test Process

Abstract

The variability of cyber threats along with multiple cyber testing practices distracts teams from applying classical statistical measurements of test power and coverage to describe test effectiveness. In most cases, cyber test teams continue to test until reaching funding or schedule limits. This article explores the traditional measurements of test coverage and test power to propose multiple metrics for cyber test efficiency. The metrics are then applied to data collected on recent Defense Department programs. Original findings include feedback on cyber table top efforts, the use of linear approximation in predicting an optimum duration for penetration testing, predicting the length of time at the range by the number of machines in test, and the optimization of test teams between 9 and 14 testers. These findings greatly assist in what Scientific Test and Analysis Techniques (STAT) were designed to do: help anyone involved (customers, management, and the test team) understand, and agree upon, test risk, coverage, and decisions.