Defining coverage of an operational domain using a modified nearest-neighbor metric

Abstract

Validation experiments are conducted at discrete settings within an operational domain to assess the predictive maturity of a model that is ultimately used to predict over the entire operational domain. Unless this domain is sufficiently explored with validation experiments, satisfactory model performance at these discrete, tested settings would be insufficient to ensure satisfactory model performance throughout the entire operational domain. The goal of coverage metrics is then to reveal how well a set of validation experiments represents an operational domain. The authors identify the criteria of an exemplary coverage metric, evaluate the ability of existing coverage metrics to fulfill these criteria, and propose a new, improved coverage metric. The proposed metric favors interpolation over extrapolation through a penalty function, causing the metric to prefer a design of validation experiments near the boundaries of the domain, while simultaneously exploring inside the domain. Furthermore, the proposed metric allows the coverage to account for the relative influence of each dimension of the domain on the model output. Application of the proposed coverage metric on a practical, non-trivial two-dimensional problem is demonstrated on the Viscoplastic Self-Consistent material plasticity code for 5182 aluminum alloy. Furthermore, the proposed metric is compared to existing coverage metrics considering a high dimensional problem with application to the Rosenbrock function.