Special Issue: Advancing Statistical Methods for Testing and Evaluating Defense Systems

Abstract

The DoD acquires highly complex systems that must be tested and evaluated before they are fielded to military users and used in actual operations. The statistical challenges are vast. Analysis challenges include non-normal data, messy and missing data, functional outcomes, limited data, and data from various types of activities that needs to be combined into a single evaluation. Additionally, these tests are often expensive and are conducted on ranges with limited availability. The systems are operated by military users and their time is valuable and hard to acquire. For all of these reasons, every data point that is requested from a test is scrutinized and evaluated. The tools and methods of experimental design have proven very useful in this regard. The articles in this journal highlight a collection of recent research from the consortium. They show the breadth of research necessary. The articles included showcase advances in experimental design for multiple response variables and space filling designs for simulation experiments that allow for the estimation of second-order effects. Bayesian methods highlighted addressing challenges for multiphase reliability assessments, accelerated reliability growth testing, and profile monitoring for nonlinear profiles. Model validation is addressed through a case study showing the usefulness of wavelets. Unique analysis challenges and their links to experimental designs are addressed in articles on split-plot reliability experiments and D-optimal designs for logistic regression with separation. Finally, one article address practical sample size challenges for testers and answers the canonical question of “How much testing is enough?” A unique aspect of this special issue of the journal is that we included reviewers from both the statistical academic community and those practicing statistics in DoD. I am extremely grateful for their suggestions and reviews, which further improved this special issue. The result is a diversity of research that showcases the depth and breadth of statistical challenges faced in the Defense context. However, they will also provide useful for anyone who works with complex systems or emerging technologies.