Designed Simulation Experiments, Part 1: Roots, Myths, and Limitations of Conventional DOE

Abstract

This paper illustrates specific challenges encountered in attempting to follow standardized principles and methods, established during the earlier eras of design of experiments (DOE), when the task is the design and analysis of simulation experiments (DASE). These problems are accentuated when DASE is used to develop and improve software-centric systems, such as guided missiles, which manifest a new level of dynamic complexity. An analysis of the current issues paves the way for successful DASE, and a companion paper (Part 2) describes how DOE is being modified and expanded to increase likelihood success in this emerging era. I. Introduction NGINEERED systems and the methods for developing them are as diverse as the imagination of mankind. Robust interaction has always existed between those who are passionate about “making a better mousetrap” (product) and those who are committed to finding a better way (process) to make a better mousetrap. This paper and its companion paper (Part 2) describe lessons learned over several decades of this product/process interaction in the use of design of experiments (DOE). DOE usage is examined when the source of experimental samples is a complex simulation, not an industrial process or an instrumented, “real-world” device under test. Several key issues arise and must be addressed in order for DOE to be used successfully with simulation, especially when the system being simulated includes embedded software, which fundamentally affects hardware functions, leading to a combinatoric explosion in the number system behavioral modes. As developers and users of performance simulations, the authors draw from experience gained during the creation and analysis of guided missile systems. Each of these systems includes real-time software that controls hardware, operating across a vast space of conditions and behavioral modes. Such complex systems call for collaborative engineering methods that are both disciplined and yet able to accommodate rapid rates of change during system design and development. The paper uses specific examples from the authors’ experience, emphasizing the challenges imposed by software-centric systems and simulations. These fast-evolving, multi-mode systems are a far cry from the original agricultural crops where DOE’s roots were established; they defy application of anything approaching an “ANSI-standard” DOE. Two premises are assumed at the outset: (a) that an organized, disciplined approach to engineering generally—and especially for simulating software-centric systems—is preferable to an approach lacking structure and discipline, and that (b) contrary to portrayal by some skeptics, DOE is not intended to be a cookie-cutter recipe that stifles creativity or obviates the need for intense involvement of domain experts. Both points are well-established generally. The devil emerges in the details of implementing DOE for simulation. After a brief review of the histories of DOE and simulation, the paper illustrates several of these troublesome details. A. The Dynamic Complexity of Simulation. Fully testing all combinations of system modes and environments is highly impractical. As a result of the Department of Defense policy of Simulation Based Acquisition, the fundamental requirement for a missile performance simulation is to verify performance in lieu of conducting expensive flight tests. 1 Usage of the simulation for this final assessment occurs late in the development cycle, with a mature simulation that adequately represents the final system. However,