A CASE STUDY OF SUCCESSFUL TEST AND EVALUATION IN A SYSTEM OF SYSTEMS ENVIRONMENT, THE AIR FORCE MODELING AND SIMULATION TRAINING TOOLKIT

Abstract

Modern systems are becoming exponentially more complex and interconnected, particularly in software-intensive systems. Department of Defense mandates for net-centric capabilities have greatly complicated the challenges for testing and evaluating new systems which are employed in a system of systems (SoS) environment. Traditional "platform-centric" test methodologies have not fared well in this environment, yet systems must still be tested for both performance and regulatory reasons. This paper will discuss how system complexity and potential benefits of interconnectivity along with DoD net-centric mandates have driven systems into an SoS testing environment that is far more difficult than was initially thought. The desire to "test everything" has been recognized as unrealistic and unaffordable, both in cost and in time. A case study of the Air Force Modeling and Simulation Training Toolkit and how it has attacked challenges of connecting into multiple other systems will be presented to illustrate the test and evaluation challenges in a SoS environment. Lessons learned will be discussed along with a preliminary framework for more general cases of testing in a SoS environment.