Verification, Validation, and Replication Methods for Agent-Based Modeling and Simulation: Lessons Learned the Hard Way!

Abstract

Reproducible modeling and simulation research has been identified as one of the Modeling and Simulation (M&S) Grand Challenge activities. Recently, uncertainty quantification has seen a renewed emphasis. While methods for verification and validation (V&V) have been widely developed for discrete-event simulations, newer simulation approaches such as the agent-based, agent-directed, and multi-agent simulation approaches introduce new V&V challenges. The active elements in these newer approaches have greater heterogeneity, e.g., every agent can be unique, with complex attributes and behaviors. Those behaviors can result in actions based on interaction with other agents, the environment, and even the outcome of simulated or artificial intelligence. The simulation spaces are often less constrained, e.g., rather than a network of servers and queues, the space can be continuous 2D Euclidian space with multiple associated geographic information systems (GIS) layers influencing the behavior of the actors. Over the last decade, a multitude of techniques has been used in agent-based modeling and simulation (ABMS) to perform V&V as well as replication and reproducibility (R&R) of the models. In this chapter, we present an overview of contributions in V&V, quality assurance (QA), and R&R of simulation studies, with special focus on ABMS. We also discuss the lessons learnt in V&V and replication from a series of simulation experiments using agent-based models (ABMs).