Model laboratories: A quick-start guide for design of simulation experiments for dynamic systems models

Abstract

The use of dynamic systems models by scientists, managers, and policy-makers is becoming more common due to the increasingly complex nature of ecological and socio-economic problems. Unfortunately, most scientific training in the life sciences only includes dynamic modeling as elective, supplementary courses at a beginners-level, which is not conducive to generating the expertise needed to properly develop, test, and learn from dynamic modeling approaches and risks utilization of poor quality models and adoption of unreliable recommendations. The objective of this paper is to fill part of that gap, particularly regarding model experimentation, by summarizing key concepts in experimental design for simulation experiments and illustrating hands-on examples of experiments needed for developing a deeper understanding of complex, dynamic systems. The experiments include extreme conditions testing, sensitivity analyses of model behaviors given variation in both parameter values and graphical (table) functions, and “what-if?” experiments (e.g., counterfactual trajectories, boundary-adequacy tests, and intervention threshold experiments). Each experimental example describes the theoretical foundation of the test, illustrates its application using an ecological systems model, and increases in degree of difficulty from novice to advanced skill levels. By doing so, we demonstrate consistent, scientific means to glean valuable insights about the model's structure-behavior link, uncover any unforeseen model flaws or incorrect formulations, and enhance the confidence (validity) of the model for its intended use.