The Complexities of Agent-Based Modeling Output Analysis

Ju-Sung Lee,Forrest Stonedahl,Zhanli Sun,Arika Ligmann-Zielinska,Tatiana Filatova,Dawn C Parker,Behrooz Hassani-Mahmooei,Iris Lorscheid,Alexey Voinov,Gary Polhill

doi:10.18564/jasss.2897

Abstract

The proliferation of agent-based models (ABMs) in recent decades has motivated model practitioners to improve the transparency, replicability, and trust in results derived from ABMs. The complexity of ABMs has risen in stride with advances in computing power and resources, resulting in larger models with complex interactions and learning and whose outputs are often high-dimensional and require sophisticated analytical approaches. Similarly, the increasing use of data and dynamics in ABMs has further enhanced the complexity of their outputs. In this article, we offer an overview of the state-of-the-art approaches in analyzing and reporting ABM outputs highlighting challenges and outstanding issues. In particular, we examine issues surrounding variance stability (in connection with determination of appropriate number of runs and hypothesis testing), sensitivity analysis, spatio-temporal analysis, visualization, and effective communication of all these to non-technical audiences, such as various stakeholders.

Highlights

1.1 Agent-based models (ABMs) have been gaining popularity across disciplines and have become increasingly sophisticated
5.1 While ABM as a technique offers many exciting opportunities to open research frontiers across a range of disciplines, there are a number of issues that requires rigorous attention when dealing with ABM output data
We group them into 3 themes: (i) Statistical issues related to defining the number of appropriate runs and hypothesis testing, (ii) Solution space exploration and sensitivity analysis, and (iii) Processing ABM output data over time and space

Summary

Introduction

1.1 Agent-based models (ABMs) have been gaining popularity across disciplines and have become increasingly sophisticated. Visualizing variance decomposition temporally will reveal parameter stability over the course of the model run (see Figure 7) Spatial outputs such as land use change maps may receive similar treatment to reveal the extent of outcome uncertainty in regions (or clusters) due to specific parameters (Ligmann-Zielinska 2013). 3.20 SA for multiple outcome variables in an ABM incurs additional challenges due to differences in each parameter's impact on the outcomes (see Figure 4) This issue is of particular concern for model simplification and demands either one single, well-chosen outcome that is adequately representative of the model's behaviour or more conservatively, the difficult task of undergoing SA across the whole spectrum of outputs: scalars, time- and space-dependent measures. Model behaviour may be captured as animated GIF files, the small sizes of which facilitate their use in presentations and web pages (Lee & Carley 2004; Lee 2004)

Discussion and conclusions

Solution space exploration and sensitivity analysis

Processing ABM output data over time and space

Communicating ABM results to stakeholders

Findings

Directions for future research