Modeling Agent Behaviors for Policy Analysis via Reinforcement Learning

Abstract

Agent-based Models (ABMs) are valuable tools for policy analysis. ABMs help analysts explore the emergent consequences of regulatory and policy interventions in multi-agent decision-making settings. But the validity of inferences drawn from ABM explorations depends on the quality of the ABM agents' behavioral models. Prior approaches for specifying behaviors have limitations. This paper examines the value of reinforcement learning (RL) models as adaptive, high-performing, and behaviorally-valid models of agent decision-making in ABMs. We discuss the value of RL for modeling agents' utility-maximizing behaviors in policy-relevant ABMs. We address the problem of adapting RL algorithms to handle multi-agency in games by adapting and extending methods from recent literature. We evaluate examples of such RL-based ABM agents via experiments on two policy-relevant ABMs: a Minority Game ABM, and an ABM of Influenza Transmission. The RL behavioral models can outperform the default adaptive behavioral models. We also run analytic experiments on our RL-equipped ABMs: explorations of the effects of dynamic behavioral heterogeneity in a population, the impact of social network factors on adaptability, and the emergence of synchronization in a community. Our results suggest that the RL formalism can be an efficient abstraction for behavioral models in ABMs.