How Many Simulation Runs are Required to Achieve Statistically Confident Results: A Case Study of Simulation-Based Surrogate Safety Measures

Abstract

This research explores how to compute the minimum number of runs (MNR) required to achieve a specified confidence level for multiple measures of performance (MOP) of a simulated traffic network. Traditional methods to calculate MNR consider the confidence intervals of multiple MOPs separately and hence are not able to control the overall confidence level. A new method to calculate MNR is proposed, which sequentially runs the model and recalculates sample standard deviations and means whenever an additional run is made until a stopping condition based on the Bonferroni inequality is satisfied. The overall confidence level is controlled by the Bonferroni inequality. The proposed method is computationally practical since it can be implemented automatically in most traffic micro-simulation packages. The proposed method is evaluated using a case study with multiple simulation-based surrogate safety measures, including time to collision (TTC) or deceleration rate required to avoid a crash (DRAC), and an empirical confidence level analysis based on a very large number of runs. Evaluation results indicate the effectiveness of the proposed method as it enables all MOPs at the same time to be estimated accurately at the desired confidence level whereas traditional methods do not. In addition, the proposed method is not conservative since it does not require significantly more runs compared to traditional methods.