Examining the performance of engineering treatment evaluation methodologies using the hypothetical treatment and actual treatment settings

Abstract

The selection of treatment evaluation methodology is paramount in determining reliable crash modification factors (CMFs) for engineering treatments. A lack of ground truth makes it cumbersome to examine the performance of treatment evaluation methodologies. In addition, a sound methodological framework is critical for evaluating the performances of treatment evaluation methodologies. In addressing these challenges, this study proposed a framework for assessing treatment evaluation methodologies by hypothetical treatments with known ground truth and actual real-world treatments. In particular, this study examined three before-after treatment evaluation approaches: 1) Empirical Bayes, 2) Simulation-based Empirical Bayes, and 3) Full Bayes methods. In addition, this study examined the Cross-Sectional treatment evaluation methodology. The methodological framework utilized five datasets of hypothetical treatment with known ground truth based on the hotspot identification method and a real-world dataset of wide centerline treatment on two-lane, two-way rural highways in Queensland, Australia. Results showed that all the methods could identify the ground truth of hypothetical treatments, but the Full Bayes approach better predicts the known ground truth compared to Empirical Bayes, Simulation-based Empirical Bayes, and Cross-Sectional methods. The Full Bayes approach was also found to provide the most precise estimate for real-world wide centerline treatment along rural highways compared to other methods. Moreover, the current study highlighted that the Cross-Sectional method offers a viable estimate of treatment effectiveness in case the before-period data is limited.