A large language model framework to uncover underreporting in traffic crashes

Abstract

Introduction: Crash reports support the development of traffic safety countermeasures, but these reports often suffer from underreporting of crucial crash factors due to miscoded entries during data collection. To rectify these issues, the current practice relies on manual information rectification, which is time consuming and error prone, especially with large data volumes. To address these hurdles, we develop a framework to analyze traffic crash narratives and uncover underreported crash factors by capitalizing on the capabilities of Large Language Models (LLM). Method: The framework integrates procedures for prompt definition, selection of LLM generation parameters, output parsing, and underreporting determination. For evaluation, we present a case study on identification of underreported alcohol involvement in traffic crashes. We investigate the framework’s identification accuracy in relation to different underlying LLMs (i.e., ChatGPT, Flan-UL2, and Llama-2), prompt framings (i.e., explicit vs. implicit matching), and generation parameters (i.e., sampling temperature and nucleus probability). Our validation dataset consists of 500 crash reports from the State of Massachusetts. Results: Analysis results demonstrate that the developed framework achieves a recall and precision of up to 1.0 and 0.93, respectively, indicating a successful retrieval of underreported instances. These findings indicate that the developed framework addresses a critical gap in the existing traffic safety analysis workflow by enabling safety analysts to uncover underreporting in crash data efficiently and accurately, without the need for extensive expertise in natural language processing. Practical Applications: Thus, the developed approach offers unprecedented opportunities to maximize the quality and comprehensiveness of traffic crash records, paving the way for more effective countermeasure development.