Development of BDMS Utilization Review Technology (BURT): An Artificial Intelligence Tool Using Thai Natural Language Processing to Assess Appropriateness of Hospitalization

Abstract

OBJECTIVES: To develop an effective artificial intelligence (AI) driven platform to optimize the process of assessing appropriateness of hospitalization. MATERIALS AND METHODS: Anonymized data of 22,020 insured-patient admissions in a BDMS network hospital were included to build a prediction model based on a comprehensive guideline for appropriate hospitalization. To develop Thai Natural Language Processing (NLP) model, 77,707 sentences from medical records were used and separated into two datasets, 80% for training and 20% for testing. A combined NLP and rule-based algorithms formed an AI engine and outputs were displayed using a web-based application. An expert panel of five Utilization Management (UM) physicians had several collaborative discussions to fine tune the NLP model, application of clinical criteria, and classification engine. Eventually, NLP model in the latest version (BURT1.1), had satisfactory features with overall higher than 99% accuracy, precision, recall, and F1.  RESULTS: Performance of BURT1.1 was assessed using 300 cases randomly selected from the main dataset, against other methods, including concurrent review by UM nurses at the participating hospital, and UM nurses at Bangkok Hospital Headquarters (BHQ). Agreement upon UM Physician Panel consensus was set as one of the performance indicators, and BURT1.1 showed a favorable outcome with the highest rate of agreement (86%) among all the methods. The precision rate was 99% as compared to insurance claim approval status. Additionally, dramatic time savings were achieved with 0.59 second of processing time as compared to 10-15 minutes per case by conventional manual review. CONCLUSION: BURT1.1 should be effectively implemented as an automatic daily tool to screen inappropriate hospitalization. It can immediately identify patients at high risk of inappropriate hospitalization that require further assessment by UM nurse, thus providing feedback to attending physicians on the completeness and quality of documentation, with parallel notification to UM physicians. Ultimately, BURT1.1 can contribute to increase UM efficiency, speeding up the claim process, reducing health care costs due to unnecessary hospitalization, and reduction of claim denials.