Toward AI-supported evaluation for safety control measures against near-miss events in pharmaceutical products

Abstract

The Japan Council for Quality Health Care (JCQHC) promotes medical safety by providing health professionals in pharmacies and the public with information regarding near-miss events from pharmacies. The Pharmaceuticals and Medical Devices Agency (PMDA) evaluates the pharmaceutical near-miss events published by the JCQHC to determine whether safety measures need to be taken in terms of drug names and packaging. We propose an artificial intelligence (AI) evaluation model that performs efficient and reproducible evaluations to support PMDA reviewers. We prepared a dataset consisting of pairs of pharmaceutical near-miss events and their human-annotated evaluation in consultation with the PMDA Safe Use Measures Review Committee reports. Pharmaceutical near-miss events consist of semi-structured texts such as drug names and free descriptive comments. To extract text features such as the similarity of drug names from the semi-structured text and predict the evaluation from these text features, a light gradient boosting machine was used. The AI model was evaluated by ablation experiments on these features. Model construction using the metrics of precision, recall, f1-score, and macro-f1 and verification were performed using cross-validation. The developed AI model classified events with a high degree of accuracy close to that of PMDA's human evaluation; however, achieving human classification accuracy proved difficult. Further study is needed to improve the accuracy of the classification evaluation model. The developed model should serve as an operational rule for primary screening in PMDA operations.