InferBERT: A Transformer-Based Causal Inference Framework for Enhancing Pharmacovigilance.

Xingqiao Wang,Weida Tong,Ruth Roberts,Zhichao Liu,Xiaowei Xu

doi:10.3389/frai.2021.659622

Abstract

Background: T ransformer-based language models have delivered clear improvements in a wide range of natural language processing (NLP) tasks. However, those models have a significant limitation; specifically, they cannot infer causality, a prerequisite for deployment in pharmacovigilance, and health care. Therefore, these transformer-based language models should be developed to infer causality to address the key question of the cause of a clinical outcome. Results: In this study, we propose an innovative causal inference model–InferBERT, by integrating the A Lite Bidirectional Encoder Representations from Transformers (ALBERT) and Judea Pearl’s Do-calculus to establish potential causality in pharmacovigilance. Two FDA Adverse Event Reporting System case studies, including Analgesics-related acute liver failure and Tramadol-related mortalities, were employed to evaluate the proposed InferBERT model. The InferBERT model yielded accuracies of 0.78 and 0.95 for identifying Analgesics-related acute liver failure and Tramadol-related death cases, respectively. Meanwhile, the inferred causes of the two clinical outcomes, (i.e. acute liver failure and death) were highly consistent with clinical knowledge. Furthermore, inferred causes were organized into a causal tree using the proposed recursive do-calculus algorithm to improve the model’s understanding of causality. Moreover, the high reproducibility of the proposed InferBERT model was demonstrated by a robustness assessment. Conclusion: The empirical results demonstrated that the proposed InferBERT approach is able to both predict clinical events and to infer their causes. Overall, the proposed InferBERT model is a promising approach to establish causal effects behind text-based observational data to enhance our understanding of intrinsic causality. Availability and implementation: The InferBERT model and preprocessed FAERS data sets are available on GitHub at https://github.com/XingqiaoWang/DeepCausalPV-master.

Highlights

The rise of artificial intelligence (AI) has transformed many aspects of human life, especially in healthcare, personal transport, law-making, and entertainment (Silver et al, 2017; Awad et al, 2018; Topol, 2019; Woo, 2019)
In this study, we propose an innovative causal inference model–InferBERT, by integrating the A Lite Bidirectional Encoder Representations from Transformers (ALBERT) and Judea Pearl’s Do-calculus to establish potential causality in pharmacovigilance
The high reproducibility of the proposed InferBERT model was demonstrated by a robustness assessment

Summary

Introduction

The rise of artificial intelligence (AI) has transformed many aspects of human life, especially in healthcare, personal transport, law-making, and entertainment (Silver et al, 2017; Awad et al, 2018; Topol, 2019; Woo, 2019). One of the breakthroughs in AI is the advent of transformer-based language models, that can achieve state-of-the-art (SOTA) performance in a wide range of natural language processing (NLP) tasks (Devlin et al, 2018; Lan et al, 2019; Brown et al, 2020; Zaheer et al, 2020). The achieved high prediction performance came at the expense of model interpretability (Moraffah et al, 2020) Another critical limitation of transformer-based language models is the lack of ability to infer causality. T ransformer-based language models have delivered clear improvements in a wide range of natural language processing (NLP) tasks. Those models have a significant limitation; they cannot infer causality, a prerequisite for deployment in pharmacovigilance, and health care.

Objectives

Methods

Results

Discussion

Conclusion