Reply: The evaluation of data mining methods for the simultaneous and systematic detection of safety signals in large databases: lessons to be learned

Abstract

In three recent letters to the editor of this journal, Drs Hauben and Reich undertake a performance comparison of two methods to detect over-represented associations of drug–event combinations (‘signals’) in the Adverse Events Reporting System (AERS) database maintained by the US Food and Drug Administration (FDA) [1–3]. The two methods discussed in these letters, the Multi-item Gamma Poisson Shrinker (MGPS) [4–6] and the Proportional Reporting Ratio (PRR), can be used to classify adverse events as signals based on the disproportionality of these events in databases. The three letters acknowledge the potential utility of disproportionality analyses as a pharmacovigilance tool and thus seek to compare the utility of the two methods in signal detection. AERS contains over 2.5 million adverse event reports spontaneously submitted by health care providers, pharmaceutical companies, and the public since 1968. For coding adverse events AERS currently utilizes the Medical Dictionary for Regulatory Activities (MedDRA) classification system with over 15 000 preferred terms (PT). AERS currently has about 10 000 PTs and 4000 decoded generic drug names in use; thus, approximately 43 million drug–event combinations (DECs) are possible in this database. However, considered as a two-way (drug-by-event) table, the AERS database is quite sparsely populated – approximately 2.8 million (0.7%) of approximately 43 million possible DECs have ever been reported. A large proportion (67%) of the 2.8 million DECs that have ever been reported contain fewer than three reports, and approximately half of the DECs exist only once [5]. The sparsity of AERS is important to consider when comparing MGPS and PRR [4, 5]. In the first letter [1], the author selects for analysis the currently labelled association between trimethoprim and hyperkalaemia, and assumes this DEC to be detectable as early as 1979 in the AERS data. In the second letter [2], the authors select the association of pancreatitis with various drugs based upon ‘definite causal relationships’ from external published reports and observational studies and assume that signals should be detectable during early, but unspecified time periods in AERS. In the third letter [3], the authors select the association of rhabdomyolysis with four anti-infectives (pentamidine, isoniazid, trimethoprim/sulfamethoxazole, and lamivudine) based upon at least two published case reports out of 765 Medline citations of rhabdomyolysis with drug products and assume that signals should be detectable during early, but unspecified time periods in AERS. In this third letter, the authors assert that they selected ‘replicated’ findings (i.e. two drug-specific case reports) in the published literature. However, the authors fail to mention in this letter that the specific drug–event in the title, ‘rhabdomyolysis with pentamidine’[3], never reaches an n > 1 throughout all the years of suspect cases in AERS. In these three letters and in similar publications by the same authors [7–14] the authors assume that the DECs they selected should be signalled in the AERS data, and assume that the DECs are true causal associations if either MGPS or PRR signals them in AERS. The authors also assume that if the selected DECs are not signalled by either MGPS or PRR in AERS, the method has failed to signal true positive associations. In this paper we discuss the flaws with three major aspects of the comparative analyses used by these authors [1–3,7–14]: (i) the disparate decision rules these authors choose to define signals for each method; (ii) the focus of the analyses on generating additional signals while excluding an analysis of specificity; and (iii) the use of a stratified MGPS vs. an unstratified PRR.