Tree-based Scan Statistic Research Articles

Mining clinical data for novel medications to treat alcohol use disorder

BackgroundAlcohol use disorder (AUD) is a highly prevalent and often debilitating condition associated with high morbidity and mortality. Current AUD medications have limited efficacy and uptake. Alternative pharmacological options are needed. MethodsWe constructed a mechanistic tree of all US Food and Drug Administration approved medications and used a tree-based scan statistic, TreeScan, to identify medications associated with greater than expected improvements in alcohol consumption. Our cohort included all United States (US) Department of Veterans Affairs (VA) patients with a diagnosis of AUD between 10/1/1999 and 9/30/2019 with multiple Alcohol Use Disorders Identification Test-Consumption Module scores within the VA electronic health record data. ResultsMedications statistically associated with decreased alcohol consumption had, at large, minor effect sizes. Medications used in the treatment of chronic or life-threatening conditions like diabetes, chronic kidney disease, hepatitis C virus, or cancer produced larger effect sizes. Asenapine, an atypical antipsychotic, had a large effect with an observed to expected ratio of 1.78 (p = 0.003). Our findings were replicated in a propensity score matched population. ConclusionMost medications significantly associated with decreased alcohol consumption in our analysis were either contraindicated with alcohol or likely attributable to patients abstaining from alcohol due to severe illness. However, the large effect of asenapine is notable, and a worthwhile candidate for more careful analysis.

Signal detection statistics of adverse drug events in hierarchical structure for matched case-control data.

The tree-based scan statistic is a data mining method used to identify signals of adverse drug reactions in a database of spontaneous reporting systems. It is particularly beneficial when dealing with hierarchical data structures. One may use a retrospective case-control study design from spontaneous reporting systems (SRS) to investigate whether a specific adverse event of interest is associated with certain drugs. However, the existing Bernoulli model of the tree-based scan statistic may not be suitable as it fails to adequately account for dependencies within matched pairs. In this article, we propose signal detection statistics for matched case-control data based on McNemar's test, Wald test for conditional logistic regression, and the likelihood ratio test for a multinomial distribution. Through simulation studies, we demonstrate that our proposed methods outperform the existing approach in terms of the type I error rate, power, sensitivity, and false detection rate. To illustrate our proposed approach, we applied the three methods and the existing method to detect drug signals for dizziness-related adverse events related to antihypertensive drugs using the database of the Korea Adverse Event Reporting System.

Signal detection of adverse events associated with gabapentinoid use for chronic pain.

Gabapentinoids (GABA) prescribing as a potential and conceivably safer substitute for opioids has substantially increased. Understanding all potential adverse drug events (ADEs) associated with GABA will guide clinical decision-making for pain management. A 20% sample of Medicare enrollees with new chronic pain diagnoses in 2017-2018 was selected. GABA users were those with >=30 consecutive days prescription in a year without opioid prescription. Opioid users were similarly defined. The control group used neither of these drugs. Propensity score match across three groups based on demographics and comorbidity was performed. We used proportional reporting ratio (PRR), Gamma Poisson Shrinker, and tree-based scan statistic (TBSS) to detect ADEs within 3, 6, and 12 months of follow-up. Immunity disorder was detected within 3 months of follow-up by PRR compared to opioid use (PRR:2.33), and by all three methods compared to controls. Complications of transplanted organs/tissues and schizophrenia spectrum/other psychotic disorders were consistently detected by PRR and TBSS within 3 months. Skin disorders were detected by TBSS; and stroke was detected by PRR within 3 months compared to opioid use (PRR:4.74). Some malignancies were detected by PRR within 12 months. Other signals detected in GABA users were neuropathy and nerve disorders. Our study identified expected and unexpected ADE signals in GABA users. Neurological signals likely related to indications for GABA use. Signals for immunity, mental/behavior, and skin disorders were found in the FDA adverse event reporting system database. Unexpected signals of stroke and cancer require further confirmatory analyses to verify.

A machine learning-based phenotype for long COVID in children: An EHR-based study from the RECOVER program.

As clinical understanding of pediatric Post-Acute Sequelae of SARS CoV-2 (PASC) develops, and hence the clinical definition evolves, it is desirable to have a method to reliably identify patients who are likely to have post-acute sequelae of SARS CoV-2 (PASC) in health systems data. In this study, we developed and validated a machine learning algorithm to classify which patients have PASC (distinguishing between Multisystem Inflammatory Syndrome in Children (MIS-C) and non-MIS-C variants) from a cohort of patients with positive SARS- CoV-2 test results in pediatric health systems within the PEDSnet EHR network. Patient features included in the model were selected from conditions, procedures, performance of diagnostic testing, and medications using a tree-based scan statistic approach. We used an XGboost model, with hyperparameters selected through cross-validated grid search, and model performance was assessed using 5-fold cross-validation. Model predictions and feature importance were evaluated using Shapley Additive exPlanation (SHAP) values. The model provides a tool for identifying patients with PASC and an approach to characterizing PASC using diagnosis, medication, laboratory, and procedure features in health systems data. Using appropriate threshold settings, the model can be used to identify PASC patients in health systems data at higher precision for inclusion in studies or at higher recall in screening for clinical trials, especially in settings where PASC diagnosis codes are used less frequently or less reliably. Analysis of how specific features contribute to the classification process may assist in gaining a better understanding of features that are associated with PASC diagnoses.

Drug safety signal detection in a regional healthcare database using the tree-based scan statistic and comparison to 3 other mining methods.

To evaluate and compare the relative performance of the tree-based scan statistic (TreeScan) with the crude cohort study, Bayesian confidence propagation neural network (BCPNN) and Gamma Poisson Shrinker (GPS) in detecting statin-related adverse events (AEs) in an electronic healthcare database. Data from a Chinese healthcare database from 2010 to 2016 were evaluated. We identified statin users based on prescription information in their out-/in-patient records, and AEs were defined according to the ICD-10 codes in patients' diagnosis records. TreeScan was applied to detect AE signals related to statin use and was compared with 3 other methods based on sensitivity, specificity, positive predictive value, negative predictive value, accuracy, the Youden index, area under the precision-recall curve and the area under the receiver operating characteristic curve. A total of 224 187 patients were enrolled and divided into 85 758 statin users and 138 429 nonusers. TreeScan generated 29 positive signals, of which 9 were known AEs. The sensitivities of TreeScan, BCPNN and GPS were all 69.2%, which was higher than that of the crude cohort study (46%). The specificity (82.3%), positive predictive value (31.0%), negative predictive value (95.9%), accuracy (81.0%), Youden index (51.5%) and area under the receiver operating characteristic curve (75.8%) of TreeScan were the highest among the 4 methods. TreeScan outperformed the crude cohort, BCPNN and GPS in detecting statin-related AEs in an electronic healthcare database. Therefore, it can be used as a complementary tool for other signal detection methods in drug safety surveillance.

Understanding pediatric long COVID using a tree-based scan statistic approach: an EHR-based cohort study from the RECOVER Program.

Post-acute sequalae of SARS-CoV-2 infection (PASC) is not well defined in pediatrics given its heterogeneity of presentation and severity in this population. The aim of this study is to use novel methods that rely on data mining approaches rather than clinical experience to detect conditions and symptoms associated with pediatric PASC. We used a propensity-matched cohort design comparing children identified using the new PASC ICD10CM diagnosis code (U09.9) (N = 1309) to children with (N = 6545) and without (N = 6545) SARS-CoV-2 infection. We used a tree-based scan statistic to identify potential condition clusters co-occurring more frequently in cases than controls. We found significant enrichment among children with PASC in cardiac, respiratory, neurologic, psychological, endocrine, gastrointestinal, and musculoskeletal systems, the most significant related to circulatory and respiratory such as dyspnea, difficulty breathing, and fatigue and malaise. Our study addresses methodological limitations of prior studies that rely on prespecified clusters of potential PASC-associated diagnoses driven by clinician experience. Future studies are needed to identify patterns of diagnoses and their associations to derive clinical phenotypes. We identified multiple conditions and body systems associated with pediatric PASC. Because we rely on a data-driven approach, several new or under-reported conditions and symptoms were detected that warrant further investigation.

Sequential Data-Mining for Adverse Events After Recombinant Herpes Zoster Vaccination Using the Tree-Based Scan Statistic.

Tree-based scan statistics have been successfully used to study the safety of several vaccines without prespecifying health outcomes of concern. In this study, the binomial tree-based scan statistic was applied sequentially to detect adverse events in days 1-28 compared with days 29-56 after recombinant herpes zoster (RZV) vaccination, with 5 looks at the data and formal adjustment for the repeated analyses over time. IBM MarketScan data on commercially insured persons ≥50 years of age receiving RZV during January 1, 2018, to May 5, 2020, were used. With 999,876 doses of RZV included, statistically significant signals were detected only for unspecified adverse effects/complications following immunization, with attributable risks as low as 2 excess cases per 100,000 vaccinations. Ninety percent of cases in the signals occurred in the week after vaccination and, based on previous studies, likely represent nonserious events like fever, fatigue, and headache. Strengths of our study include its untargeted nature, self-controlled design, and formal adjustment for repeated testing. Although the method requires prespecification of the risk window of interest and may miss some true signals detectable using the tree-temporal variant of the method, it allows for early detection of potential safety problems through early initiation of ongoing monitoring.

A tree-based scan statistic for zero-inflated count data in post-market drug safety surveillance

After new drugs enter the market, adverse events (AE) induced by their use must be tracked; rare AEs may not be detected during clinical trials. Some organizations have been collecting information on suspected drugs and AEs via a spontaneous reporting system to conduct post-market drug safety surveillance. These organizations use the information to detect a signal representing potential causality between drugs and AEs. The drug and AE data are often hierarchically structured. Accordingly, the tree-based scan statistic can be used as a statistical data mining method for signal detection. Most of the AE databases contain a large number of zero-count cells. Notably, not only an observational zero from the Poisson distribution, but also a true zero exists in zero-count cells. True zeros represent theoretically impossible observations or possible but unreported observations. The existing tree-based scan statistic assumes that all zeros are zero-valued observations from the Poisson distribution. Therefore, true zeros are not considered in the modeling, which can lead to bias in the inferences. In this study, we propose a tree-based scan statistic for zero-inflated count data in a hierarchical structure. According to our simulation study, in the presence of excess zeros, our proposed tree-based scan statistic provides better performance than the existing tree-based scan statistic. The two methods were illustrated using Korea Adverse Event Reporting System data from the Korea Institute of Drug Safety and Risk Management.

Signal Detection of Adverse Events Associated with Trastuzumab in a Cohort of Elderly Patients with Breast Cancer.

AimUtilization of signal detection methods in longitudinal claims data can improve post-marketing drug surveillance, but to date there has been limited application. The aim of this study is to use 3 approaches, the proportional reporting ratio, Gamma Poisson Shrinker, and tree-based scan statistic in detecting adverse drug events (ADEs) attributed to trastuzumab using an administrative claims dataset.MethodsUsing data from the Texas Cancer Registry and SEER linked to Medicare from 2010 to 2013, we conducted 1:2 propensity score matching. Breast cancer HER2+ patients treated with trastuzumab in addition to standard chemotherapy were matched to HER2– patients treated with standard chemotherapy. Inpatient and outpatient encounters up to 6 months from start of therapy were used to identify adverse events.ResultsA total of 4191 patients were included in the study. Across all methods, use of trastuzumab generated signals on 9 distinct body systems. Cardiomyopathy and heart valve disease were the most consistently detected signals. Clinical review determined that most signals represented known ADEs.ConclusionsWe showed that claims data can be used to complement current ADE monitoring using common data mining methods with propensity score matching. Our analysis identified all expected ADEs associated with trastuzumab, and additional signals of valvular heart disorders.

Mining Clinical Data for Novel Posttraumatic Stress Disorder Medications

BackgroundDespite the prevalence and negative impact of posttraumatic stress disorder (PTSD), there are few medications approved by the U.S. Food and Drug Administration for treatment, and approved medications do not work well enough. We leveraged large-scale electronic health record data to identify existing medications that may be repurposed as PTSD treatments. MethodsWe constructed a mechanistic tree of all Food and Drug Administration–approved medications and used the tree-based scan statistic to identify medications associated with greater than expected levels of clinically meaningful improvement in PTSD symptoms using electronic health record data from the U.S. Department of Veterans Affairs. Our cohort included patients with a diagnosis of PTSD who had repeated symptom measurements using the PTSD Checklist over a 20-year period (N = 168,941). We calculated observed numbers based on patients taking each drug or mechanistically related class of drugs and the expected numbers based on the tree as a whole. ResultsMedications typically used to treat PTSD, such as the Food and Drug Administration–approved agent sertraline, were associated with improvement in PTSD symptoms, but the effects were small. Several, but not all, direct-acting antivirals used in the treatment of hepatitis C virus demonstrated a strong association with PTSD improvement. The finding was robust to a sensitivity analysis excluding patients who received established PTSD treatments, including trauma-focused psychotherapy, concurrent with hepatitis treatment. ConclusionsOur exploratory approach both demonstrated findings that are consistent with what is known about pharmacotherapy for PTSD and uncovered a novel class of medications that may improve PTSD symptoms.

Review on tree-based scan statistic in drug and vaccine safety monitoring

To summarize the development and application of tree-based scan statistic (TreeScan), explain the methodology and provide a reference for future use of this method by reviewing the original pharmacoepidemiological and vaccine studies using the TreeScan. Medline, Embase and Web of Science databases were used for the retrieval of eligible studies using keywords related to TreeScan. A total of 15 eligible studies were included, in which 9 studies explored the adverse events of drugs and 6 studies focused on the safety of vaccines. Three types of models (Poisson probability model, Bernoulli probability model and tree-temporal scan statistic model) of TreeScan were used. The major differences among the three models were 1) whether predefined control was used according to research question, 2) whether the time from exposure to onset of adverse events was considered. Several studies explored its ability by comparing with other methods for adverse event detection or by using known adverse events. This review shows that TreeScan is an effective method for the safety signal detection of drugs or vaccines, which develops rapidly and globally. It is very necessary to promote its use in drug safety monitoring and other related fields in China.

A novel data mining application to detect safety signals for newly approved medications in routine care of patients with diabetes.

ABSTRACTBackgroundClinical trials are often underpowered to detect serious but rare adverse events of a new medication. We applied a novel data mining tool to detect potential adverse events of canagliflozin, the first sodium glucose co‐transporter 2 (SGLT2 inhibitor) in the United States, using real‐world data from shortly after its market entry and before public awareness of its potential safety concerns.MethodsIn a U. S. commercial claims dataset (29 March 2013–30 Sept 2015), two pairwise cohorts of patients over 18 years of age with type 2 diabetes (T2D) who were newly dispensed canagliflozin or an active comparator, that is a dipeptidyl peptidase 4 inhibitor (DPP4) or a glucagon‐like peptide 1 receptor agonist (GLP1), were identified and propensity score‐matched. We used variable ratio matching with up to four people receiving a DPP4 or GLP1 for each person receiving canagliflozin. We identified potential safety signals using a hierarchical tree‐based scan statistic data mining method with the hierarchical outcome tree constructed based on international classification of disease coding. We screened for incident adverse events where there were more outcomes observed among canagliflozin vs. comparator initiators than expected by chance, after adjusting for multiple testing.ResultsWe identified two pairwise propensity score variable ratio matched cohorts of 44,733 canagliflozin vs. 99,458 DPP4 initiators, and 55,974 canagliflozin vs. 74,727 GLP1 initiators. When we screened inpatient and emergency room diagnoses, diabetic ketoacidosis was the only severe adverse event associated with canagliflozin initiation with p < .05 in both cohorts. When outpatient diagnoses were also considered, signals for female and male genital infections emerged in both cohorts (p < .05).Conclusions and relevanceIn a large population‐based study, we identified known but no other adverse events associated with canagliflozin, providing reassurance on its safety among adult patients with T2D and suggesting the tree‐based scan statistic method is a useful post‐marketing safety monitoring tool for newly approved medications.

A General Propensity Score for Signal Identification Using Tree-Based Scan Statistics

The tree-based scan statistic (TreeScan; Martin Kulldorff, Harvard Medical School, Boston, Massachusetts) is a data-mining method that adjusts for multiple testing of correlated hypotheses when screening thousands of potential adverse events for signal identification. Simulation has demonstrated the promise of TreeScan with a propensity score (PS)-matched cohort design. However, it is unclear which variables to include in a PS for applied signal identification studies to simultaneously adjust for confounding across potential outcomes. We selected 4 pairs of medications with well-understood safety profiles. For each pair, we evaluated 5 candidate PSs with different combinations of 1) predefined general covariates (comorbidity, frailty, utilization), 2) empirically selected (data-driven) covariates, and 3) covariates tailored to the drug pair. For each pair, statistical alerting patterns were similar with alternative PSs (≤11 alerts in 7,996 outcomes scanned). Inclusion of covariates tailored to exposure did not appreciably affect screening results. Inclusion of empirically selected covariates can provide better proxy coverage for confounders but can also decrease statistical power. Unlike tailored covariates, empirical and predefined general covariates can be applied "out of the box" for signal identification. The choice of PS depends on the level of concern about residual confounding versus loss of power. Potential signals should be followed by pharmacoepidemiologic assessment where confounding control is tailored to the specific outcome(s) under investigation.

Data Mining for Adverse Events of Tumor Necrosis Factor-Alpha Inhibitors in Pediatric Patients: Tree-Based Scan Statistic Analyses of Danish Nationwide Health Data.

Background and ObjectivesTumor necrosis factor-alpha (TNF-α) inhibitors are efficacious and considered generally safe in adults. However, pediatric-specific safety evidence is scarce. The aim of this study was to screen for signals of previously unknown adverse events of TNF-α inhibitors in pediatric patients.MethodsWe conducted a data-mining study based on routinely collected, nationwide Danish healthcare data for 2004–2016. Using tree-based scan statistics to identify events with unexpectedly high incidence during TNF-α inhibitor use among patients with inflammatory bowel disease or juvenile idiopathic arthritis, two analyses were performed: comparison with episodes of no use and with other time periods from the same patient. Based on incident physician-assigned diagnosis codes from outpatient and inpatient visits in specialist care, we screened thousands of potential adverse events while adjusting for multiple testing.ResultsWe identified 1310 episodes of new TNF-α inhibitor use that met the eligibility criteria. Two signals of adverse events of TNF-α inhibitors, as compared with no use, were detected. First, there were excess events of dermatologic complications (ICD-10: L00-L99, 87 vs. 44 events, risk difference [RD] 3.3%), which have been described previously in adults and children. Second, there were excess events of psychiatric diagnosis adjustment disorders (ICD-10: F432, 33 vs. 7 events, RD 2.0%), which was likely associated with the underlying disease and its severity, rather than with the treatment. The self-controlled analysis generated no signal.ConclusionsNo signals of previously unknown adverse events of TNF-α inhibitors in pediatric patients were detected. The study showed that real-world data and newly developed methods for adverse events data mining can play a particularly important role in pediatrics where pre-approval drug safety data are scarce.Electronic supplementary materialThe online version of this article (10.1007/s40261-020-00977-5) contains supplementary material, which is available to authorized users.

Comparison of Data Mining Methods for the Signal Detection of Adverse Drug Events with a Hierarchical Structure in Postmarketing Surveillance.

There are several different proposed data mining methods for the postmarketing surveillance of drug safety. Adverse events are often classified into a hierarchical structure. Our objective was to compare the performance of several of these different data mining methods for adverse drug events data with a hierarchical structure. We generated datasets based on the World Health Organization’s Adverse Reaction Terminology (WHO-ART) hierarchical structure. We evaluated different data mining methods for signal detection, including several frequentist methods such as reporting odds ratio (ROR), proportional reporting ratio (PRR), information component (IC), the likelihood ratio test-based method (LRT), and Bayesian methods such as gamma Poisson shrinker (GPS), Bayesian confidence propagating neural network (BCPNN), the new IC method, and the simplified Bayesian method (sB), as well as the tree-based scan statistic through an extensive simulation study. We also applied the methods to real data on two diabetes drugs, voglibose and acarbose, from the Korea Adverse event reporting system. Only the tree-based scan statistic method maintained the type I error rate at the desired level. Likelihood ratio test-based methods and Bayesian methods tended to be more conservative than other methods in the simulation study and detected fewer signals in the real data example. No method was superior to the others in terms of the statistical power and sensitivity of detecting true signals. It is recommended that those conducting drug‒adverse event surveillance use not just one method, but make a decision based on several methods.

33-OR: A Novel Data Mining Approach to Detect Adverse Events of New Diabetes Drugs in Routine Care

Clinical trials are underpowered to detect serious but rare adverse events (AEs). We applied a novel data mining approach to detect potential AEs for canagliflozin (CANA), the first marketed SGLT2 inhibitor, prior to public awareness of its potential safety concerns. In a U.S. commercial claims dataset (3/2013-9/2015), two pair-wise cohorts of patients with T2DM initiating CANA or a comparator, i.e., a DPP-4i or a GLP-1RA, were identified and propensity score matched. We used variable ratio matching with up to 4 comparators for each CANA initiator (44,733 CANA vs. 99,458 DPP-4i; and 55,974 CANA vs. 74,727 GLP-1RA). Using a tree-based scan statistic data mining method, we assessed thousands of incident outcomes in CANA vs. comparator initiators, scanning for statistical AE alerts after adjusting for multiple testing. Incident outcomes were defined by hierarchical groupings of clinically related ICD-9 codes. For inpatient and emergency room diagnoses, diabetic ketoacidosis was the only severe AE associated with CANA (p&amp;lt;0.05 in both cohorts) (Table). When outpatient diagnoses were also considered, alerts for female and male genital infections emerged in both cohorts (p&amp;lt;0.05). The identification of known but no other AEs provides reassurance for the safety of CANA in adults and suggests that the tree-based scan statistic is a useful post-market safety monitoring tool for new diabetes drugs. Disclosure M. Fralick: None. M. Kulldorff: None. S. Wang: Research Support; Self; Boehringer Ingelheim International GmbH, Johnson &amp; Johnson, Novartis Pharmaceuticals Corporation. S. Schneeweiss: None. D. Redelmeier: None. E. Patorno: Other Relationship; Self; Boehringer Ingelheim International GmbH. Funding National Institute on Aging (K08AG055670)

Safety Surveillance of Pneumococcal Vaccine Using Three Algorithms: Disproportionality Methods, Empirical Bayes Geometric Mean, and Tree-Based Scan Statistic.

Introduction: Diverse algorithms for signal detection exist. However, inconsistent results are often encountered among the algorithms due to different levels of specificity used in defining the adverse events (AEs) and signal threshold. We aimed to explore potential safety signals for two pneumococcal vaccines in a spontaneous reporting database and compare the results and performances among the algorithms. Methods: Safety surveillance was conducted using the Korea national spontaneous reporting database from 1988 to 2017. Safety signals for pneumococcal vaccine and its subtypes were detected using the following the algorithms: disproportionality methods comprising of proportional reporting ratio (PRR), reporting odds ratio (ROR), and information component (IC); empirical Bayes geometric mean (EBGM); and tree-based scan statistics (TSS). Moreover, the performances of these algorithms were measured by comparing detected signals with the known AEs or pneumococcal vaccines (reference standard). Results: Among 10,380 vaccine-related AEs, 1135 reports and 101 AE terms were reported following pneumococcal vaccine. IC generated the most safety signals for pneumococcal vaccine (40/101), followed by PRR and ROR (19/101 each), TSS (15/101), and EBGM (1/101). Similar results were observed for its subtypes. Cellulitis was the only AE detected by all algorithms for pneumococcal vaccine. TSS showed the best balance in the performance: the highest in accuracy, negative predictive value, and area under the curve (70.3%, 67.4%, and 64.2%). Conclusion: Discrepancy in the number of detected signals was observed between algorithms. EBGM and TSS calibrated noise better than disproportionality methods, and TSS showed balanced performance. Nonetheless, these results should be interpreted with caution due to a lack of a gold standard for signal detection.

Bacillus Calmette–Guérin (BCG) vaccine safety surveillance in the Korea Adverse Event Reporting System using the tree-based scan statistic and conventional disproportionality-based algorithms

BackgroundSubstantial variations in the safety profiles of different formulations of the bacillus Calmette–Guérin (BCG) vaccine exist. Therefore, we aimed to detect safety signals of BCG vaccine for intradermal injection (BCG-ID) and percutaneous injection (BCG-PC) in the Korea Adverse Event Reporting System (KAERS). MethodsWe conducted a vaccine safety surveillance study from the adverse events (AEs) reported following BCG vaccine in the Korea Institute of Drug Safety and Risk Management KAERS Database (KIDS-KD) between 2005 and 2017. We used the tree-based scan statistic (TSS) and four disproportionality-based algorithms for signal detection: empirical Bayesian geometric mean; proportional reporting ratio; reporting odds ratio; and information component. The detected signals from each algorithm was compared with the known AEs of BCG vaccine (reference standard) to present positive predictive value (PPV) and area under the receiver operating curve (AUC). ResultsFrom the total of 52,191 vaccine-related AE pairs, 963 AE pairs were reported following the BCG vaccine, in which BCG-ID and –PC accounted for 71.1% and 28.9%, respectively. Overall, 97.2% of AE reports were non-serious; lymphadenopathy (583/963; 50.3%) and injection site discharge (193/963; 20.0%) were the most commonly reported AEs detected by all algorithms. Tuberculous osteitis was reported solely from BCG-PC (15/279; 5.4%), while most of the AEs from BCG-ID were related to injection site complications. The TSS demonstrated the best balance of the performance measures, with PPV and AUC of 66.7% and 63.1%, respectively. ConclusionsTSS was successfully applied in the KAERS to detect safety signals of BCG vaccine. In addition, difference in the safety profiles of BCG-ID and BCG-PC warrants further investigation to confirm our findings. Although TSS performed the best in our study, caution should be taken when interpreting the results owing to the lack of a robust “gold standard” and the relatively small number of reports for data mining.

An Implementation and Visualization of the Tree-Based Scan Statistic for Safety Event Monitoring in Longitudinal Electronic Health Data.

Longitudinal electronic healthcare data hold great potential for drug safety surveillance. The tree-based scan statistic (TBSS), as implemented by the TreeScan® software, allows for hypothesis-free signal detection in longitudinal data by grouping safety events according to branching, hierarchical data coding systems, and then identifying signals of disproportionate recording (SDRs) among the singular events or event groups. The objective of this analysis was to identify and visualize SDRs with the TBSS in historical data from patients using two antifungal drugs, itraconazole or terbinafine. By examining patients who used either itraconazole or terbinafine, we provide a conceptual replication of a previous TBSS analyses by varying methodological choices and using a data source that had not been previously used with the TBSS, i.e., the Optum Clinformatics™ claims database. With this analysis, we aimed to test a parsimonious design that could be the basis of a broadly applicable method for multiple drug and safety event pairs. The TBSS analysis was used to examine incident events and any itraconazole or terbinafine use among US-based patients from 2002 through 2007. Event frequencies before and after the first day of drug exposure were compared over 14- and 56-day periods of observation in a Bernoulli model with a self-controlled design. Safety events were classified into a hierarchical tree structure using the Clinical Classifications Software (CCS) which mapped International Classification of Diseases, 9th Revision (ICD-9) codes to 879 diagnostic groups. Using the TBSS, the log likelihood ratio of observed versus expected events in all groups along the CCS hierarchy were compared, and groups of events that occurred at disproportionally high frequencies were identified as potential SDRs; p-values for the potential SDRs were estimated with Monte-Carlo permutation based methods. Output from TreeScan® was visualized and plotted as a network which followed the CCS tree structure. Terbinafine use (n = 223,968) was associated with SDRs for diseases of the circulatory system (14- and 56-day p = 0.001) and heart (14-day p = 0.026 and 56-day p = 0.001) as well as coronary atherosclerosis and other heart disease (14-day p = 0.003 and 56-day p = 0.004). For itraconazole use (n = 36,025), the TBSS identified SDRs for coronary atherosclerosis and other heart disease (p = 0.002) and complications of an implanted or grafted device (14-day p = 0.001 and 56-day p < 0.05). Use of both drugs was associated with SDRs for diseases of the digestive system at 14days (p < 0.05) and this SDR had been observed among terbinafine users in a previous TBSS analysis with a different data source. The TreeScan® visualization facilitated the identification of the atherosclerosis and other heart disease SDRs as well as highlighting the consistency of the SDR for diseases of the digestive system across drugs and data sources. With the TBSS, we identified potential SDRs related to the circulatory system that may reflect the cardiac risk that was described in the itraconazole product label. SDRs for diseases of the digestive system among terbinafine users were also reported in a previous signal detection analysis, although other SDRs from the previous publications were not replicated. The TBSS visualizations aided in the understanding and interpretation of the TBSS output, including the comparisons to the previous publications. In this conceptual replication, differences in the results observed in our analysis and the previous analyses could be attributable to variation in modeling and design choices as well as factors that were intrinsic to the underlying data sources. The broad consistency, but far from perfect concordance, of our results with the known safety profile of these antifungals including the risks from the itraconazole product label supports the rationale for continued investigations of signal detection methods across differing data sources and populations.

Tree-based scan statistic – Application in manufacturing-related safety signal detection

Background and objectivesOver the last decades, medicinal regulations have been put into place and have considerably improved manufacturing practices. Nevertheless, safety issues may still arise. Using the simulation described in this manuscript, our aim is to develop adequate detection methods for manufacturing-related safety signals, especially in the context of biological products. MethodsPharmaceutical companies record the entire batch genealogies, from seed batches over intermediates to final product (FP) batches. We constructed a hierarchical tree based on this genealogy information and linked it to the spontaneous safety data available for the FP batch numbers. The tree-based scan statistic (TBSS) was used on simulated data as a proof of concept to locate the source that may have subsequently generated an excess of specific adverse events (AEs) within the manufacturing steps, and to evaluate the method’s adjustment for multiple testing. All calculations were performed with a customized program in SAS v9.2. ResultsThe TBSS generated a close to expected number of false positive signals, demonstrating that it adjusted for multiple testing. Overall, the method detected 71% of the simulated signals at the correct production step when a 6-fold increase in reports with AEs of interest (AEOI) was applied, and 31% when a 2-fold increase was applied. The relatively low detection performance may be attributed to the higher granularity associated with the lower levels of the hierarchy, leading to a lack of power and the stringent definition criteria that were applied for a true positive result. ConclusionAs a data-mining method for manufacturing-related safety signal detection, the TBSS may provide advantages over other disproportionality analyses (using batch information) but may benefit from complementary methods (not relaying on batch information). While the method warrants further refinement, it may improve safety signal detection and contribute to improvements in the quality of manufacturing processes.