Real-world evidence: Methods for assessing long-term health and effectiveness of allergy immunotherapy

Abstract

Recently, scientific societies and regulatory authorities have stressed the need for more robust real-world evidence (RWE) in allergic respiratory disease.1FDASubmitting documents using real-world data and real-world evidence to FDA for drugs and biologics guidance for industry May 2019.https://www.fda.gov/regulatory-information/search-fda-guidance-documents/submitting-documents-using-real-world-data-and-real-world-evidence-fda-drugs-and-biologics-guidanceDate accessed: December 17, 2021Google Scholar, 2Paoletti G. Di Bona D. Chu D.K. Firinu D. Heffler E. Agache I. et al.Allergen immunotherapy: the growing role of observational and randomized trial “Real-World Evidence”.Allergy. 2021; 76: 2663-2672Crossref PubMed Scopus (12) Google Scholar, 3Roche N. Anzueto A. Bosnic Anticevich S. Kaplan A. Miravitlles M. Ryan D. et al.The importance of real-life research in respiratory medicine: manifesto of the Respiratory Effectiveness Group: endorsed by the International Primary Care Respiratory Group and the World Allergy Organization.Eur Respir J. 2019; 5419011511Crossref Scopus (22) Google Scholar, 4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Despite the importance of complementing randomized clinical trials (RCTs) with high-quality RWE, and although a relative increase in the numbers of RWE studies (RWSs) in allergy immunotherapy (AIT) has been seen in recent years, the number of RWSs conducted in AIT is still limited4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar (Fig 1). The highest level of evidence is suggested if coexisting RCTs and observational studies provide consistent findings.2Paoletti G. Di Bona D. Chu D.K. Firinu D. Heffler E. Agache I. et al.Allergen immunotherapy: the growing role of observational and randomized trial “Real-World Evidence”.Allergy. 2021; 76: 2663-2672Crossref PubMed Scopus (12) Google Scholar,5Gershon A.S. Jafarzadeh S.R. Wilson K.C. Walkey A.J. clinical knowledge from observational studies. Everything you wanted to know but were afraid to ask.Am J Respir Crit Care Med. 2018; 198: 859-867Crossref PubMed Scopus (31) Google Scholar If appropriate methods and standardized protocols are applied in RWSs there is the opportunity to transform real-world data into evidence of high clinical relevance consistent with the 2021 position of the European Academy of Allergy and Clinical Immunology.2Paoletti G. Di Bona D. Chu D.K. Firinu D. Heffler E. Agache I. et al.Allergen immunotherapy: the growing role of observational and randomized trial “Real-World Evidence”.Allergy. 2021; 76: 2663-2672Crossref PubMed Scopus (12) Google Scholar However, even if RWSs are of good quality and scientifically reliable, the objective will be to assess associations, and in the case of AIT, studies are commonly conducted after a medicine has been marketed. RCTs are still needed to assess causal effects, which currently are the basis for obtaining regulatory approval and marketing authorizations for AIT. As the robustness of RWSs can be hampered by several factors, this article aims to identify the caveats and emphasize how the quality of RWSs can be improved by mirroring certain methodologic steps from RCTs. Selecting the appropriate data source is central for the quality of generated data. As heterogeneity of the study population is an intrinsic characteristic of RWSs, large cohorts are required. RWE may be generated prospectively through primary data collection or retrospectively by using secondary data sources (eg, registries, health care claims, or prescription databases), which have become more complete and comprehensive in recent years. Faced with the challenge of obtaining large enough sample sizes in AIT and the importance of looking at outcomes over long time horizons as AIT is a disease-modifying treatment, retrospective data sources are highly suitable. Although routinely collected health care data offer advantages in providing access to large representative samples of patients in routine clinical practice along with the potential for long-term follow-up, the data quality of such sources can be a challenge (ie, owing to erroneous data entries and incomplete data). Recent publications have discussed strengths and limitations of such data sources in AIT.2Paoletti G. Di Bona D. Chu D.K. Firinu D. Heffler E. Agache I. et al.Allergen immunotherapy: the growing role of observational and randomized trial “Real-World Evidence”.Allergy. 2021; 76: 2663-2672Crossref PubMed Scopus (12) Google Scholar,6Devillier P. Demoly P. Molimard M. Allergen immunotherapy: what is the added value of real-world evidence from retrospective claims database studies?.Expert Rev Respir Med. 2020; 14: 445-452Crossref PubMed Scopus (10) Google Scholar Because of differences in data collection, data protection issues, and restrictions in access to data sources, combining retrospective data sources globally remains a challenge. In AIT specifically, an additional challenge is the difference in treatment modalities and products available across the world. The following sections will focus on retrospective data sources and the assessment of effectiveness in RWSs. Conducting a high-quality and scientifically reliable RWS requires rigorous methodology largely mirroring what is done in RCTs, although without randomization, which alone provides unbiased estimates (Table I). Quality standards for reporting RCTs are well described in the Consolidated Standards of Reporting Trials (CONSORT) statement.7Schulz K.F. Altman D.G. Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.BMJ. 2010; 340: c332Crossref PubMed Scopus (4162) Google Scholar Frameworks to assess the quality of RWSs are described by the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist8von Elm E. Altman D.G. Egger M. Pocock S.J. Gøtzsche P.C. Vandenbroucke J.P. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.PLoS Med. 2007; 4: e296Crossref PubMed Scopus (2064) Google Scholar or the Real Life Evidence Assessment Tool (RELEVANT) tool, with key principles reflecting the requirements for RCTs.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google ScholarTable IKey methodologic aspects for assessing the effects of AIT in retrospective RWSs of high quality by mirroring RCTsMethodologic aspectRCTsRetrospective real-world evidence studiesTransparencyPreregister the study protocol in a public registry or report the study design in a publicationPreregister the study protocol in a public registry or report the study design in a publicationReport conflicts of interest and follow ICMJE authorship recommendationsReport conflicts of interest and follow ICMJE authorship recommendationsAdhere to reporting standards (eg, CONSORT)Adhere to reporting standards (eg, STROBE)Minimize risk of bias and confoundingPrespecify eligibility criteria, outcomes, and statistical analyses planPrespecify eligibility criteria, outcomes, and statistical analyses planEnsure comparable groups through randomizationEnsure comparable groups through appropriate methods (eg, propensity score matching)AnalysesEfficacy of AIT is assessed by using daily diaries and often reported as the total combined score, daily symptoms score, daily medication score, and Rhinitis Quality of Life Questionnaire scoreEffectiveness of AIT can be assessed by using proxies such as prescriptions, diagnosis codes, health care resource utilization, and costDatabase lock before analyses and/or unblindingCohort lock before analysesInterpretationIdentify key strengths and limitations and critically discuss resultsIdentify key strengths and limitations, and critically discuss results in the context of existing RCT evidenceCONSORT, Consolidated Standards of Reporting Trials; ICMJE, International Committee of Medical Journal Editors; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology. Open table in a new tab CONSORT, Consolidated Standards of Reporting Trials; ICMJE, International Committee of Medical Journal Editors; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology. Hypotheses and research questions must be prespecified to avoid post hoc “fishing” for interesting outcomes.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Transparency about the study design and analysis before execution is key and requires preregistration of the RWS in a public registry or publication of the study design and a prespecified statistical analysis plan.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Similar to an RCT, the prespecified analysis plan should include considerations about estimands and handling of intercurrent events and should also address the sample size and statistical power needed to detect differences. As in an RCT in which the population is selected according to eligibility criteria and the database is locked before unblinding of the results, an RWS should have prespecified enrollment criteria, and study cohorts can be locked before testing of the protocol-defined hypotheses. To minimize the risk of bias and confounding, it is important to identify, control for, and address potential confounders in retrospective database studies.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Because of the absence of randomization, retrospective studies are subject to confounding by indication and/or disease severity in those cases in which exposure is associated with additional unmeasured risk (eg, in AIT studies, patients with allergic rhinitis [AR] who are treated with AIT are likely to have more severe AR than are those patients with AR who are not treated with AIT). Rosenbaum and Rubin, who derived the methods for propensity score analyses, pointed out that to be valid, exposure to the treatment of interest should be “strongly ignorable” as a source of additional information on underlying risk.9Rosenbaum P.R. Rubin D.B. The central role of the propensity score in observational studies for causal effects.Biometrika. 1983; 70: 41-55Crossref Scopus (15958) Google Scholar Hence, if a treatment is routinely given to higher-risk subjects, that risk must be described separately and accounted for.9Rosenbaum P.R. Rubin D.B. The central role of the propensity score in observational studies for causal effects.Biometrika. 1983; 70: 41-55Crossref Scopus (15958) Google Scholar The more dissimilar the compared groups, the greater the risk for confounding, and ideally, matching designs such as propensity score matching or instrumental variable techniques are used.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Comparison of the study groups at baseline before and after matching is important.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar If key confounding variables differ at baseline, there is a higher likelihood for substantial confounding in the results.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar To assess validity and potential confounding, RWSs should aim to replicate the findings of RCTs in similar populations where possible, before bridging into different populations and longer follow-up. The primary outcome(s) must be defined and measured in a valid way. In retrospective studies, it is often necessary to use proxies for outcomes of effectiveness or disease severity (eg, in the case of AIT studies, the prescription medications for AR and asthma, as well as confirmed diagnoses, are used as proxies of disease severity in the lack of symptom scores).4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar RWS results should be presented for primary and secondary outcomes.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Study quality is enhanced if the outcomes are reported as absolute and relative measures.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar The uncertainty of the findings (CIs) should be reported, as should the results of sensitivity analyses.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Findings should be adequately discussed in line with available information and their clinical relevance.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Limitations, including potential biases and confounding factors, should be described following a discussion on how these may influence the results.4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar Any conflict of interest must be transparently reported,4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar and measures should be taken to mitigate the conflict of interest (eg, through involvement of third parties in the design, conduct, and analysis).4Roche N. Campbell J.D. Krishnan J.A. Brusselle G. Chisholm A. Bjermer L. et al.Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group-European Academy of Allergy and Clinical Immunology Task Force.Clin Transl Allergy. 2019; 9: 20Crossref PubMed Scopus (13) Google Scholar The aim of the REACT study was to assess the effectiveness of AIT and provide high-quality RWE regarding how AIT works in the long term and in real life.10Fritzsching B. Contoli M. Porsbjerg C. Buchs S. Larsen J.R. Elliott L. et al.Long-term real-world effectiveness of allergy immunotherapy in patients with allergic rhinitis and asthma: results from the REACT study, a retrospective cohort study.Lancet Reg Health Eur. 2021; 13100275PubMed Google Scholar In this retrospective database study, rigorous methodology was applied, comprising a prespecified objective and processes that were described in the protocol, preregistration at ClinicalTrials.gov, and cohort lock before outcomes analyses; in addition, all analyses were conducted by an independent third party. Propensity score matching was used to ensure comparable groups and mitigate confounding. Subjects were matched 1:1 by using the nearest neighbour approach without replacement with a caliper of 0.01, based on 30 variables available in the database, including demographics, diagnosis codes for relevant comorbidities, prescriptions for AR and asthma medication, health resource utilization, and costs. The standardized mean differences for variables included in the matching had to be less than 10%. The study found that AIT was associated with greater reductions in AR and asthma prescriptions (both controller and reliever medication) compared to control subjects.10Fritzsching B. Contoli M. Porsbjerg C. Buchs S. Larsen J.R. Elliott L. et al.Long-term real-world effectiveness of allergy immunotherapy in patients with allergic rhinitis and asthma: results from the REACT study, a retrospective cohort study.Lancet Reg Health Eur. 2021; 13100275PubMed Google Scholar The AIT group had a significantly greater likelihood of stepping down asthma treatment (P < .0001).10Fritzsching B. Contoli M. Porsbjerg C. Buchs S. Larsen J.R. Elliott L. et al.Long-term real-world effectiveness of allergy immunotherapy in patients with allergic rhinitis and asthma: results from the REACT study, a retrospective cohort study.Lancet Reg Health Eur. 2021; 13100275PubMed Google Scholar On top of the reduction in asthma treatment, a greater reduction in severe asthma exacerbations was seen in the AIT group (P < .05).10Fritzsching B. Contoli M. Porsbjerg C. Buchs S. Larsen J.R. Elliott L. et al.Long-term real-world effectiveness of allergy immunotherapy in patients with allergic rhinitis and asthma: results from the REACT study, a retrospective cohort study.Lancet Reg Health Eur. 2021; 13100275PubMed Google Scholar Reductions in pneumonia managed with antibiotic prescriptions, hospitalizations, and duration of inpatients stays all favored AIT.10Fritzsching B. Contoli M. Porsbjerg C. Buchs S. Larsen J.R. Elliott L. et al.Long-term real-world effectiveness of allergy immunotherapy in patients with allergic rhinitis and asthma: results from the REACT study, a retrospective cohort study.Lancet Reg Health Eur. 2021; 13100275PubMed Google Scholar The findings of the REACT study complement the existing evidence from RCTs in AIT and support the clinical decision making regarding use of AIT for the treatment and sustained control of AR and asthma.10Fritzsching B. Contoli M. Porsbjerg C. Buchs S. Larsen J.R. Elliott L. et al.Long-term real-world effectiveness of allergy immunotherapy in patients with allergic rhinitis and asthma: results from the REACT study, a retrospective cohort study.Lancet Reg Health Eur. 2021; 13100275PubMed Google Scholar With an increasing amount of data, researchers have unique opportunities to generate valid and good-quality RWE of the effectiveness of AIT with the application of rigorous and high scientific standards. Although the focus herein was retrospective RWS, other RWE study types will likely benefit from using clinical RCT knowledge. Altogether, robust assessments of the effectiveness of AIT in RWSs complement the existing evidence of effect and safety of AIT in RCTs.