Current Value Frameworks—What's New?

Several organizations have developed frameworks to systematically assess the value of new drugs. To evaluate the convergent validity and interrater reliability of 4 value frameworks to understand the extent to which these tools can facilitate value-based treatment decisions in oncology. Eight panelists used the American Society of Clinical Oncology (ASCO), European Society for Medical Oncology (ESMO), Institute for Clinical and Economic Review (ICER), and National Comprehensive Cancer Network (NCCN) frameworks to conduct value assessments of 15 drugs for advanced lung and breast cancers and castration-refractory prostate cancer. Panelists received instructions and published clinical data required to complete the assessments, assigning each drug a numeric or letter score. Kendall's Coefficient of Concordance for Ranks (Kendall's W) was used to measure convergent validity by cancer type among the 4 frameworks. Intraclass correlation coefficients (ICCs) were used to measure interrater reliability for each framework across cancers. Panelists were surveyed on their experiences. Kendall's W across all 4 frameworks for breast, lung, and prostate cancer drugs was 0.560 (P= 0.010), 0.562 (P = 0.010), and 0.920 (P < 0.001), respectively. Pairwise, Kendall's W for breast cancer drugs was highest for ESMO-ICER and ICER-NCCN (W = 0.950, P = 0.019 for both pairs) and lowest for ASCO-NCCN (W = 0.300, P = 0.748). For lung cancer drugs, W was highest pairwise for ESMO-ICER (W = 0.974, P = 0.007) and lowest for ASCO-NCCN (W = 0.218, P = 0.839); for prostate cancer drugs, pairwise W was highest for ICER-NCCN (W = 1.000, P < 0.001) and lowest for ESMO-ICER and ESMO-NCCN (W = 0.900, P = 0.052 for both pairs). When ranking drugs on distinct framework subdomains, Kendall's W among breast cancer drugs was highest for certainty (ICER, NCCN: W = 0.908, P = 0.046) and lowest for clinical benefit (ASCO, ESMO, NCCN: W = 0.345, P = 0.436). Among lung cancer drugs, W was highest for toxicity (ASCO, ESMO, NCCN: W = 0. 944, P < 0.001) and lowest for certainty (ICER, NCCN: W = 0.230, P = 0.827); and among prostate cancer drugs, it was highest for quality of life (ASCO, ESMO: W = 0.986, P = 0.003) and lowest for toxicity (ASCO, ESMO, NCCN: W = 0.200, P = 0.711). ICC (95% CI) for ASCO, ESMO, ICER, and NCCN were 0.800 (0.660-0.913), 0.818 (0.686-0.921), 0.652 (0.466-0.834), and 0.153 (0.045-0.371), respectively. When scores were rescaled to 0-100, NCCN provided the narrowest band of scores. When asked about their experiences using the ASCO, ESMO, ICER, and NCCN frameworks, panelists generally agreed that the frameworks were logically organized and reasonably easy to use, with NCCN rated somewhat easier. Convergent validity among the ASCO, ESMO, ICER, and NCCN frameworks was fair to excellent, increasing with clinical benefit subdomain concordance and simplicity of drug trial data. Interrater reliability, highest for ASCO and ESMO, improved with clarity of instructions and specificity of score definitions. Continued use, analyses, and refinements of these frameworks will bring us closer to the ultimate goal of using value-based treatment decisions to improve patient care and outcomes. This work was funded by Eisai Inc. Copher and Knoth are employees of Eisai Inc. Bentley, Lee, Zambrano, and Broder are employees of Partnership for Health Analytic Research, a health services research company paid by Eisai Inc. to conduct this research. For this study, Cohen, Huynh, and Neville report fees from Partnership for Health Analytic Research. Outside of this study, Cohen receives grants and direct consulting fees from various companies that manufacture and market pharmaceuticals. Mei reports a grant from Eisai Inc. during this study. The other authors have no disclosures to report. Study concept and design were contributed by Bentley and Broder, with assistance from Elkin and Cohen. Bentley took the lead in data collection, along with Elkin, Huynh, Mukherjea, Neville, Mei, Popescu, Lee, and Zambrano. Data interpretation was performed by Bentley and Broder, along with Elkin, Cohen, Copher, and Knoth. The manuscript was written primarily by Bentley, along with Elkin and Broder, and revised by Bentley, Broder, Elkin, Cohen, Copher, and Knoth. Select components of this work's methods were presented at ISPOR 19th Annual European Congress held in Vienna, Austria, October 29-November 2, 2016, and Society for Medical Decision Making 38th Annual North American Meeting held in Vancouver, Canada, October 23-26, 2016.

BI3 - Assessing Prescription Drug Value In The United States: A Hypothetical Example Comparing Asco And Icer Framework Outcomes

Validity and Reliability of Value Assessment Frameworks for New Cancer Drugs.

6624 Background: Although the development of CIs has led to a dramatic change in the oncology landscape, these agents are associated with significant costs and toxicity. ASCO and ESMO have developed separate frameworks to define the value of emerging cancer treatments in order to encourage cost-effective therapies. We apply these frameworks to trials supporting FDA approvals of CIs and explore the correlation between these two scoring systems. Methods: We searched the FDA database for CIs and indications approved between January 1, 2011 and January 1, 2019. Only randomized phase II/III trials for solid tumors were included. Data on survival, toxicity and quality of life were extracted from the most recent publications by two reviewers independently. A trial showing a substantial benefit was defined as an ASCO score of ≥ 45, or ESMO Grade 4-5 (palliative setting) or Grade A/B (curative setting). Concordance for substantial benefit was assessed using Cohen’s Kappa while Spearman coefficients were used to determine the degree of correlation in individual scores. Results: We identified 40 FDA indications for 7 CIs. Of these, 18 indications based on Phase I/II single-arm trials were excluded. The remaining 22 indications were based on 21 randomized phase II/III trials (3 adjuvant, 18 metastatic). In the palliative setting, 73% and 86% trials showed substantial benefit based on ASCO and ESMO frameworks respectively [median ASCO score: 54.8, interquartile range (IQR) 46.2-64.0; median ESMO score: 5, IQR: 4-5]. 27% of trials were scored intermediate or low benefit by ASCO, while 9% were ineligible for ESMO scoring. Weighted kappa was 0.719 between the two frameworks. Spearman rho was 0.84. All 3 adjuvant trials were assigned ESMO grade A but low benefit with ASCO (median 37.7, IQR 20.5-40.9). Conclusions: In the palliative setting, the majority of trials supporting FDA approved CI indications demonstrated substantial benefit using both ASCO and ESMO frameworks. There was a strong correlation between the two frameworks. However, in the curative setting scores were discordant. The ASCO framework may require further refinement for adjuvant trials.

e18317 Background: The increasing prevalence of cancer coupled with approvals of new drugs and technologies have brought increased scrutiny to cost/benefit of treatments in oncology. To address the rising concern about oncology drug costs, several stakeholders have developed frameworks to help assess the value of oncology regimens. The objective of this study was to assess oncologists’ perceptions, awareness and knowledge of all oncology value frameworks in the US. Methods: Data were collected from an electronic cross-sectional survey of 200 US-based oncologists recruited from an online panel. Oncologists were asked about their knowledge and perceptions of four value frameworks – the National Comprehensive Cancer Network’s (NCCN) Evidence Blocks, American Society of Clinical Oncology (ASCO) Value Framework, Institute for Clinical and Economic Review (ICER) Value Framework and Memorial Sloan Kettering (MSK) Drug Abacus framework. Findings were reported descriptively. Results: Oncologists were most familiar with the NCCN Evidence Blocks (90%) followed by the ASCO Value Framework (84%), ICER value framework (57%) and Drug Abacus (56%). Among those that were familiar with the value frameworks, almost equal proportions found the NCCN Evidence Blocks vs. ASCO Value Framework to be “useful/very useful” (75% vs. 74%) followed by ICER Value Framework (64%) and Drug Abacus (56%). More than three out of four oncologists surveyed (76%) had used value frameworks in the past, with NCCN Evidence Blocks being used most often (61.5%) followed by ASCO Value Framework (48.4%), ICER Value Framework (21%) and Drug Abacus (15%). NCCN Evidence Blocks was ranked highest for its ease of use, comprehensiveness, feasibility for use in the clinical setting, ability of patients to understand and most favorable overall, followed by ASCO Value Framework, ICER Value Framework and Drug Abacus. Conclusions: The majority of oncologists were familiar with value frameworks and had used them in the past. Of the four oncology Value Frameworks, NCCN Evidence Blocks were perceived most favorably by oncologists followed by ASCO Value Framework.

Objective: To assess the clinical value of tumor treating fields (TTFields) in malignant pleural mesothelioma (MPM) by means of the ASCO and ESMO frameworks. Background: The potential effectiveness and safety of TTFields in addition to pemetrexed and cisplatin or carboplatin in MPM was recently shown by the analysis of the Phase II single arm EF-23 STELLAR trial. To account for the need of physicians and policymakers to objectively and comparably capture the clinical value of new cancer treatments the American Society of Clinical Oncology (ASCO) and the European Society for Medical Oncology (ESMO) have both developed assessment frameworks. We quantified the clinical value of the TTFields treatment in MPM by applying ASCO and ESMO frameworks to the comparison of the STELLAR data to historical controls. Materials/Methods: The EF-23 STELLAR Trial (n=80) demonstrated that adding TTFields to pemetrexed and cisplatin or carboplatin for malignant pleural mesothelioma patients resulted in overall survival of 18.2 months (95% CI 12.1-25.8) and progression free survival of 7.6 months (95% CI 6.7-8.6). The ESMO Magnitude of Clinical Benefit Scale (MCBS) and the ASCO Net Health Benefit (NHB) frameworks were applied to the EF-23 trial data using a historical control as comparator. Results: The application of the ASCO framework to the EF-23 data resulted in a NHB score of 52. This result was at the higher end of the score range of novel cancer treatments and compares well to the results for nivolumab in advanced non-squamous NSCLC and advanced squamous NSCLC as reference points. Applying the ESMO framework resulted in MCBS scores of A/5 (adjuvant/advanced) which would be the first score reported for MPM. The MCBS scores of A/5 are the highest scores achievable in the ESMO framework, and higher then the ESMO MCBS scores reported in the literature for NSCLC treatments. Conclusions: Despite differences in their respective concepts, both the ASCO and ESMO frameworks suggest that adding TTFields to Pemetrexed and Cisplatin or Carboplatin in malignant pleural mesothelioma patients may provide a significant clinical benefit. The high scores underline the fact that treatment with TTFields may extended progression free and overall survival without additional systemic toxicities. Citation Format: Justin Kelly, Uri Weinberg, Christina Proescholdt. Application of the ASCO and ESMO frameworks to TTFields treatment of mesothelioma [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr LB-161.

Assessment of Value Using the American Society of Clinical Oncology (ASCO) and European Society of Medical Oncology (ESMO) Frameworks for Novel Therapies for the Hematologic Malignancies

Reply to the letter to the editor ‘Re-aligning the ASCO and ESMO clinical benefit frameworks or modern cancer therapies’

6509 Background: Whether the American Society of Clinical Oncology (ASCO) Value Framework and the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS) measure similar constructs of clinical benefit is unclear. It is also unclear how they relate to quality-adjusted life-years (QALYs) and funding recommendations in the UK and Canada. Methods: Randomized clinical trials (RCTs) of oncology drug approvals by the Food and Drug Administration, European Medicines Agency and Health Canada between January 2006 and August 2015 were identified and scored using the ASCO version 1 (v1) framework (August 10, 2015), ASCO version 2 (v2) framework (May 31, 2016) and ESMO-MCBS (May 30, 2015) by at least two independent reviewers. Spearman correlation coefficients were calculated to assess construct (between frameworks) and criterion validity (against incremental QALYs from the National Institute of Clinical Excellence (NICE) and the pan-Canadian Oncology Drug Review (pCODR)). Associations between scores and NICE/pCODR recommendations were examined by logistic regression models. Inter-rater reliability was assessed using intra-class correlation coefficients. Results: From 109 included RCTs, 108 ASCOv1, 111 ASCOv2 and 83 ESMO scores were determined. Correlation coefficients for ASCOv1 vs. ESMO, ASCOv2 vs. ESMO, and ASCOv1 vs. ASCOv2 were 0.36 (95% CI 0.15-0.54), 0.17 (95% CI -0.06-0.37) and 0.50 (95% CI 0.35-0.63), respectively. Compared with NICE QALYs, correlation coefficients were 0.45 (ASCOv1), 0.53 (ASCOv2) and 0.46 (ESMO); with pCODR QALYs, coefficients were 0.19 (ASCOv1), 0.20 (ASCOv2) and 0.36 (ESMO). None of the frameworks were significantly associated with NICE/pCODR recommendations. Inter-rater reliability was good for all frameworks. Conclusions: The weak-to-moderate correlations between the ASCO frameworks and ESMO-MCBS, with QALYs, and with NICE/pCODR funding recommendations suggest different constructs of clinical benefit measured. Construct convergent validity with the ESMO-MCBS in fact did not increase with the updated ASCO framework.

Clinical Value of TTFields Treatment in Mesothelioma Using ASCO and ESMO Frameworks

PCN26 - ASSESSING THE POTENTIAL VALUE OF AN INNOVATIVE ONCOLOGY THERAPY FROM THE HEALTH TECHNOLOGY ASSESSMENT (HTA) PERSPECTIVE: MARRYING CLINICAL VALUE FRAMEWORKS WITH ECONOMIC ASSESSMENT METHODOLOGY

17 Background: The advent of checkpoint inhibitor therapy (CIT) has dramatically changed the oncology landscape, but is associated with significant costs. A positive randomized clinical trial (RCT) may not translate to meaningful outcomes for patients. The American Society of Clinical Oncology (ASCO) and European Society of Medical Oncology (ESMO) have developed frameworks to quantify the value of cancer treatment. We applied these frameworks to RCTs involving CIT in order to explore the relationship between trial outcomes and magnitude of clinical benefit. Methods: A literature search was conducted to identify CIT RCTs. Data extracted included study characteristics, pre-specified estimated hazard ratios (eHR) and observed HR (oHR). ASCO Value Framework version 2016 and ESMO Magnitude of Clinical Benefit (MCB) scale v1.1 were applied to each publication by 2 authors independently. Results: 30 RCTs (3 adjuvant, and 27 advanced setting) using CIT were identified between January 2010- October 2018. The majority of trials were in lung cancer (37%) and melanoma (36%). The eHR was 0.71±0.06 (range: 0.55-0.78), and oHR was 0.76±0.15 (range: 0.49 – 1.11). 54% RCTs did not achieve eHR, with a difference of 0.16±0.12 (range: 0.01 – 0.41). ASCO framework scores ranged -24.0 to 71.3, far below the maximum potential score of 180. 18 RCTs formed the basis for FDA approvals. The mean ASCO framework score was 45.6 ± 16.6 (range: 14.4 - 71.3) for FDA approved indications, and 14.0 ± 18.4 (range: -24 – 49) for non-FDA approvals (p &lt; 0.001). All FDA approvals scored grade 4 or 5 on the ESMO MCB scale, indicating a meaningful clinical benefit. Many non-FDA approved RCTs did not receive an MCB grade as they were negative trials. There was no difference in the ASCO framework scores between MCB grade 4 and 5 RCTs. 3 adjuvant RCTs had an ASCO framework score ranging from 20.5 to 38.7 despite an MCB Grade A. Conclusions: Many trials did not meet the pre-specified eHR. FDA approval had statistically significantly higher NHB scores than non-FDA approvals, and they were deemed to have a meaningful clinical benefit according to the ESMO MCB scale. ASCO framework scores may require re-calibration since the highest score achieved in CIT RCTs was only 40% of the maximum score.

BACKGROUND: Since its inception in 2006, the Institute for Clinical and Economic Review (ICER) has rapidly gained influence on drug pricing and reimbursement decisions despite historical resistance to the use of cost-effectiveness thresholds in the US health care system. Although patient groups, physicians, and pharmaceutical manufacturers voiced their concerns about the potential negative effects of increased use of ICER's assessments on patient access to innovative medications, there is little guidance and consensus on how the stakeholders should collaborate with ICER to ensure that its reviews reflect the best clinical and economic evidence. OBJECTIVES: To (1) summarize the evolution of ICER's evaluation procedure, scope, and topics; (2) evaluate the effectiveness of stakeholder engagement approaches; and (3) inform stakeholders of their potential role in collaborating with ICER in estimating the cost-effectiveness of new interventions. METHODS: Publicly available ICER evaluations from 2008 to 2019 were systematically reviewed. Changes in evaluation procedures, scope, and topics were summarized. For evaluations that occurred in 2018 (n = 12) and 2019 (n = 8), key characteristics were extracted from 172 letters documenting interactions between ICER and all stakeholders who provided comments to draft reports. Stakeholder suggestions were analyzed in terms of their effectiveness indicated by ICER's reconsideration of its original cost-effectiveness analysis approach. RESULTS: The number of ICER evaluations increased consistently from 2 to 12 per year between 2008 and 2018 but declined to 8 in 2019. Stakeholder opportunity to engage with ICER increased from 1 to 3 per evaluation between 2008 and 2015. ICER initially focused on reviewing general treatment strategies but shifted its focus to specific pharmaceuticals and medical devices in 2014. In 2018 and 2019, 30% of 172 stakeholder letters resulted in a revision in the base-case analysis (49 comments in 2018, 23 in 2019); nearly half of comments in these letters included specific alternative data or a published article to rationalize recommendations. Other common types of suggestions that resulted in ICER's base-case analysis revisions included comments relating to inconsistent methods used to derive model inputs across different treatments (12/49 in 2018, 5/23 in 2019); clinical justifications (12/49 in 2018, 0/23 in 2019); and justifications based on patient perspectives (1/49 in 2018, 5/23 in 2019). These revisions rarely affected ICER's conclusions on the cost-effectiveness of evaluated interventions. Among the 20 assessments that involved 172 stakeholder engagements in 2018 and 2019, only 2% (n = 3) of the engagements (all from 2018) were associated with a change in the cost-effectiveness conclusion. CONCLUSIONS: Between 2018 and 2019, stakeholders leveraged ICER evaluations as opportunities to promote dialogue for better understanding of the value of technologies. Actionable, evidence-based recommendations were accepted more often than other recommendations. DISCLOSURES: No outside funding supported this study. The authors have no conflicts of interest to disclose. Findings from this study were presented as a poster at Virtual ISPOR, May 17-20, 2021.

With the continuous rise in costs for oncology drugs, the American Society of Clinical Oncology (ASCO), the Institute for Clinical and Economic Review (ICER), the Memorial Sloan Kettering Cancer Center's Drug Abacus (DrugAbacus), and the National Comprehensive Cancer Network (NCCN) have developed value-based frameworks (VBFs) to assist stakeholders in formulary and treatment decision-making processes. Since emerging VBFs have the potential to affect available treatment options for patients, it is important to understand the differences associated with these VBFs within various therapeutic areas. To (a) compare VBFs across 3 therapeutic options for relapsed or refractory multiple myeloma (RRMM) and (b) identify challenges and limitations associated with real-world decision making using VBFs in the U.S. marketplace. The values of regimens carfilzomib (CFZ), elotuzumab (ELO), and ixazomib (IX) were generated using the ASCO, NCCN, ICER, and DrugAbacus VBFs. These regimens, used for second- or third-line treatment of RRMM, shared a common comparator in clinical trials: lenalidomide + dexamethasone (LEN + DEX). ASCO's 2016 VBF, which incorporated clinical benefit, toxicity, and bonus points, was used to generate a net health benefit score, along with the drug wholesale acquisition cost, for each regimen compared with LEN + DEX. Results of the 2016 NCCN Evidence Blocks for multiple myeloma and the ICER 2016 report of treatment options for RRMM were extracted to generate the value of CFZ, ELO, and IX. No output was generated from DrugAbacus because of the lack of regimens included in the test case. Shortcomings associated with running the test case in RRMM for each VBF were also identified. Among the 3 therapeutic agents, CFZ, in combination with LEN + DEX, was the most valued. ASCO and ICER VBFs suggested that CFZ + LEN + DEX may be the most valued, followed by ELO + LEN + DEX and IX + LEN + DEX. NCCN suggested that LEN + DEX may be the most valued followed by CFZ + LEN + DEX, IX + LEN + DEX, and ELO + LEN + DEX. A number of shortcomings were noted across each VBF, such as complexities of drug evidence evaluation with the ASCO VBF, the inability to adjust the ICER and NCCN VBFs to specific populations, and subjectivity associated with the NCCN VBF and DrugAbacus. Although the test case provided some consensus on treatment decisions, there is much nuance and limitations with the VBFs available for RRMM. Clearer objectivity and better adaptability to specific treatment decisions are warranted. No outside funding supported this study. The authors have nothing to disclose. All authors contributed to study concept and design, as well data collection and interpretation. Djatche and Goble wrote and revised the manuscript, along with Chun and Varga. Portions of this work have previously been presented at the AMCP Managed Care and Specialty Pharmacy Annual Meeting 2017 in Denver, Colorado, March 27-30, 2017, and at the ISPOR 22nd Annual International Meeting in Boston, Massachusetts, May 20-24, 2017.

BACKGROUND: The Institute for Clinical and Economic Review (ICER) is a prominent health technology assessment (HTA) entity in the United States that considers costs and applies economic analyses to derive price-based recommendations. ICER continues to adjust its value framework, yet discussion persists regarding whether ICER methodologies align with established research standards. This work evaluates ICER assessments relative to those standards, providing a benchmark with the release of ICER's most recent value framework update. OBJECTIVES: To evaluate ICER economic assessments for trends, factors related to recommendations, and quality for use in U.S. decision making. METHODS: We evaluated all ICER final evidence reports published between 2006 and August 31, 2019, with regard to base-case result trends over time, pricing sources, comparator selection, analytic perspectives, model uncertainty, how modeling results aligned with ICER's determinations of value for money, and comparison of ICER methodological approaches with established modeling standards. Analyses were stratified by time period, where appropriate, to account for changes in ICER's framework over time. RESULTS: Of 58 ICER final evidence reports, 47 used the most commonly reported outcome (cost per quality-adjusted life-year [QALY]); ICER-developed models evaluated 131 interventions and comparators with 238 base-case results. Pricing sources for ICER reports became more standardized in 2017, although sources were not associated with the likelihood of falling below ICER's cost-effectiveness thresholds. In 30% of base-case analyses (n = 72), ICER did not use a clinical comparator, although reasonable treatments were available. In modified societal perspectives scenarios applied in later assessments, 75% of analyses (n = 76) included productivity but did not specify how it was quantified. Reports did not explain how sensitivity and scenario analyses were selected or implications of results. ICER value for money determinations generally aligned with cost-effectiveness results, although 2 of 33 (6%) interventions ranked as low value and 3 of 5 (60%) interventions ranked as low-moderate value, met a $150,000 per QALY threshold, and 14 of 37 (38%) moderate-value interventions exceeded this threshold; the most common rationale was related to national budget impact. CONCLUSIONS: While some progress has been made, further improvement is needed to ensure that ICER assessments address the most relevant questions for target audiences, adhere to established research standards, and are reported in a manner that can be readily interpreted and applied to policymaking. DISCLOSURES: No outside funding supported this study. The authors have nothing to disclose.