Examining Convergent Construct Validity and Inter-Rater Reliability Between ESMO-MCBS and ASCO-VF When Evaluating Hematologic Malignancies

Abstract

▪IntroductionThe American Society of Clinical Oncology Value Framework (ASCO-VF) and European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS) have been developed to quantify clinical benefit and establish value of anticancer therapies. While ASCO-VF is indifferent to the cancer type, ESMO-MCBS does not endorse their framework for scoring hematologic malignancies. The reason for this exclusion and whether ESMO-MCBS can be applied in the hematologic setting is still unclear. The purpose of this study was to determine whether measurement characteristics of ESMO-MCBS are similar to ASCO-VF when evaluating hematologic malignancies.MethodsPhase III randomized controlled trials (RCTs) of hematological malignancy drugs approved by the US Food and Drug Administration, European Medicines Agency and Health Canada between 2006 to 2017 were identified and scored using ASCO-VF (version 2) and ESMO-MCBS (version 1.1) by two independent reviewers. We assessed the convergent construct validity of the two frameworks to determine the degree to which these two theoretically similar measures assess the same construct of clinical benefit. Spearman correlation coefficients were calculated for the preliminary framework scores (using only the survival efficacy components of the scores) and the final framework scores (following adjustment of the preliminary scores with toxicity and quality of life (QoL) data). Correlation coefficients were also calculated for each subtype of blood cancer separately. The inter-rater reliability of the frameworks was assessed using intra-class correlation coefficients (ICC).ResultsIn total, 48 studies were included and scored using the ASCO-VF (median score: 34.38; IQR: 22.26 - 51.69) and ESMO-MCBS (median score: 3; IQR: 2.75 - 4). Out of the 48 RCTs, 16 (33%) evaluated novel anticancer treatments for leukemia, 23 (48%) for myeloma, 8 (17%) for lymphoma and 1 (2%) for myelodysplastic syndromes. Spearman correlation coefficients between both frameworks for preliminary and final scores were 0.01 (95% CI, -0.23 to 0.25) and -0.02 (95% CI, -0.28 to 0.24), respectively. When stratified by indication, the correlation coefficients for the preliminary scores were -0.01 (95% CI, -0.50 to 0.49), -0.06 (95% CI, -0.46 to 0.29), and 0.58 (95% CI, -0.21 to 0.91) for leukemia, myeloma, and lymphoma, respectively. The correlation coefficients for the final scores by cancer indications were -0.09 (95% CI, -0.56 to 0.42), 0.15 (95% CI, -0.22 to 0.49), and -0.29 (95% CI, -0.86 to 0.52) for leukemia, myeloma, and lymphoma, respectively (Table 1). For both preliminary and final scores, ASCO-VF showed excellent and ESMO-MCBS showed good inter-rater reliability (Table 2).ConclusionsOur results suggest divergent validity of ESMO-MCBS and ASCO-VF when assessing hematologic malignancies, indicating that these frameworks likely measure different constructs of clinical benefit in this setting. Furthermore, the ASCO-VF demonstrated improved inter-rater reliability compared to ESMO-MCBS. It remains unclear which framework is correctly assessing the clinical benefit of hematologic drugs. To better understand if one framework is more accurate in its evaluation, future research that compares these frameworks to other established measures of clinical benefit (such as quality adjusted life years) in the hematologic setting is warranted. Due to the lack of construct validity and ESMO's endorsement of their framework solely for solid cancers, the ASCO-VF should currently be considered for use in the hematologic setting. DisclosuresNo relevant conflicts of interest to declare.