Reply to the letter to the editor ‘Re-aligning the ASCO and ESMO clinical benefit frameworks or modern cancer therapies’

Abstract

We appreciate the ongoing interest and contributions of Dr Del Paggio and his colleagues in the development of clinical benefit scales. In this letter [1.Del Paggio J.C. Sullivan R. Hopman W.M. Booth C.M. Re-aligning the ASCO and ESMO clinical benefit frameworks for modern cancer therapies.Ann Oncol. 2018; 29: 773-774Abstract Full Text Full Text PDF PubMed Scopus (6) Google Scholar], they report that the relatively low correlation of ‘clinically meaningful benefit’ identified using the American Society of Clinical Oncology (ASCO) Value Framework v2 and ESMO-MCBS v1.0 that they have previously described [2.Del Paggio J.C. Sullivan R. Schrag D. et al.Delivery of meaningful cancer care: a retrospective cohort study assessing cost and benefit with the ASCO and ESMO frameworks.Lancet Oncol. 2017; 18: 887-894Abstract Full Text Full Text PDF PubMed Scopus (70) Google Scholar], is slightly improved using the new ESMO-MCBS v1.1. We believe that the low correlations identified by Del Paggio et al. do not accurately represent the true correlation between the two scales, and that their findings are compromised by a flawed approach to determining the threshold scores for ‘clinical meaningful benefit’ that they have used in their correlations. The inaccuracies derive from the accumulated errors in correlating an inaccurate understanding of the ESMO-MCBS scoring with regard to ‘clinically meaningful benefit’ with a non-verified threshold for ‘meaningful clinical benefit’ for scores derived from the ASCO Value Framework. Interestingly, Del Paggio et al. have foreseen the potential for this sort of error in their initial publication where the authors ‘recognise that changing this threshold will change the degree of correlation between the two frameworks’ [2.Del Paggio J.C. Sullivan R. Schrag D. et al.Delivery of meaningful cancer care: a retrospective cohort study assessing cost and benefit with the ASCO and ESMO frameworks.Lancet Oncol. 2017; 18: 887-894Abstract Full Text Full Text PDF PubMed Scopus (70) Google Scholar]. In the methods section of their initial study [2.Del Paggio J.C. Sullivan R. Schrag D. et al.Delivery of meaningful cancer care: a retrospective cohort study assessing cost and benefit with the ASCO and ESMO frameworks.Lancet Oncol. 2017; 18: 887-894Abstract Full Text Full Text PDF PubMed Scopus (70) Google Scholar] they write: ‘ESMO-MCBS defines “meaningful clinical benefit” as a grade of 4, 5 (in a non-curative setting) B, or A (in a curative setting)’. In so doing the researchers conflate the term ‘substantial benefit’, which is used by the ESMO-MCBS [3.Cherny N. Dafni U. Bogaerts J. et al.ESMO-magnitude of clinical benefit scale version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (325) Google Scholar, 4.Cherny N.I. Sullivan R. Dafni U. et al.A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).Ann Oncol. 2015; 26: 1547-1573Abstract Full Text Full Text PDF PubMed Scopus (532) Google Scholar, 5.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar], with the term ‘meaningful clinical benefit’ which it is not. This is an inaccuracy that perpetuates an incorrect understanding of the ESMO-MCBS grading that we want to correct. While is it true that ESMO-MCBS, in both v1.0 and v1.1, distinguishes high benefit from low benefit studies, the scale does not set a threshold for ‘clinically meaningfulness’ [3.Cherny N. Dafni U. Bogaerts J. et al.ESMO-magnitude of clinical benefit scale version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (325) Google Scholar, 4.Cherny N.I. Sullivan R. Dafni U. et al.A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS).Ann Oncol. 2015; 26: 1547-1573Abstract Full Text Full Text PDF PubMed Scopus (532) Google Scholar, 5.Dafni U. Karlis D. Pedeli X. et al.Detailed statistical assessment of the characteristics of the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS) threshold rules.ESMO Open. 2017; 2: e000216Abstract Full Text Full Text PDF PubMed Scopus (18) Google Scholar]. Using the ESMO-MCBS, scores of A or B (for treatments of curative intent), or 5 or 4 (for treatments used in the non-curative/palliative setting) indicate ‘a high level of proven clinical benefit’ or ‘substantial benefit’ which is likely to be clinically meaningful. However, the scale makes no claim that these scores are a threshold for ‘meaningful clinical benefit’. Indeed, it is plausible that studies demonstrating slightly lesser gains and achieving slightly lower scores, such as a grade of 3 in the non-curative/palliative setting, may provide benefit that is less than substantial, but which may still be considered as ‘clinically meaningful’. This point is illustrated by the study of Hammerman et al. correlating the decisions of the Israeli health technology assessment (HTA) body to ESMO-MCBS scoring for the relevant reimbursement decisions [6.Hammerman A. Greenberg-Dotan S. Feldhamer I. et al.The ESMO-Magnitude of Clinical Benefit Scale for novel oncology drugs: correspondence with three years of reimbursement decisions in Israel.Expert Rev Pharmacoecon Outcomes Res. 2017; PubMed Google Scholar]. They found that in the non-curative setting, most medications attaining a score of ≥3 were approved for reimbursement whereas those with a score of <3 were very rarely approved. This study suggests that in a country with a developed economy and health care system and a refined HTA process, ESMO-MCBS scores ≥3 were correlated with the judgment by HTA that a new therapy provides enough meaningful benefit to justify reimbursement in a resource limited environment [6.Hammerman A. Greenberg-Dotan S. Feldhamer I. et al.The ESMO-Magnitude of Clinical Benefit Scale for novel oncology drugs: correspondence with three years of reimbursement decisions in Israel.Expert Rev Pharmacoecon Outcomes Res. 2017; PubMed Google Scholar]. Regarding the correlation between these ASCO and ESMO approaches, the two organisations are currently collaborating to tighten the convergence between their approaches. In preliminary work to this end, the respective development teams of the ESMO-MCBS and ASCO frameworks have recently completed scoring of 91 studies in the non-curative setting applying the Net Health Benefit segment of the ASCO Value Framework v2 [7.Schnipper L.E. Davidson N.E. Wollins D.S. et al.Updating the American Society of Clinical Oncology Value Framework: revisions and reflections in response to comments received.J Clin Oncol. 2016; 34: 2925-2934Crossref PubMed Scopus (429) Google Scholar] and the ESMO-MCBS v1.1 [3.Cherny N. Dafni U. Bogaerts J. et al.ESMO-magnitude of clinical benefit scale version 1.1.Ann Oncol. 2017; 28: 2340-2366Abstract Full Text Full Text PDF PubMed Scopus (325) Google Scholar]. Scores generated by experts in the use of each of the frameworks, indicate a higher degree of correlation (correlation coefficient of r = 0.591) than indicated Del Paggio et al. A report of this collaborative work is in preparation and will include a detailed analysis of factors contributing to discordance between scores generated using the current versions of the ESMO and ASCO frameworks. Along with need for methodological diligence in data acquisition and careful application of the ESMO-MCBS and other value frameworks, the experience of the correlative studies by Del Paggio et al. highlights the need for a nuanced and accurate understanding of the meaning (and acknowledged limitations) of the grades generated by such scales.