Assessment of Interrater Reliability for Comorbidity Indices in Determining Eligibility for Treatment Intensification in Head and Neck Cancer

Abstract

With an aging population, reliable measures of comorbidity are increasingly crucial for clinical decision-making in head and neck cancer patients. Concomitant medical conditions are recognized as significant predictors of survival and treatment tolerance. Multiple validated tools exist to evaluate various fitness measurements and assist in determining eligibility for treatment with concurrent chemotherapy and clinical trial enrollment in head and neck cancer patients. We measured the interrater reliability (IRR) in scoring comorbidity indices and determining fitness for treatment intensification between users in various clinical roles. We analyzed a cohort of twenty patients with locoregionally advanced head and neck cancer. Baseline characteristics, self-rating of health, and comorbidities were collected at time of initial consultation. Four independent observers evaluated each patient based on six previously validated metrics. The indices included a Generalized Competing Event (GCE) model based on RTOG data, G8, Charlson Comorbidity Index (CCI), Cancer and Aging Research Group (CARG) model, Adult Comorbidity Evaluation-27 (ACE-27), and Cumulative Illness Rating Scale for Geriatrics (CIRS-G). Intraclass correlation coefficient (ICC) was generated for continuous GCE values and Fleiss’ kappa statistics were used for binary, nominal, and ordinal variables to evaluate interrater reliability. Fleiss’ kappa greater than 0.6 is generally accepted to indicate substantial agreement. The ICC for the GCE model was 0.909; when the GCE score was transformed into a binary variable using eligibility criteria of GCE <0.6, the kappa statistic was 0.918. Fleiss’ kappa scores for G8, CCI, CARG, ACE-27 and CIRS-G were 0.463, 0.537, 0.727, 0.527, and 0.223, respectively. After converting individual scores to binary values based on eligibility criteria from an ongoing randomized trial, the kappa scores for G8, CCI, CARG, ACE-27 and CIRS-G were 0.817, 0.682, 0.614, 0.624, and 0.733, respectively. The kappa statistic for treatment intensification eligibility was 0.722, and all raters agreed on eligibility status for eighteen of twenty patients. Of the two patients for which raters disagreed, the ACE-27 score was the source of discrepancy. Variability in IRR between comorbidity indices suggests potential issues when applying any one of these measures to head and neck cancer patients in the context of clinical trial enrollment and practice. Despite differences in IRR between evaluation models and various user backgrounds, there was 90% agreement on patient eligibility for treatment intensification after comprehensive calculation of comorbidity metrics. Additional systematic study is required to determine consistent methods of quantifying comorbid conditions and their magnitude of effect on a patient’s fitness for and tolerance of treatment intensification.