Assessing the Performance of the Clinician-reported Genetic Testing Utility InDEx (C-GUIDE): Further Evidence of Inter-rater Reliability

Abstract

Advanced genomic and genetic testing technologies are quickly diffusing into clinical practice, but standardized approaches to assessing their clinical utility are limited. Previous work developed and generated preliminary evidence of validity for a novel outcome measure, the Clinician-reported Genetic testing Utility InDEx (C-GUIDE). C-GUIDE is a 17-item measure that captures the utility of genetic testing from the providers' perspective. Preliminary evidence of its inter-rater reliability was obtained through a clinical vignette study. The purpose of this study was to further assess its inter-rater reliability using actual clinical cases. One genetic counselor and one medical geneticist independently completed C-GUIDE Version 1.1 after genetic test results were disclosed to a shared set of 42 patients. Raters also completed a case description questionnaire, including information about the patient's age, indication for testing, and type of test performed. Inter-rater reliability was assessed by comparing the raters' C-GUIDE scores using ANOVA to generate intra-class correlation coefficients (ICCs), absolute agreement, and mixed repeated measures ANOVA. Of the 42 patients studied, the most common indications for testing were hearing loss (n=18) and craniosynostosis (n=11), and the most common tests ordered were gene panels (n=20) and microarrays (n=10). Test results were diagnostic or partially diagnostic for 11 patients, potentially diagnostic for 14 patients, or nondiagnostic for 17 patients. The overall ICC was 0.95 (95% CI, 0.89-0.97) and absolute agreement was acceptable (>70%) for 15 individual items. Inter-rater agreement was excellent (ICC > 0.90) for 8 items, good (ICC=0.75-0.89) for 3 items, moderate (ICC=0.50-0.74) for 4 items and poor (ICC < 0.50) for 2 items. Absolute agreement was unacceptable (<70%), and rater agreement was fair (ICC=0.40-0.59) for 2 items. For the global rating, the ICC was 0.62 (95% CI, 0.39-0.77), and the absolute agreement was 61.9%. Rater instructions for item completion have been modified to improve consistency of item interpretation. Although further assessments of reliability are warranted after modifications, these findings provide additional tentative evidence of C-GUIDE's inter-rater reliability and suggest that it may be useful as a strategy for measuring the value of genetic testing, as perceived by genetics providers.