Statistical Issues on Evaluating Agreement Between the Grade Groups Determined by Nonlinear Microscopy vs Formalin-Fixed Paraffin-Embedded Histology

Abstract

We read with interest the recent article entitled “Real-time diagnosis and Gleason grading of prostate core needle biopsies using nonlinear microscopy,” which was published online in Modern Pathology on November 2, 2021.1Cahill L.C. Rosen S. Yoshitake T. et al.Real-time diagnosis and Gleason grading of prostate core needle biopsies using nonlinear microscopy.Mod Pathol. 2022; 35: 539-548Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar In their article, the authors described a method for evaluating prostate core needle biopsy specimens in minutes using nonlinear microscopy (NLM) and evaluated this method by reading specimens from patients who underwent magnetic resonance imaging–guided core needle biopsy procedures in real time. After assessing the accuracy and procedure time of NLM for diagnosing prostate carcinoma, Gleason scoring, and estimating the amount of biopsy core involved by carcinoma, the authors proposed that NLM is a promising method for future rapid evaluation of prostate needle core biopsy specimens. We highly appreciate the authors’ efforts. Nonetheless, we would like to present our concern about the authors’ work when they used unweighted Cohen's kappa statistic to estimate agreement between the grade groups determined by NLM vs formalin-fixed paraffin-embedded (FFPE) histology. Normally, unweighted Cohen's kappa statistic is suitable for evaluating 2 raters in the presence of 2 categories.2Dano H. Altinay S. Arnould L. et al.Interobserver variability in upfront dichotomous histopathological assessment of ductal carcinoma in situ of the breast: the DCISion study.Mod Pathol. 2020; 33: 354-366Abstract Full Text Full Text PDF PubMed Scopus (19) Google Scholar Meanwhile, the weighted Cohen's kappa statistic should be used to calculate agreement in the presence of 3 or more categories between 2 raters.3Bulten W. Balkenhol M. Belinga J.A. et al.Artificial intelligence assistance significantly improves Gleason grading of prostate biopsies by pathologists.Mod Pathol. 2021; 34: 660-671Abstract Full Text Full Text PDF PubMed Scopus (37) Google Scholar The popular weights for weighted Cohen's kappa are the so-called linear weights and quadratic weights.4Warrens M.J. Conditional inequalities between Cohen’s kappa and weighted kappas.Stat Methodol. 2013; 10: 14-22Crossref Scopus (28) Google Scholar McHugh5McHugh M.L. Interrater reliability: the kappa statistic.Biochem Med (Zagreb). 2012; 22: 276-282Crossref PubMed Google Scholar suggested that the kappa value should be interpreted as follows: 0 to 0.20 as indicating no agreement, 0.21 to 0.39 as indicating minimal agreement, 0.40 to 0.59 as indicating weak agreement, 0.60 to 0.79 as indicating moderate agreement, 0.80 to 0.90 as indicating strong agreement, and 0.91 to 1.00 as indicating almost perfect agreement. Thus, in our opinion, weighted Cohen's kappa is more applicable than unweighted Cohen's kappa for the presence of 5 such categories in this research. According to the authors’ data, we recalculated the agreements between the grade groups determined by NLM vs FFPE histology using weighted Cohen's kappa. We report, as one would expect, improvement in the Cohen's kappa statistic when comparing the unweighted Cohen's kappa with linearly and quadratic weighted Cohen's kappa statistic (kappa value improved from 0.588 to 0.687 to 0.781). Accordingly, the interpretation of the kappa value changed from weak agreement to moderate agreement. Apparently, the authors underestimated the agreement between the grade groups determined by NLM vs FFPE histology. In a word, we want to emphasize that any scientific conclusion needs to be supported by a reasonable application of methodological and statistical methods. Using appropriate statistical methods can improve the scientific nature of research results. M.L. conceived and prepared the first draft of the manuscript. C.Z. analyzed and interpreted the data. T.Y. critically reviewed the draft. All authors read and approved the final manuscript. Not available.