Pairwise Kappa Values Research Articles

Unsatisfactory reproducibility of interstitial inflammation scoring in allograft kidney biopsy

Interstitial inflammation scoring is incorporated into the Banff Classification of Renal Allograft Pathology and is essential for the diagnosis of T-cell mediated rejection. However, its reproducibility, including inter-rater and intra-rater reliabilities, has not been carefully investigated. In this study, eight renal pathologists from different hospitals independently scored 45 kidney allograft biopsies with varying extents of interstitial inflammation. Inter-rater reliabilities and intra-rater reliabilities were investigated by kappa statistics and conditional agreement probabilities. Individual pathologists’ scoring patterns were examined by chi-squared tests and proportions tests. The mean pairwise kappa values for inter-rater reliability were 0.27, 0.30, and 0.26 for the Banff i score, ti score, and i-IFTA, respectively. No rater pair performed consistently better or worse than others on all three scorings. After dichotomizing the scores into two groups (none/mild and moderate/severe inflammation), the averaged conditional agreements ranged from 47.1% to 50.0%. The distributions of the scores differed, but some pathologists persistently scored higher or lower than others. Given the important role of interstitial inflammation scoring in the diagnosis of T-cell mediated rejection, transplant practitioners should be aware of the possible clinical implications of the far-from-optimal reproducibility.

Interobserver Variability in the Assessment of Tumor Budding in pT 3/4 Colon Cancer: Improvement by Supporting Immunohistochemistry?

The prognostic significance of tumor budding in colon cancer is unequivocally documented, and the recommendations of the International Tumor Budding Consensus Conference (ITBCC) are currently the accepted basis for its assessment. Up to now, it is unknown whether the general use of a supporting cytokeratin immunohistochemistry can improve the interobserver variability and prognostic significance. Six investigators with different levels of experience reassessed 229 cases of colon carcinoma (pT3/4, N+/−, M0) with a supporting cytokeratin immunohistochemistry. The results were compared to previous assessments, which have been performed only on H & E. Bd3 was significantly associated with the occurrence of distant metastases according to the assessments of three out of six investigators (p < 0.05). Only one single investigator reached significant results concerning the cancer specific survival (p = 0.01). The pairwise kappa values range between a poor and moderate level of agreement (range 0.17–0.45; median 0.21). In conclusion, the results show no superiority of the use of an additional cytokeratin immunohistochemistry compared to the conventional analysis on sole H & E slides. Therefore, the general supporting use of a cytokeratin immunohistochemical staining seems to be inadvisable in colon cancer in consideration of necessary resources and costs.

Tu1675 Interobserver and Intraobserver Variation in the Diagnosis of Minimal Change Esophagitis by High Definition and I-SCAN Upper Endoscopy

Interobserver and Intraobserver Variation in the Diagnosis of Minimal Change Esophagitis by High Definition and I-SCAN Upper Endoscopy Jaksin Sottisuporn*, Nisa Netinatsunton, Naichaya Chamroonkul, Teepawit Witeerungrot, Theeratus Jongboonyanuparp, Siriboon Attasaranya, Bancha Ovartlarnporn NKC institute of Gastroenterology and Hepatology, Hatyai, Thailand Background: Minimal changes esophagitis lesions (MCL) has been regarded as one of the spectrum of erosive esophagitis in some studies. The MCLs are subtle changes and are difficult to identify by conventional endoscopy. I-Scan endoscopy (SE) is a new imaging technology that enhances the mucosal details. Limited studies showed the SE to be better than white light endoscopy in detecting MCL. The intraobserver and interobserver variations of SE in detecting MCL have never been reported before. Aim: To assess the intraand interobserver variations in detecting MCL by high-definition endoscopy (HD) and SE in tone enhancement mode for esophagus (TE-e) in dyspeptic patients. Method: The endoscopic images in HD and TE-e modes of distal esophagus of all patients with upper abdominal symptoms scheduled for upper gastrointestinal endoscopy at the NKC institute from February 2010 till August 2011 were used to assess the consistency of images interpretation by 6 endoscopists. A set of images representing MCL that included punctate erythema (PE), blurred vessels (BV), minute erosion (ME) and triangular lesion with elongated pit (TLE) were used to educate all the endoscopists. 6 endoscopists randomly evaluated the endoscopic images of 89 patients in HD and TE-e mode twice with at least 24 hours interval in between. The intra and inter-observer consistency of images interpretation was analyzed. Result: Intraobserver kappa values (95% CI) for HD group were 0.44(0.24-0.65), 0.44(0.24-0.65), 0.410.2-0.62), 0.59(0.39-0.8), 0.45(0.24-0.65), 0.13(0.00-0.34) and for TE-e SE were 0.35(0.15-0.56), 0.56(0.360.77), 0.97(076-1.00), 0.51(0.31-0.71), 0.44 (0.61-0.27), 0.32(0.11-0.53) respectively. These represent fair to excellent agreement for intraobserver image interpretation in both HD and TE-e. One of the endoscopist had significant improvement in detecting MCL on the second image interpretation (kappa 0.13 versus 0.32 p 0.05). In the interobserver consistency of image interpretation, the pairwise kappa values for HD and TE-e of all 6 endoscopists were rather low (range 0.16-0.3 and 0.05-0.37 respectively). When the evaluations in both groups were analyzed by pairing individual endoscopist into 15 pairs, the kappa values improved with agreement in 7 of 15 pairs without statistical significance. (p 1) Conclusions: The intraobsever agreement for MCL by HD and i-Scan is fair to excellent. The poor interobserver agreement by HD and TE-e for MCL limits the value of the MCL in clinical practice. The SE imaging technology did not improve the interobserver image interpretation consistency.

Good interobserver agreement was attainable on outcome adjudication in patients with stable coronary heart disease

ObjectiveIn clinical trials, agreement on outcomes is of utmost importance for valid estimation of intervention effects. As there is limited knowledge about adjudicator agreement in cardiology, we examined the level of agreement among three cardiology specialists adjudicating all possible events in a randomized controlled clinical trial of patients with stable coronary heart disease. Study Design and SettingAll information (hospital records, death certificates, etc.) was forwarded to two randomly selected blinded adjudicators. If they disagreed, the third arbiter had to choose the more likely of the two alternatives. Files of 5,475 nonfatal and 362 fatal events were evaluated. ResultsFor nonfatal outcomes, pairwise kappa values ranged from 0.75 to 0.80. The three adjudicators had 4.3%, 9.5%, and 6.1% of their nonfatal outcome classifications overruled by their arbiter. If stable angina pectoris, unstable angina pectoris, and acute myocardial infarction were treated as one, agreement increased minimally. For fatal outcomes, the pairwise kappa values ranged from 0.65 to 0.90. The three adjudicators had 12%, 9%, and 10% of their death classifications overruled. ConclusionSpecialists in cardiology can attain a reasonably high agreement on outcomes in patients with stable coronary heart disease.

Characterization of Dihydrofolate Reductases from Multiple Strains of Plasmodium falciparum Using Mathematical Descriptors of Their Inhibitors

Two classes of graph-theoretic molecular descriptors, viz., topological indices (TIs) and atom pairs (APs), have been used to derive high-quality quantitative structure-activity relationships (QSARs) for inhibitors of dihydrofolate reductases (DHFRs) isolated from the wild and four mutant strains of Plasmodium falciparum. Of the three methods used for QSAR formulation, viz., principal-components regression (PCR), partial least squares (PLS), and ridge regression (RR), the RR method outperformed the other two. Cohen's kappa values, based on the overlap of significant and insignificant structural descriptors calculated for the QSAR development, show that DHFR from the wild strain is substantially different from the four mutant strains. The differential QSAR approach reported in this study can be used in protocols for the development of drugs to combat drug-resistant pathogens arising continuously in nature due to mutations. The pairwise kappa values in conjunction with appropriate drug targets and their corresponding set of ligands may be a useful tool in gauging the evolving mutual similarities and dissimilarities of pathogenic organisms from purely computed mathematical descriptors of the ligands.

Histologic grading of invasive lobular carcinoma: does use of a 2-tiered nuclear grading system improve interobserver variability?

The Nottingham histologic grade (NHG) is a prognostic marker for infiltrating ductal carcinoma. Its usefulness for invasive lobular carcinoma (ILC) has been less clear, given that 2 of the 3 parameters, tubule formation and mitotic activity, show little variation in ILC, placing much of the emphasis on nuclear grade. We have previously reported a trend for improved overall and relapse-free survival in patients with ILC of low nuclear grade, as classified by a 2-tiered nuclear grading system. Given the inherent potential for interobserver variability with any grading system, the goal of this study is to compare interobserver variability in the grading of ILC using a 2-tiered nuclear grade vs the NHG. Thirty-eight cases of ILC were graded independently by 5 pathologists using NHG criteria. Tumors were also categorized by a nuclear grading system as low grade (grade 1 nuclei) or high grade (grades 2-3 nuclei). Pairwise kappa values and interobserver agreement rates were calculated for both NHG and nuclear grade. Results were compared using the paired t test. Mean interobserver agreement rates and kappa values improved with use of the nuclear grading system as compared to NHG (83% vs 70%, 0.4738 vs 0.3228, respectively). The differences between the 2 were statistically significant. Because histologic grade has significant prognostic implications for patients with breast cancer, accurate reporting is paramount. For ILC, where use of the NHG places substantial weight on nuclear pleomorphism, a 2-tiered nuclear grading system may reduce interobserver variability yet still provide useful prognostic information.

Lumbar Isthmic Spondylolisthesis Detection with Palpation: Interrater Reliability and Concurrent Criterion-Related Validity

Lumbar spinous process palpation to detect spondylolisthesis is a technique frequently used by clinicians and taught in professional education courses, but it has not been shown to be reliable or valid. The purpose of this study was to assess the diagnostic utility of lumbar spinous palpation as a means of detecting isthmic spondylolisthesis in patients with low back pain. Interrater reliability was assessed by pair-wise comparison of the findings of three therapists palpating the spinous processes on 30 subjects with low back pain. Validity was evaluated by comparing the findings of one therapist to a reference standard of plain film radiographs in 44 patients. The pair-wise kappa values were poor to fair at 0.179, 0.394, and 0.314. Validity testing revealed a sensitivity of 60% (95% CI, 14.7-94.7) and a specificity of 87.2% (95% CI, 72.6-95.7%). The positive likelihood ratio was 4.68 (95% confidence interval (CI): 1.57-13.88) and the negative likelihood ratio was 0.458 (95% CI, 0.155-1.35). Both likelihood ratios produced only a small change in pre- to post-test probabilities. Based on our results, static spinous process palpation by itself is not a definitive method for the detection of spondylolisthesis. However, the study data do not preclude the possible future use of palpation along with other, yet to be identified, validated predictive variables for isthmic spondylolisthesis.

Reproducibility of Subclassification of Squamous Intraepithelial Lesions: Conventional versus ThinPrep Paps

Liquid-based cytologic methods are increasingly used, and classification of squamous intraepithelial lesions (SIL) affects patient management. This study compared interobserver reproducibility in SIL subclassification on conventional (CV) and ThinPrep (TP) cytologic specimens. Four reviewers independently subclassified SIL on 69 CV and 60 TP Paps. Specimens were retrieved by computer search of biopsy-confirmed SIL cases. A consensus interpretation of low-grade SIL (LSIL) or high-grade SIL (HSIL) was assigned when three or four observers agreed. All four observers agreed in 40 of 69 CV with consensus reached in 56 of 69 CV Paps (81%; 20 LSIL, 36 HSIL). For TP Paps, 38 of 60 had 100% agreement, with consensus reached in 56 of 60 TP Paps (93%; 28 LSIL, 26 HSIL, 2 SIL, difficult to grade). kappa values for all four observers were 0.48 for CV (fair agreement) and 0.63 for TP (substantial). Pairwise kappa values ranged from 0.44 to 0.60 for CV and 0.54 to 0.76 for TP. Most of the nonconsensus cases included SIL, difficult to grade interpretations; in several, the original cytologic or biopsy SIL classification, or both, was also indeterminate, or cytologic and biopsy results did not correlate exactly. High-grade biopsies followed 15% of LSIL CV and 36% of LSIL TP. Interobserver reproducibility in SIL subclassification may be better on TP Paps; however, both CV and TP have indeterminate lesions with low interobserver agreement. The TP specimens did not show improved correlation with histologic analysis, and specimens with consensus do not always have correlating biopsy findings.

Interobserver reproducibility of the Nottingham modification of the Bloom and Richardson histologic grading scheme for infiltrating ductal carcinoma.

The interobserver reproducibility of the Nottingham modification of the Bloom and Richardson histologic grading scheme for invasive breast carcinoma was tested. Six surgical pathologists from four institutions independently evaluated histologic grade and each of its three components for 75 infiltrating ductal carcinomas. The number of slides per case ranged from one to nine (median 3). Pairwise kappa values for agreement ranged from moderate to substantial (0.43-0.74) for histologic grade. Generalized kappa values indicated substantial agreement for tubule formation (0.64), moderate agreement for mitotic count (0.52), and near moderate agreement for nuclear pleomorphism (0.40). Normalizing the mitotic counts per mm2 showed only slight improvement in agreement over the published range of mitotic counts for three different field areas. The results suggest that steps to discriminate between categories for nuclear pleomorphism would likely be of benefit for improving the interobserver reproducibility of histologic grade. Nevertheless, the Nottingham modification of the Bloom and Richardson grading system is recommended as a suitable scheme for evaluating invasive breast carcinomas in the routine clinical setting.