Observer agreement about laryngoscopic assessment of papilloma

Abstract

Background: Observer bias may confound the assessment of therapies utilized for recurrent respiratory papilloma, a recidivistic fluctuating disease. Any convincing change in a patient's disease must exceed the imprecision of the measuring system. Materials and Methods: Videotapes of ten children, who had airway endoscopy for care of laryngeal papilloma, were edited. The videotapes were independently reviewed by six otolaryngologists, of whom five repeated their assessments 5 to 20 weeks later. The ‘reference standard’ was the operating surgeon's categorization of his patient's videotape. Agreement was calculated by the kappa statistic, the observed proportions of positive and negative agreement, and the proportion categorized as exact opposites, for each of twelve anatomic sites. Results: Moderate agreement was found for both (a) the operative findings versus the later-determined ‘reference standard’, and (b) the ‘reference standard’ versus categorizations by other otolaryngologists. Intra-observer agreements were better than inter-observer agreements. Agreement as to whether or not papilloma involved each of twelve sites was within about 20% for the five pediatric otolaryngologists vs the ‘reference standard’, and within about 10% for the same reviewer. Conclusion: Videotapes may add objectivity to the assessment of laryngeal papilloma. For changes to be considered significant, an observer should identify a 10% change in the number of sites involved, or a 30% change in the extent of overall airway obstruction. For different observers, changes would have to be even greater to be meaningful.