The dilemma of digital imaging in retinopathy of prematurity

Abstract

Two articles, one in the August issue of the Journal of AAPOS by Wallace et al1Wallace D.K. Zhao Z. Freedman S.F. A pilot study using “ROPtool” to quantify plus disease in retinopathy of prematurity.J AAPOS. 2007; 11: 381-387Abstract Full Text Full Text PDF PubMed Scopus (91) Google Scholar and one in this issue of the Journal of AAPOS by Gelman et al,2Gelman R. Jiang L. Du Y.E. Martinez-Perez M.E. Flynn J.T. Chiang M.F. Plus disease in retinopathy of prematurity: Pilot study of computer-based and expert diagnosis.J AAPOS. 2007; 11: 532-540Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar highlight the most vexing dilemma facing the ophthalmologist caring for prematurely born babies who develop “serious” retinopathy of prematurity (ROP): Are the posterior pole vessels of this eye sufficiently abnormal to qualify as having plus disease?3The International Classification of Retinopathy of Prematurity revisited.Arch Ophthalmol. 2005; 123: 991-999Crossref PubMed Scopus (2089) Google Scholar, 4An international classification of retinopathy of prematurity The Committee for the Classification of Retinopathy of Prematurity.Arch Ophthalmol. 1984; 102: 1130-1134Crossref PubMed Scopus (1123) Google Scholar The answer to this question largely determines whether or not an eye undergoes peripheral retinal ablation, based on the clinical algorithm developed by investigators of the Early Treatment for ROP Trial that established new treatment indications for serious ROP in late 2003.5Early Treatment for Retinopathy of Prematurity Cooperative GroupRevised indications for the treatment of retinopathy of prematurity: Results of the early treatment for retinopathy of prematurity randomized trial.Arch Ophthalmol. 2003; 121: 1684-1694Crossref PubMed Scopus (1493) Google Scholar Answering this critical question is not straightforward, relying on a clinical judgment in which the examined baby’s fundus appearance is compared with a 20-year-old fundus photograph selected by ROP experts as demonstrating the “minimum” dilation and tortuosity sufficient to qualify as having plus disease.6Cryotherapy for Retinopathy of Prematurity Cooperative GroupMulticenter trial of cryotherapy for retinopathy of prematurity: Preliminary results.Pediatrics. 1988; 81: 697-706PubMed Google Scholar, 7Good W.V. Hardy R.J. Dobson V. et al.The incidence and course of retinopathy of prematurity: Findings from the early treatment for retinopathy of prematurity study.Pediatrics. 2005; 116: 15-23Crossref PubMed Scopus (383) Google Scholar Therein lies the dilemma—there is no quantitative standard for the diagnosis of plus disease—and yet the term has been clinically useful in describing an important ocular finding that indicates potentially sight-threatening retinopathy.The two groups of investigators have taken different and creative approaches to addressing the problem of plus disease diagnosis in ROP. Wallace and coworkers1Wallace D.K. Zhao Z. Freedman S.F. A pilot study using “ROPtool” to quantify plus disease in retinopathy of prematurity.J AAPOS. 2007; 11: 381-387Abstract Full Text Full Text PDF PubMed Scopus (91) Google Scholar have developed “ROPtool,” a semi-automated program that takes high-quality digital images and develops measures of vascular tortuosity and dilation by comparing scores with those of vessels in the reference image of plus disease used in clinical trials.6Cryotherapy for Retinopathy of Prematurity Cooperative GroupMulticenter trial of cryotherapy for retinopathy of prematurity: Preliminary results.Pediatrics. 1988; 81: 697-706PubMed Google Scholar, 7Good W.V. Hardy R.J. Dobson V. et al.The incidence and course of retinopathy of prematurity: Findings from the early treatment for retinopathy of prematurity study.Pediatrics. 2005; 116: 15-23Crossref PubMed Scopus (383) Google Scholar First, each quadrant of the 20 posterior pole images was judged by the two clinician authors to have vascular abnormalities consistent with plus disease, preplus disease (a term that indicates the presence of vascular abnormalities of the posterior pole insufficient to qualify as plus disease),3The International Classification of Retinopathy of Prematurity revisited.Arch Ophthalmol. 2005; 123: 991-999Crossref PubMed Scopus (2089) Google Scholar or no abnormal dilation and tortuosity (normal). Then images were analyzed independently by the same authors using ROPtool to determine a tortuosity index and dilation index for vessels they identified in each quadrant. Though certainly open to a criticism of recall bias, the results showed very good eye-level interobserver agreement on the presence of plus disease clinically (90%) and using ROPtool (95%). Using ROPtool in its current iteration, it appears that eye-level vascular tortuosity scores were more useful indicators of the presence of plus disease than were measures of vessel dilation, as shown with ROC curves showing high sensitivity and specificity for tortuosity and barely above chance for dilation.The second group of investigators, led by Chiang2Gelman R. Jiang L. Du Y.E. Martinez-Perez M.E. Flynn J.T. Chiang M.F. Plus disease in retinopathy of prematurity: Pilot study of computer-based and expert diagnosis.J AAPOS. 2007; 11: 532-540Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar at Columbia University, have chosen to examine the accuracy of ROP experts in judging the presence or absence of plus disease, based on the appearance of digital images that showed only the posterior pole. Based on a comparison to a consensus diagnosis developed for each the 34 images (ie, what most of the 22 experts judged the eye to demonstrate), they found that the “accuracy of ROP experts for plus disease diagnosis is imperfect.” This is not a novel observation. Clinicians disagree on treatment indications not infrequently as shown by the 12% disagreement between experienced observers on whether or not an eye fulfilled the criteria for treatment in the cryotherapy for ROP trial.8Reynolds J.D. Dobson V. Quinn G.E. et al.Evidence-based screening criteria for retinopathy of prematurity: Natural history data from the CRYO-ROP and LIGHT-ROP studies.Arch Ophthalmol. 2002; 120: 1470-1476Crossref PubMed Scopus (216) Google Scholar In the Gelman et al2Gelman R. Jiang L. Du Y.E. Martinez-Perez M.E. Flynn J.T. Chiang M.F. Plus disease in retinopathy of prematurity: Pilot study of computer-based and expert diagnosis.J AAPOS. 2007; 11: 532-540Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar study reported here, sensitivity and specificity for the presence of plus disease varied widely across the experts with only nine (40.9%) having both sensitivity and specificity of more than 80%. As in the Wallace article, the images then underwent computer-assisted analysis to determine estimates of curvature, diameter, and tortuosity for all vessels in the image. These parameters and various combinations of these parameters were then examined to determine sensitivity and specificity for detection of plus disease. The computer-assisted analyses generally did well and some combinations predicted better than the vast majority of the clinicians evaluating the posterior pole vessels.These efforts to use computerized systems to evaluate eyes for plus disease represent a start toward standardizing approaches to plus disease. However, there are years of work ahead as we determine the validity of imaging in ROP clinical diagnosis. At present, we should view the quantification of ROP abnormalities using digital images as important investigational exercises. Since a number of studies have shown disagreements in the diagnosis of plus disease on clinical examination,9Freedman S.F. Klystra J. Capowski J. Realini T. Rich C. Hunt D. Observer sensitivity to retinal vessel diameter and tortuosity in retinopathy of prematurity: A model system.J AAPOS. 1996; 33: 248-254Google Scholar, 10Chiang M. Keenan J. Starren J. et al.Accuracy and reliability of remote retinopathy of prematurity diagnosis.Arch Ophthalmol. 2006; 124: 322-327Crossref PubMed Scopus (107) Google Scholar using the clinical diagnosis of plus disease as the gold standard is not sufficient for validating this new tool. The luxury of examining, measuring, and comparing findings in detail, rather than simply discussing a cartoon/diagram of the ocular findings in a tiny baby who is none too happy to have us there examining eyes, provides an opportunity to develop quantitative scores of the various vascular abnormalities that characterize both the normal and abnormal retinal vascular development in the prematurely born baby. Initially these scores need to be based on what clinicians are currently calling normal, preplus disease, and plus disease and such an exercise may well help parse out the amorphous line between “near” plus and “clear” plus disease. Eye scores could then be used to develop a portfolio of images that provide examples of vascular abnormalities that the clinician could use during indirect ophthalmoscopy in the NICU to distinguish worrisome vascular abnormalities from nonthreatening ones. Ultimately, although, the utility of using vessel scores needs to be proved or disproved in clinical trials that examine the outcome of treatment decisions made on some combination of stage, location, and vascular score. Such studies will require participating institutions to have ready access to imaging and image evaluation, as well as a group of investigators who can maintain equipoise in a trial that may go against their clinical impression.Before we get to clinical trials, however, we need to determine the validity of using digital images to substitute for the eyes of the clinician performing a diagnostic examination. There are a variety of settings in which digital imaging and image analysis of eyes of babies at risk may prove very useful. Digital imaging has great potential as a screening tool when used to identify “referral-warranted ROP” (RW-ROP) as defined by Ells et al.11Ells A. Holmes J.M. Astle W. et al.Telemedicine approach to screening for severe retinopathy of prematurity: A pilot study.Ophthalmology. 2003; 110: 2113-2117Abstract Full Text Full Text PDF PubMed Scopus (190) Google Scholar By stepping back from developing an eye score that indicates that an eye needs treatment to using a score to indicate that the eye should be evaluated by an ophthalmologist familiar with ROP, digital imaging becomes clinically useful in areas of the United States that are underserved by experienced ophthalmologists and, importantly, useful in areas of developing countries, where ophthalmic expertise in ROP is sparse and where the “third epidemic” of blindness due to ROP is occurring.12Gilbert C. Fielder A. Gordillo L. et al.Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate, and high levels of development: Implications for screening programs.Pediatrics. 2005; 115: e518-e525Crossref PubMed Scopus (534) Google Scholar, 13Gilbert C. Rahi J. Eckstein M. O’Sullivan J. Foster A. Retinopathy of prematurity in middle-income countries.Lancet. 1997; 350: 12-24Abstract Full Text Full Text PDF PubMed Scopus (352) Google Scholar Such a use of digital retinal imaging is certainly appealing, but the cost in terms of cases missed and eyes blinded (ie, negative predictive value) must be carefully examined in several contexts. In the United States, where ophthalmic expertise is generally available and there are large expenditures on preserving lives of premature babies, the number of cases missed must necessarily be very low, while in areas of developing countries where no or minimal screening is currently available and resources are scarce, different values of sensitivity and specificity for detecting serious ROP may well be acceptable. Research in this important area must concentrate not only on the validity of digital imaging, but also on the societal impact of its use.There are barriers to the wide use of such image analysis including availability of computer-assisted evaluation of ROP in the NICU and the cost of digital imaging. The difficulty of introducing computerized ROP evaluation systems into the NICU was demonstrated in the ETROP study5Early Treatment for Retinopathy of Prematurity Cooperative GroupRevised indications for the treatment of retinopathy of prematurity: Results of the early treatment for retinopathy of prematurity randomized trial.Arch Ophthalmol. 2003; 121: 1684-1694Crossref PubMed Scopus (1493) Google Scholar when investigators chose not to recommend new treatment levels based on the risk determined by the computer-based RM-ROPII program14Hardy R.J. Palmer E.A. Dobson V. et al.Risk analysis of prethreshold retinopathy of prematurity.Arch Ophthalmol. 2003; 121: 1697-1701Crossref PubMed Scopus (97) Google Scholar on which the study was designed. Treatment recommendations were, instead, based on the clinical appearance of the eye; the patient demographic and pace of disease information that are integral to the RM-ROP program were essentially disregarded by assigning them to “clinical judgment.” This decision allowed the clinician to make the treatment decision at the bedside without the need for consulting a computerized program. Nonetheless, as technology advances, barriers to computerized methodologies for ROP evaluation will fall as the cost of imaging systems decreases and less imperfect systems for determining whether an eye needs treatment are developed. Using recent and future breakthroughs in digital retinal imaging technology, we in ophthalmology will hopefully embrace the opportunity to provide better care to our smallest patients. Two articles, one in the August issue of the Journal of AAPOS by Wallace et al1Wallace D.K. Zhao Z. Freedman S.F. A pilot study using “ROPtool” to quantify plus disease in retinopathy of prematurity.J AAPOS. 2007; 11: 381-387Abstract Full Text Full Text PDF PubMed Scopus (91) Google Scholar and one in this issue of the Journal of AAPOS by Gelman et al,2Gelman R. Jiang L. Du Y.E. Martinez-Perez M.E. Flynn J.T. Chiang M.F. Plus disease in retinopathy of prematurity: Pilot study of computer-based and expert diagnosis.J AAPOS. 2007; 11: 532-540Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar highlight the most vexing dilemma facing the ophthalmologist caring for prematurely born babies who develop “serious” retinopathy of prematurity (ROP): Are the posterior pole vessels of this eye sufficiently abnormal to qualify as having plus disease?3The International Classification of Retinopathy of Prematurity revisited.Arch Ophthalmol. 2005; 123: 991-999Crossref PubMed Scopus (2089) Google Scholar, 4An international classification of retinopathy of prematurity The Committee for the Classification of Retinopathy of Prematurity.Arch Ophthalmol. 1984; 102: 1130-1134Crossref PubMed Scopus (1123) Google Scholar The answer to this question largely determines whether or not an eye undergoes peripheral retinal ablation, based on the clinical algorithm developed by investigators of the Early Treatment for ROP Trial that established new treatment indications for serious ROP in late 2003.5Early Treatment for Retinopathy of Prematurity Cooperative GroupRevised indications for the treatment of retinopathy of prematurity: Results of the early treatment for retinopathy of prematurity randomized trial.Arch Ophthalmol. 2003; 121: 1684-1694Crossref PubMed Scopus (1493) Google Scholar Answering this critical question is not straightforward, relying on a clinical judgment in which the examined baby’s fundus appearance is compared with a 20-year-old fundus photograph selected by ROP experts as demonstrating the “minimum” dilation and tortuosity sufficient to qualify as having plus disease.6Cryotherapy for Retinopathy of Prematurity Cooperative GroupMulticenter trial of cryotherapy for retinopathy of prematurity: Preliminary results.Pediatrics. 1988; 81: 697-706PubMed Google Scholar, 7Good W.V. Hardy R.J. Dobson V. et al.The incidence and course of retinopathy of prematurity: Findings from the early treatment for retinopathy of prematurity study.Pediatrics. 2005; 116: 15-23Crossref PubMed Scopus (383) Google Scholar Therein lies the dilemma—there is no quantitative standard for the diagnosis of plus disease—and yet the term has been clinically useful in describing an important ocular finding that indicates potentially sight-threatening retinopathy. The two groups of investigators have taken different and creative approaches to addressing the problem of plus disease diagnosis in ROP. Wallace and coworkers1Wallace D.K. Zhao Z. Freedman S.F. A pilot study using “ROPtool” to quantify plus disease in retinopathy of prematurity.J AAPOS. 2007; 11: 381-387Abstract Full Text Full Text PDF PubMed Scopus (91) Google Scholar have developed “ROPtool,” a semi-automated program that takes high-quality digital images and develops measures of vascular tortuosity and dilation by comparing scores with those of vessels in the reference image of plus disease used in clinical trials.6Cryotherapy for Retinopathy of Prematurity Cooperative GroupMulticenter trial of cryotherapy for retinopathy of prematurity: Preliminary results.Pediatrics. 1988; 81: 697-706PubMed Google Scholar, 7Good W.V. Hardy R.J. Dobson V. et al.The incidence and course of retinopathy of prematurity: Findings from the early treatment for retinopathy of prematurity study.Pediatrics. 2005; 116: 15-23Crossref PubMed Scopus (383) Google Scholar First, each quadrant of the 20 posterior pole images was judged by the two clinician authors to have vascular abnormalities consistent with plus disease, preplus disease (a term that indicates the presence of vascular abnormalities of the posterior pole insufficient to qualify as plus disease),3The International Classification of Retinopathy of Prematurity revisited.Arch Ophthalmol. 2005; 123: 991-999Crossref PubMed Scopus (2089) Google Scholar or no abnormal dilation and tortuosity (normal). Then images were analyzed independently by the same authors using ROPtool to determine a tortuosity index and dilation index for vessels they identified in each quadrant. Though certainly open to a criticism of recall bias, the results showed very good eye-level interobserver agreement on the presence of plus disease clinically (90%) and using ROPtool (95%). Using ROPtool in its current iteration, it appears that eye-level vascular tortuosity scores were more useful indicators of the presence of plus disease than were measures of vessel dilation, as shown with ROC curves showing high sensitivity and specificity for tortuosity and barely above chance for dilation. The second group of investigators, led by Chiang2Gelman R. Jiang L. Du Y.E. Martinez-Perez M.E. Flynn J.T. Chiang M.F. Plus disease in retinopathy of prematurity: Pilot study of computer-based and expert diagnosis.J AAPOS. 2007; 11: 532-540Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar at Columbia University, have chosen to examine the accuracy of ROP experts in judging the presence or absence of plus disease, based on the appearance of digital images that showed only the posterior pole. Based on a comparison to a consensus diagnosis developed for each the 34 images (ie, what most of the 22 experts judged the eye to demonstrate), they found that the “accuracy of ROP experts for plus disease diagnosis is imperfect.” This is not a novel observation. Clinicians disagree on treatment indications not infrequently as shown by the 12% disagreement between experienced observers on whether or not an eye fulfilled the criteria for treatment in the cryotherapy for ROP trial.8Reynolds J.D. Dobson V. Quinn G.E. et al.Evidence-based screening criteria for retinopathy of prematurity: Natural history data from the CRYO-ROP and LIGHT-ROP studies.Arch Ophthalmol. 2002; 120: 1470-1476Crossref PubMed Scopus (216) Google Scholar In the Gelman et al2Gelman R. Jiang L. Du Y.E. Martinez-Perez M.E. Flynn J.T. Chiang M.F. Plus disease in retinopathy of prematurity: Pilot study of computer-based and expert diagnosis.J AAPOS. 2007; 11: 532-540Abstract Full Text Full Text PDF PubMed Scopus (63) Google Scholar study reported here, sensitivity and specificity for the presence of plus disease varied widely across the experts with only nine (40.9%) having both sensitivity and specificity of more than 80%. As in the Wallace article, the images then underwent computer-assisted analysis to determine estimates of curvature, diameter, and tortuosity for all vessels in the image. These parameters and various combinations of these parameters were then examined to determine sensitivity and specificity for detection of plus disease. The computer-assisted analyses generally did well and some combinations predicted better than the vast majority of the clinicians evaluating the posterior pole vessels. These efforts to use computerized systems to evaluate eyes for plus disease represent a start toward standardizing approaches to plus disease. However, there are years of work ahead as we determine the validity of imaging in ROP clinical diagnosis. At present, we should view the quantification of ROP abnormalities using digital images as important investigational exercises. Since a number of studies have shown disagreements in the diagnosis of plus disease on clinical examination,9Freedman S.F. Klystra J. Capowski J. Realini T. Rich C. Hunt D. Observer sensitivity to retinal vessel diameter and tortuosity in retinopathy of prematurity: A model system.J AAPOS. 1996; 33: 248-254Google Scholar, 10Chiang M. Keenan J. Starren J. et al.Accuracy and reliability of remote retinopathy of prematurity diagnosis.Arch Ophthalmol. 2006; 124: 322-327Crossref PubMed Scopus (107) Google Scholar using the clinical diagnosis of plus disease as the gold standard is not sufficient for validating this new tool. The luxury of examining, measuring, and comparing findings in detail, rather than simply discussing a cartoon/diagram of the ocular findings in a tiny baby who is none too happy to have us there examining eyes, provides an opportunity to develop quantitative scores of the various vascular abnormalities that characterize both the normal and abnormal retinal vascular development in the prematurely born baby. Initially these scores need to be based on what clinicians are currently calling normal, preplus disease, and plus disease and such an exercise may well help parse out the amorphous line between “near” plus and “clear” plus disease. Eye scores could then be used to develop a portfolio of images that provide examples of vascular abnormalities that the clinician could use during indirect ophthalmoscopy in the NICU to distinguish worrisome vascular abnormalities from nonthreatening ones. Ultimately, although, the utility of using vessel scores needs to be proved or disproved in clinical trials that examine the outcome of treatment decisions made on some combination of stage, location, and vascular score. Such studies will require participating institutions to have ready access to imaging and image evaluation, as well as a group of investigators who can maintain equipoise in a trial that may go against their clinical impression. Before we get to clinical trials, however, we need to determine the validity of using digital images to substitute for the eyes of the clinician performing a diagnostic examination. There are a variety of settings in which digital imaging and image analysis of eyes of babies at risk may prove very useful. Digital imaging has great potential as a screening tool when used to identify “referral-warranted ROP” (RW-ROP) as defined by Ells et al.11Ells A. Holmes J.M. Astle W. et al.Telemedicine approach to screening for severe retinopathy of prematurity: A pilot study.Ophthalmology. 2003; 110: 2113-2117Abstract Full Text Full Text PDF PubMed Scopus (190) Google Scholar By stepping back from developing an eye score that indicates that an eye needs treatment to using a score to indicate that the eye should be evaluated by an ophthalmologist familiar with ROP, digital imaging becomes clinically useful in areas of the United States that are underserved by experienced ophthalmologists and, importantly, useful in areas of developing countries, where ophthalmic expertise in ROP is sparse and where the “third epidemic” of blindness due to ROP is occurring.12Gilbert C. Fielder A. Gordillo L. et al.Characteristics of infants with severe retinopathy of prematurity in countries with low, moderate, and high levels of development: Implications for screening programs.Pediatrics. 2005; 115: e518-e525Crossref PubMed Scopus (534) Google Scholar, 13Gilbert C. Rahi J. Eckstein M. O’Sullivan J. Foster A. Retinopathy of prematurity in middle-income countries.Lancet. 1997; 350: 12-24Abstract Full Text Full Text PDF PubMed Scopus (352) Google Scholar Such a use of digital retinal imaging is certainly appealing, but the cost in terms of cases missed and eyes blinded (ie, negative predictive value) must be carefully examined in several contexts. In the United States, where ophthalmic expertise is generally available and there are large expenditures on preserving lives of premature babies, the number of cases missed must necessarily be very low, while in areas of developing countries where no or minimal screening is currently available and resources are scarce, different values of sensitivity and specificity for detecting serious ROP may well be acceptable. Research in this important area must concentrate not only on the validity of digital imaging, but also on the societal impact of its use. There are barriers to the wide use of such image analysis including availability of computer-assisted evaluation of ROP in the NICU and the cost of digital imaging. The difficulty of introducing computerized ROP evaluation systems into the NICU was demonstrated in the ETROP study5Early Treatment for Retinopathy of Prematurity Cooperative GroupRevised indications for the treatment of retinopathy of prematurity: Results of the early treatment for retinopathy of prematurity randomized trial.Arch Ophthalmol. 2003; 121: 1684-1694Crossref PubMed Scopus (1493) Google Scholar when investigators chose not to recommend new treatment levels based on the risk determined by the computer-based RM-ROPII program14Hardy R.J. Palmer E.A. Dobson V. et al.Risk analysis of prethreshold retinopathy of prematurity.Arch Ophthalmol. 2003; 121: 1697-1701Crossref PubMed Scopus (97) Google Scholar on which the study was designed. Treatment recommendations were, instead, based on the clinical appearance of the eye; the patient demographic and pace of disease information that are integral to the RM-ROP program were essentially disregarded by assigning them to “clinical judgment.” This decision allowed the clinician to make the treatment decision at the bedside without the need for consulting a computerized program. Nonetheless, as technology advances, barriers to computerized methodologies for ROP evaluation will fall as the cost of imaging systems decreases and less imperfect systems for determining whether an eye needs treatment are developed. Using recent and future breakthroughs in digital retinal imaging technology, we in ophthalmology will hopefully embrace the opportunity to provide better care to our smallest patients.