Inter-expert Agreement Research Articles

Automated Classification for Visual-Only Postmortem Inspection of Porcine Pathology

Several advantages would arise from the automated detection of pathologies of pig carcasses, including avoidance of the inherent risks of subjectivity and variability between human observers. Here, we develop a novel automated classification of two porcine offal pathologies at abattoir: a focal, localized pathology of the liver and a diffuse pathology of the heart, as cases in point. We develop a pattern recognition system based on machine learning to identify those organs that exhibit signs of the pathology of interest. Specifically, deep neural networks are trained to produce probability heat maps, highlighting regions on the surface of an organ potentially affected by a given condition. A final classification stage then decides whether a given organ is affected by the condition in question based on statistics computed from the heat map. We compare outcomes of automated classification with classification by expert pathologists. Results show the classification of liver and heart pathologies in agreement with an expert at levels comparable to, or exceeding, interexpert agreement. A system using methods such as those presented here has potential to overcome the limitations of human-based abattoir inspection, especially if this is based on visual-only inspection, and ultimately to provide a new gold standard for pathology. Note to Practitioners —The motivation for this article reflects the current requirement for visual-only inspection of livestock carcasses at slaughter houses and the need to provide a gold standard for recognition of carcass pathologies. Visual-only inspection is motivated by the need to reduce cross contamination between carcasses by manual palpation, but this leads to substantial variability in detection accuracy both within and between inspectors. This has significant public health implications. Here we present a system that comprises hardware to capture images of pig offal and software to analyze those images and identify cases of liver milk spots and hearts affected by pericarditis. It can classify high proportions of offal with accuracy comparable to that of veterinarians with extensive experience in pig pathology, thus demonstrating the potential to overcome the limitations of human-based abattoir inspection (especially if it is visual-only) and ultimately to provide a new gold standard. Our work is the first to address the automation of pig offal inspection, thus shedding light on the challenges associated with both appropriate image capture and successful image analysis, such as the need to cope with wide variations in the appearance of both normal and diseased organs, as well as different types of lesions and their impact on how much effort is required from experts in order to produce data needed to train the system. Future directions of work should include extending the system to identify more pathologies and implementing a real-time system to cope with production line speed.

Analysis of inter-expert agreement when working with the AOSpine classification (TLCS, 2013): our experience, questions and contradictions

Objective . To assess inter-expert agreement among spine surgeons having different levels of clinical experience when working with the AOSpine classification (TLCS, 2013). Materials and Methods . The study involved nine surgeons divided into three equal groups depending on work experience. All respondents were asked to classify the MSCT data of 50 patients with acute injuries to the thoracic and lumbar spine pursuant to TLCS (2013) classification. To evaluate inter-expert agreement, a Kappa coefficient interpreted according to Landis – Koch criteria was used. Results. The overall coefficient of inter-expert agreement for all observations among all groups of respondents was 0.43, which reflects a moderate level of agreement. Moderate inter-expert agreement was revealed for injury types A (0.45) and C (0.56), and satisfactory – for type B (0.34). The highest levels of agreement were obtained for subtypes A1 (0.67) and A4 (0.80) in the group of advanced specialists and for type C (0.70) in the group of specialists with a basic level of experience. Conclusion. The study demonstrated predominantly moderate level of inter-expert agreement when working with the AOSpine classification (TLCS, 2013). The accuracy of its use increases with a gain in practical experience of a surgeon.

Recommendations of older adults on how to use the PROM 'TOPICS-MDS' in healthcare conversations: A Delphi study.

In shared decision making, the exploration of preferred personal health outcomes is important. Patient-reported outcome measures (PROMs) provide input for discussions between patients and healthcare professionals. The Older Persons and Informal Caregivers Survey Minimum DataSet (TOPICS-MDS) PROM is a multidimensional questionnaire on the physical and mental health and wellbeing of older adults. This study investigates how the TOPICS-MDS could be used in individual healthcare conversations. We explored views of older adults regarding 1) whether the health domains they want to discuss are included in the TOPICS-MDS and 2) the comprehensibility of the TOPICS-MDS for healthcare conversations with older adults. A three-round Delphi study was conducted. A total of 57 older adults participated in the study, the mean (SD) age was 71.5 (8.5) years, and 78.9% of the participants were female. The participants were divided into four panels based on educational level and cultural background. We used online questionnaires and focus groups. Consensus was pre-defined to be the point when ≥75% of the participants agreed that a domain was important or very important (scored on a 5-point Likert scale). The inter-expert agreement was computed for Round 1 and 3 with Kendall's W. Round 2 was a focus-group. Qualitative data were analyzed by content analysis. Older adults considered 'functional limitations', 'emotional wellbeing', 'social functioning' and 'quality of life' to be important domains of the TOPICS-MDS to discuss in healthcare conversations. The participants added 'coping with stress', 'dealing with health conditions and the effects on life' as extra domains for healthcare conversations. Challenges regarding the comprehensibility of the TOPICS-MDS included difficult words and lengthy or sensitive questions. Questions that included multiple topics were difficult to understand. The TOPICS-MDS covers the domains of life that older adults value as important to discuss with healthcare professionals, and two additional domains were identified. For older adults with a low level of education or a culturally diverse background, the TOPICS-MDS needs to be adjusted for comprehensibility.

Fetal heart rate baseline computation with a weighted median filter

BackgroundAutomated fetal heart rate (FHR) analysis removes inter- and intra-expert variability, and is a promising solution for reducing the occurrence of fetal acidosis and the implementation of unnecessary medical procedures. The first steps in automated FHR analysis are determination of the baseline, and detection of accelerations and decelerations (A/D). We describe a new method in which a weighted median filter baseline (WMFB) is computed and A/Ds are then detected. MethodThe filter weightings are based on the prior probability that the sampled FHR is in the baseline state or in an A/D state. This probability is computed by estimating the signal’s stability at low frequencies and by progressively trimming the signal. Using a competition dataset of 90 previously annotated FHR recordings, we evaluated the WMFB method and 11 recently published literature methods against the ground truth of an expert consensus. The level of agreement between the WMFB method and the expert consensus was estimated by calculating several indices (primarily the morphological analysis discordance index, MADI). The agreement indices were then compared with the values for eleven other methods. We also compared the level of method-expert agreement with the level of interrater agreement. ResultsFor the WMFB method, the MADI indicated a disagreement of 4.02% vs. the consensus; this value is significantly lower (p<10−13) than that calculated for the best of the 11 literature methods (7.27%, for Lu and Wei’s empirical mode decomposition method). The level of inter-expert agreement (according to the MADI) and the level of WMFB-expert agreement did not differ significantly (p=0.22). ConclusionThe WMFB method reproduced the expert consensus analysis better than 11 other methods. No differences in performance between the WMFB method and individual experts were observed. The method Matlab source code is available under General Public Licence at http://utsb.univ-catholille.fr/fhr-wmfb.

Consensus among healthcare stakeholders on a collaborative medication therapy management model for chronic diseases in Malaysia; A Delphi study.

BackgroundThe general problem is lack of inter-professional collaboration and the way private primary care responds to manage chronic diseases in Malaysia. Absence of prescription review, inadequate patient education, the highest percentage of prescribing errors and half of the chronic disease patients are nonadherent. Medicines are the most common and life long used interventions in chronic diseases. Hence, the need to manage medicine in chronic diseases becomes obligatory. As both general practitioner and community pharmacist can dispense medications, this has resulted in a business rivalry. There is a need to build consensus among various healthcare stakeholders for a collaborative medication therapy management model (CMTM) where community pharmacist has an active role in chronic care.MethodThis study utilized modified e-Delphi method to build consensus. A validated e-Delphi survey was administered to a purposive sample of 29 experts. Consensus was pre-defined to be the point where >85% of the experts fall in either agree or strongly agree category for each statement. The inter-expert agreement was computed in both rounds using Intra-class correlation coefficient and Kendall's W. Delphi operates in an iterative fashion till there comes stability in responses. At the end of each round, experts were provided aggregate response, their own response and choice to change their response in the light of aggregate response.ResultsResponse rate was 70.73% and 100% in 1st and 2nd round, respectively. Consensus was achieved on 119/132 statements which mainly referred to the need, structural and regulatory aspects of CMTM model in Malaysia. However, there were some flashpoints on dispensing separation and means to finance this model. Stability in response of experts was achieved after 2nd round; hence, no next round was executed.ConclusionOverall, the study findings witnessed the expert panel’s support for the CMTM model. Study helped to sketch CMTM model and facilitated development of some recommendations to the authorities which may help to formulate a policy to bring CPs under a working relationship with GPs. Hence, this study should be taken as a call for redefining of the roles of CPs and GPs in Malaysia.

Multidisciplinary team meeting for complex bone and joint infections diagnosis: The PHICTOS study.

In France, the most severe bone and joint infections (BJI), called "complex" (CBJI), are assessed in a multidisciplinary team meeting (MTM) in a reference center. However, the definition of CBJI, drawn up by the Health Ministry, is not consensual between physicians. The objective was to estimate the agreement for CBJI classification. Initially, five experts from one MTM classified twice, one-month apart, 24 cases as non-BJI, simple BJI or CBJI, using the complete medical record. Secondly, six MTMs classified the same cases using standardized information. Agreements were estimated using Fleiss and Cohen kappa (κ) coefficients. Inter-expert agreement during one MTM was moderate (κ=0.49), and fair (κ=0.23) when the four non-BJIs were excluded. Intra-expert agreement was moderate (κ=0.50, range 0.27-0.90), not improved with experience. The overall inter-MTM agreement was moderate (κ=0.58), it was better between MTMs with professor (κ=0.65) than without (κ=0.51) and with longer median time per case (κ=0.60) than shorter (κ=0.47). When the four non-BJIs were excluded, the overall agreement decreased (κ=0.40). The first step confirmed the heterogeneity of CBJI classification between experts. The seemingly better inter-MTM than inter-expert agreement could be an argument in favour of MTMs, which are moreover a privileged place to enhance expertise. Further studies are needed to assess these results as well as the quality of care and medico-economic outcomes after a MTM.

Recurrent Deep Neural Networks for Real-Time Sleep Stage Classification From Single Channel EEG.

Objective: We investigate the design of deep recurrent neural networks for detecting sleep stages from single channel EEG signals recorded at home by non-expert users. We report the effect of data set size, architecture choices, regularization, and personalization on the classification performance.Methods: We evaluated 58 different architectures and training configurations using three-fold cross validation.Results: A network consisting of convolutional (CONV) layers and long short term memory (LSTM) layers can achieve an agreement with a human annotator of Cohen's Kappa of ~0.73 using a training data set of 19 subjects. Regularization and personalization do not lead to a performance gain.Conclusion: The optimal neural network architecture achieves a performance that is very close to the previously reported human inter-expert agreement of Kappa 0.75.Significance: We give the first detailed account of CONV/LSTM network design process for EEG sleep staging in single channel home based setting.

사상체질 진단 연구의 전문가 일치도와 진단 정확률

사상체질 진단 연구의 전문가 일치도와 진단 정확률

Agreement between expert thoracic radiologists and the chest radiograph reports provided by consultant radiologists and reporting radiographers in clinical practice: Review of a single clinical site

IntroductionTo compare the clinical chest radiograph (CXR) reports provided by consultant radiologists and reporting radiographers with expert thoracic radiologists. MethodsAdult CXRs (n = 193) from a single site were included; 83% randomly selected from CXRs performed over one year, and 17% selected from the discrepancy meeting. Chest radiographs were independently interpreted by two expert thoracic radiologists (CTR1/2).Clinical history, previous and follow-up imaging was available, but not the original clinical report. Two arbiters compared expert and clinical reports independently. Kappa (Ƙ), Chi Square (χ2) and McNemar tests were performed to determine inter-observer agreement. ResultsCTR1 interpreted 187 (97%) and CTR2 186 (96%) CXRs, with 180 CXRs interpreted by both experts. Radiologists and radiographers provided 93 and 87 of the original clinical reports respectively. Consensus between both expert thoracic radiologists and the radiographer clinical report was 70 (CTR1; Ƙ = 0.59) and 70 (CTR2; Ƙ = 0.62), and comparable to agreement between expert thoracic radiologists and the radiologist clinical report (CTR1 = 76, Ƙ = 0.60; CTR2 = 75, Ƙ = 0.62). Expert thoracic radiologists agreed in 131 cases (Ƙ = 0.48). There was no difference in agreement between either expert thoracic radiologist, when the clinical report was provided by radiographers or radiologists (CTR1 χ = 0.056, p = 0.813; CTR2 χ = 0.014, p = 0.906), or when stratified by inter-expert agreement; radiographer McNemar p = 0.629 and radiologist p = 0.701. ConclusionEven when weighted with chest radiographs reviewed at discrepancy meetings, content of CXR reports from trained radiographers were indistinguishable from content of reports issued by radiologists and expert thoracic radiologists.

A single channel sleep-spindle detector based on multivariate classification of EEG epochs: MUSSDET

BackgroundStudies on sleep-spindles are typically based on visual-marks performed by experts, however this process is time consuming and presents a low inter-expert agreement, causing the data to be limited in quantity and prone to bias. An automatic detector would tackle these issues by generating large amounts of objectively marked data. New MethodOur goal was to develop a sensitive, precise and robust sleep-spindle detection method. Emphasis has been placed on achieving a consistent performance across heterogeneous recordings and without the need for further parameter fine tuning. The developed detector runs on a single channel and is based on multivariate classification using a support vector machine. Scalp-electroencephalogram recordings were segmented into epochs which were then characterized by a selection of relevant and non-redundant features. The training and validation data came from the Medical Center-University of Freiburg, the test data consisted of 27 records coming from 2 public databases. ResultsUsing a sample based assessment, 53% sensitivity, 37% precision and 96% specificity was achieved on the DREAMS database. On the MASS database, 77% sensitivity, 46% precision and 96% specificity was achieved. The developed detector performed favorably when compared to previous detectors. The classification of normalized EEG epochs in a multidimensional space, as well as the use of a validation set, allowed to objectively define a single detection threshold for all databases and participants. ConclusionsThe use of the developed tool will allow increasing the data-size and statistical significance of research studies on the role of sleep-spindles.

Inter-expert agreement and similarity analysis of traditional diagnoses and acupuncture prescriptions in textbook- and pragmatic-based practices

This study examined (1) the agreement of acupuncture experts with textbook prescriptions and among themselves, and (2) the association between similar traditional diagnoses and textbook acupuncture prescriptions, examining whether pragmatic practice (i.e., modifying prescriptions according to personal clinical practice) alters such an association. A computational analysis quantified the diagnosis-prescription association from a textbook. Eight acupuncture experts were independently interviewed. Experts modified the textbook prescriptions according to their pragmatic practice. Experts mostly agreed (19–90%) or strongly agreed (0–29%) with the textbook prescriptions, with no-better-than-chance agreement on their ratings (Light's κ = 0.036, CI95% = [0.003; 0.081]). The number of manifestations in traditional diagnoses weakly explains the variability (Spearman's ρ = 0.260, p = 0.038) of the number of acupoints in prescriptions. The association between similar traditional diagnoses and acupuncture prescriptions is strong in the textbook (γ = 0.720, CI95% = [0.658, 0.783]), whereas pragmatic practice had little effect on this association (γ = 0.724–0.769).

The effect of reducing EEG electrode number on the visual interpretation of the human expert for neonatal seizure detection

ObjectivesTo measure changes in the visual interpretation of the EEG by the human expert for neonatal seizure detection when reducing the number of recording electrodes. MethodsEEGs were recorded from 45 infants admitted to the neonatal intensive care unit (NICU). Three experts annotated seizures in EEG montages derived from 19, 8 and 4 electrodes. Differences between annotations were assessed by comparing intra-montage with inter-montage agreement (K). ResultsThree experts annotated 4464 seizures across all infants and montages. The inter-expert agreement was not significantly altered by the number of electrodes in the montage (p = 0.685, n = 43). Reducing the number of EEG electrodes altered the seizure annotation for all experts. Agreement between the 19-electrode montage (K19,19 = 0.832) was significantly higher than the agreement between 19 and 8-electrode montages (dK = 0.114; p < 0.001, n = 42) or 19 and 4-electrode montages (dK = 0.113, p < 0.001, n = 43). Seizure burden and number were significantly underestimated by the 4 and 8-electrode montage (p < 0.001). No significant difference in agreement was found between 8 and 4-electrode montages (dK = 0.002; p = 0.07, n = 42). ConclusionsReducing the number of EEG electrodes from 19 electrodes resulted in slight but significant changes in seizure detection. SignificanceFour-electrode montages for routine EEG monitoring are comparable to eight electrodes for seizure detection in the NICU.

Three dimensional kinematic analyses of finger movement control and association to brain activity responses – A pilot study on healthy individuals

To measure the inter-expert and intra-expert agreement in sleep spindle scoring, and to quantify how many experts are needed to build a reliable dataset of sleep spindle scorings.The EEG dataset was comprised of 400 randomly selected 115 s segments of stage 2 sleep from 110 sleeping subjects in the general population (57 ± 8, range: 42–72 years). To assess expert agreement, a total of 24 Registered Polysomnographic Technologists (RPSGTs) scored spindles in a subset of the EEG dataset at a single electrode location (C3-M2). Intra-expert and inter-expert agreements were calculated as F1-scores, Cohen’s kappa (κ), and intra-class correlation coefficient (ICC).We found an average intra-expert F1-score agreement of 72 ± 7% (κ: 0.66 ± 0.07). The average inter-expert agreement was 61 ± 6% (κ: 0.52 ± 0.07). Amplitude and frequency of discrete spindles were calculated with higher reliability than the estimation of spindle duration. Reliability of sleep spindle scoring can be improved by using qualitative confidence scores, rather than a dichotomous yes/no scoring system.We estimate that 2–3 experts are needed to build a spindle scoring dataset with ‘substantial’ reliability (κ: 0.61–0.8), and 4 or more experts are needed to build a dataset with ‘almost perfect’ reliability (κ: 0.81–1).Spindle scoring is a critical part of sleep staging, and spindles are believed to play an important role in development, aging, and diseases of the nervous system.

Improved Automatic Segmentation of White Matter Hyperintensities in MRI Based on Multilevel Lesion Features.

Brain white matter hyperintensities (WMHs) are linked to increased risk of cerebrovascular and neurodegenerative diseases among the elderly. Consequently, detection and characterization of WMHs are of significant clinical importance. We propose a novel approach for WMH segmentation from multi-contrast MRI where both voxel-based and lesion-based information are used to improve overall performance in both volume-oriented and object-oriented metrics. Our segmentation method (AMOS-2D) consists of four stages following a "generate-and-test" approach: pre-processing, Gaussian white matter (WM) modelling, hierarchical multi-threshold WMH segmentation and object-based WMH filtering using support vector machines. Data from 28 subjects was used in this study covering a wide range of lesion loads. Volumetric T1-weighted images and 2D fluid attenuated inversion recovery (FLAIR) images were used as basis for the WM model and lesion masks defined manually in each subject by experts were used for training and evaluating the proposed method. The method obtained an average agreement (in terms of the Dice similarity coefficient, DSC) with experts equivalent to inter-expert agreement both in terms of WMH number (DSC=0.637 vs. 0.651) and volume (DSC=0.743 vs. 0.781). It allowed higher accuracy in detecting WMH compared to alternative methods tested and was further found to be insensitive to WMH lesion burden. Good agreement with expert annotations combined with stable performance largely independent of lesion burden suggests that AMOS-2D will be a valuable tool for fully automated WMH segmentation in patients with cerebrovascular and neurodegenerative pathologies.

Gold-standard for computer-assisted morphological sperm analysis

Background and Objective: Published algorithms for classification of human sperm heads are based on relatively small image databases that are not open to the public, and thus no direct comparison is available for competing methods. We describe a gold-standard for morphological sperm analysis (SCIAN-MorphoSpermGS), a dataset of sperm head images with expert-classification labels in one of the following classes: normal, tapered, pyriform, small or amorphous. This gold-standard is for evaluating and comparing known techniques and future improvements to present approaches for classification of human sperm heads for semen analysis. Although this paper does not provide a computational tool for morphological sperm analysis, we present a set of experiments for comparing sperm head description and classification common techniques. This classification base-line is aimed to be used as a reference for future improvements to present approaches for human sperm head classification.Methods: The gold-standard provides a label for each sperm head, which is achieved by majority voting among experts. The classification base-line compares four supervised learning methods (1− Nearest Neighbor, naive Bayes, decision trees and Support Vector Machine (SVM)) and three shape-based descriptors (Hu moments, Zernike moments and Fourier descriptors), reporting the accuracy and the true positive rate for each experiment. We used Fleiss' Kappa Coefficient to evaluate the inter-expert agreement and Fisher's exact test for inter-expert variability and statistical significant differences between descriptors and learning techniques.Results: Our results confirm the high degree of inter-expert variability in the morphological sperm analysis. Regarding the classification base line, we show that none of the standard descriptors or classification approaches is best suitable for tackling the problem of sperm head classification. We discovered that the correct classification rate was highly variable when trying to discriminate among non-normal sperm heads. By using the Fourier descriptor and SVM, we achieved the best mean correct classification: only 49%.Conclusions: We conclude that the SCIAN-MorphoSpermGS will provide a standard tool for evaluation of characterization and classification approaches for human sperm heads. Indeed, there is a clear need for a specific shape-based descriptor for human sperm heads and a specific classification approach to tackle the problem of high variability within subcategories of abnormal sperm cells.

Plus Disease in Retinopathy of Prematurity: Diagnostic Trends in 2016 Versus 2007

To identify any temporal trends in the diagnosis of plus disease in retinopathy of prematurity (ROP) by experts. Reliability analysis. ROP experts were recruited in 2007 and 2016 to classify 34 wide-field fundus images of ROP as plus, pre-plus, or normal, coded as "3," "2," and "1," respectively, in the database. The main outcome was the average calculated score for each image in each cohort. Secondary outcomes included correlation on the relative ordering of the images in 2016 vs 2007, interexpert agreement, and intraexpert agreement. The average score for each image was higher for 30 of 34 (88%) images in 2016 compared with 2007, influenced by fewer images classified as normal (P < .01), a similar number of pre-plus (P= .52), and more classified as plus (P < .01). The mean weighted kappa values in 2006 were 0.36 (range 0.21-0.60), compared with 0.22 (range 0-0.40) in 2016. There was good correlation between rankings of disease severity between the 2 cohorts (Spearman rank correlation ρ= 0.94), indicating near-perfect agreement on relative disease severity. Despite good agreement between cohorts on relative disease severity ranking, the higher average score and classifications for each image demonstrate that experts are diagnosing pre-plus and plus disease at earlier stages of disease severity in 2016, compared with 2007. This has implications for patient care, research, and teaching, and additional studies are needed to better understand this temporal trend in image-based plus disease diagnosis.

Plus Disease in Retinopathy of Prematurity: More Than Meets the ICROP?

Retinopathy of prematurity (ROP), a vasoproliferative retinal disease affecting premature infants, is a leading cause of childhood blindness throughout the world. Plus disease, defined as venous dilatation and arteriolar tortuosity within the posterior retinal vessels greater than or equal to that of a standard published photograph, is the most critical finding in identifying treatment-requiring ROP. Despite an internationally accepted definition of plus disease, there is significant variability in diagnostic process and outcome, producing variable levels of reported intra- and interexpert agreement. Several potential explanations for poor agreement have been proposed, including attention to undefined vascular features such as venous tortuosity, focus on narrower or wider field of view, unfamiliarity with digital images, the magnification and apparent severity of the standard photograph, and cut-off point differences among experts as to the level of tortuosity and dilation sufficient for "plus disease" along a continuum. Moreover, differences in diagnostic consistency among groups of experts separated both geographically and chronologically have been reported. These findings have implications for clinical care, research, and education, and highlight the need for a more precise definition of plus disease and objective diagnostic methods for ROP.

Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability

To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP). We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD. Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units. Expert classification of images of plus disease in ROP. Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement). There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0-0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2= 0.82; and dataset B: P < 0.05 and adjusted R2=0.6615). There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.

Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis

To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP. We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases. Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units. Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system. Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling. There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06-0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74-0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95-0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86). Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.

Diagnostic Discrepancies in Retinopathy of Prematurity Classification

To identify the most common areas for discrepancy in retinopathy of prematurity (ROP) classification between experts. Prospective cohort study. A total of 281 infants were identified as part of a multicenter, prospective, ROP cohort study from 7 participating centers. Each site had participating ophthalmologists who provided the clinical classification after routine examination using binocular indirect ophthalmoscopy (BIO) and obtained wide-angle retinal images, which were independently classified by 2 study experts. Wide-angle retinal images (RetCam; Clarity Medical Systems, Pleasanton, CA) were obtained from study subjects, and 2 experts evaluated each image using a secure web-based module. Image-based classifications for zone, stage, plus disease, and overall disease category (no ROP, mild ROP, type II or pre-plus, and type I) were compared between the 2 experts and with the clinical classification obtained by BIO. Inter-expert image-based agreement and image-based versus ophthalmoscopic diagnostic agreement using absolute agreement and weighted kappa statistic. A total of 1553 study eye examinations from 281 infants were included in the study. Experts disagreed on the stage classification in 620 of 1553 comparisons (40%), plus disease classification (including pre-plus) in 287 of 1553 comparisons (18%), zone in 117 of 1553 comparisons (8%), and overall ROP category in 618 of 1553 comparisons (40%). However, agreement for presence versus absence of type 1 disease was >95%. There were no differences between image-based and clinical classification except for zone III disease. The most common area of discrepancy in ROP classification is stage, although inter-expert agreement for clinically significant disease, such as presence versus absence of type 1 and type 2 disease, is high. There were no differences between image-based grading and clinical examination in the ability to detect clinically significant disease. This study provides additional evidence that image-based classification of ROP reliably detects clinically significant levels of ROP with high accuracy compared with the clinical examination.