Interobserver Variability Study Research Articles

OP33.03: Visibility and dimensions of Caesarean section scars in pregnancy—an interobserver variability study

OP33.03: Visibility and dimensions of Caesarean section scars in pregnancy—an interobserver variability study

Validation of the FOUR Score (Spanish Version) in Acute Stroke: An Interobserver Variability Study

Background: Methods to assess impaired consciousness in acute stroke typically include the Glasgow Coma Scale (GCS), but the verbal component has limitations in aphasic or intubated patients. The FOUR (Full Outline of UnResponsiveness) score, a new coma scale, evaluates 4 components: eye and motor responses, brainstem reflexes and respiration. We aimed to study the interobserver variability of the FOUR score in acute stroke patients. Methods: We prospectively enrolled consecutive patients with acute stroke admitted from February to July 2008 to the stroke unit of our Neurology Department. Patients were evaluated by neurology residents and nurses using the FOUR score and the GCS. For both scales, we obtained paired and total weighted kappa values (Kw) and intraclass correlation coefficients (ICC). NIH stroke scale was also recorded on admission. Results: We obtained a total of 75 paired evaluations in 60 patients (41 cerebral infarctions, 15 cerebral hemorrhages and 4 transient ischemic attacks). Thirty-three (55%) patients were alert, 17 (28.3%) drowsy and 10 (16.7%) stuporous or comatose. The overall rater agreement was excellent in the FOUR score (Kw 0.93; 95% CI 0.89–0.97) with an ICC of 0.94 (95% CI 0.91–0.96) and in the GCS (Kw 0.96; 95% CI 0.94–0.98) with an ICC of 0.96 (95% CI 0.93–0.97). A good correlation was found between the FOUR score and the GCS (ρ 0.83; p < 0.01) and between the FOUR score and the NIH stroke scale (ρ –0.78; p < 0.001). Conclusions: The FOUR score is a reliable scale for evaluating the level of consciousness in acute stroke patients, showing a good correlation with the GCS and the NIH stroke scale.

Comparative Effectiveness Research: What It Means for Radiology

The coming years will be an ideal time for investigators in radiology to develop a comparative effectiveness research (CER) initiative and to establish a lasting precedent for CER in our field.

6521 Analysis of patterns of failure after a study of interobserver variability in target volume delineation in postoperative radiochemotherapy for gastric cancer

6521 Analysis of patterns of failure after a study of interobserver variability in target volume delineation in postoperative radiochemotherapy for gastric cancer

Reproducibility of coronary artery plaque volume and composition quantification by 64-detector row coronary computed tomographic angiography: An intraobserver, interobserver, and interscan variability study

Interscan variability of coronary arterial plaque volume and composition quantification with coronary computed tomographic angiography (CCTA), an important attribute when considering CCTA as a serial modality, has not been examined. We sought to systematically determine intraobserver- and interobserver-interscan reproducibility of these measures. Two blinded, experienced readers independently evaluated proximal coronary segments on CCTAs from 30 patients who underwent 2 scans within 200 days (median, 124 days; interquartile range, 49-155 days) without experiencing an interim acute coronary event. Readers recorded number of plaques and, in plaques that met a preset minimal length criterion, quantified total, calcified plaque (CP), and noncalcified plaque (NCP) volumes and percentage of total plaque volume occupied by NCP. Of 89 total segments studied, 36 contained detectable plaque, and 26 met criterion for quantification. Intraobserver, interobserver, and interscan agreements for normal segments were 100%. Intraobserver-interscan correlations of total, CP, and NCP volumes and percentage of NCP were excellent (r=0.93-0.97, P values<0.001). Interobserver-interscan correlations for all measures were also very good (r=0.81-0.96, P values<0.001). Variability in plaque volume quantification was significant, exceeding 60% of the averaged paired plaque volumes in the best-case scenario (interobserver-interscan CP volume). Quantification of percentage of NCP composition by volume was more consistent, with <24% variation in the worst-case scenario (interobserver-interscan). CCTA shows promise for quantifying serial coronary plaque change. Currently, the most robust measure seems to be percentage of plaque composition, rather than plaque volume. For smaller plaques, volume quantification remains challenging.

Intestinal Motility Assessment With Video Capsule Endoscopy: Automatic Annotation of Phasic Intestinal Contractions

Intestinal motility assessment with video capsule endoscopy arises as a novel and challenging clinical fieldwork. This technique is based on the analysis of the patterns of intestinal contractions shown in a video provided by an ingestible capsule with a wireless micro-camera. The manual labeling of all the motility events requires large amount of time for offline screening in search of findings with low prevalence, which turns this procedure currently unpractical. In this paper, we propose a machine learning system to automatically detect the phasic intestinal contractions in video capsule endoscopy, driving a useful but not feasible clinical routine into a feasible clinical procedure. Our proposal is based on a sequential design which involves the analysis of textural, color, and blob features together with SVM classifiers. Our approach tackles the reduction of the imbalance rate of data and allows the inclusion of domain knowledge as new stages in the cascade. We present a detailed analysis, both in a quantitative and a qualitative way, by providing several measures of performance and the assessment study of interobserver variability. Our system performs at 70% of sensitivity for individual detection, whilst obtaining equivalent patterns to those of the experts for density of contractions.

The Condylar Cutoff Sign: Quantifying Lateral Femoral Condylar Hypoplasia in a Complete Discoid Meniscus

We describe the condylar cutoff sign, a radiographic sign in knees with a discoid lateral meniscus, for diagnosis of complete discoid meniscus. We retrospectively reviewed tunnel-view radiographs of 100 patients (100) knees including 50 with complete discoid lateral meniscus and 50 with normal menisci. All patients were 18 years of age or older. All of the knees were arthroscopically examined. We developed a method to measure the prominence of the femoral condyle adjacent to the intercondylar notch on a tunnel view of the knee. The prominence ratio, the ratio of the medial and lateral condylar prominence, was compared and analyzed. The measurements were performed by three observers on two separate occasions to determine reliability. The intraobserver and interobserver variability study revealed high reliability with correlation coefficients ranging from 0.86 to 0.99. Using a cutoff ratio of 0.8, we observed a major difference between the two groups and found the ratio had 76% sensitivity, 96% specificity, 95% positive predictive value, and 80% negative predictive value in every set of measurements. The sign has high specificity and reliability but might not apply to younger patients. Level II, diagnostic study. See the Guidelines for Authors for a complete description of levels of evidence.

Early Invasive Cervical Cancer: CT and MR Imaging in Preoperative Evaluation—ACRIN/GOG Comparative Study of Diagnostic Performance and Interobserver Variability

To retrospectively compare diagnostic performance and interobserver variability for computed tomography (CT) and magnetic resonance (MR) imaging in the pretreatment evaluation of early invasive cervical cancer, with surgical pathologic findings as the reference standard. This HIPAA-compliant study had institutional review board approval and informed consent for evaluation of preoperative CT (n = 146) and/or MR imaging (n = 152) studies in 156 women (median age, 43 years; range, 22-81 years) from a previous prospective multicenter American College of Radiology Imaging Network and Gynecologic Oncology Group study of 172 women with biopsy-proved cervical cancer (clinical stage > or = IB). Four radiologists (experience, 7-15 years) interpreted the CT scans, and four radiologists (experience, 12-20 years) interpreted the MR studies retrospectively. Tumor visualization and detection of parametrial invasion were assessed with receiver operating characteristic curves (with P < or = .05 considered to indicate a significant difference). Descriptive statistics for staging and kappa statistics for reader agreement were calculated. Surgical pathologic findings were the reference standard. For CT and MR imaging, respectively, multirater kappa values were 0.26 and 0.44 for staging, 0.16 and 0.32 for tumor visualization, and -0.04 and 0.11 for detection of parametrial invasion; for advanced stage cancer (> or =IIB), sensitivities were 0.14-0.38 and 0.40-0.57, positive predictive values (PPVs) were 0.38-1.00 and 0.32-0.39, specificities were 0.84-1.00 and 0.77-0.80, and negative predictive values (NPVs) were 0.81-0.84 and 0.83-0.87. MR imaging was significantly better than CT for tumor visualization (P < .001) and detection of parametrial invasion (P = .047). Reader agreement was higher for MR imaging than for CT but was low for both. MR imaging was significantly better than CT for tumor visualization and detection of parametrial invasion. The modalities were similar for staging, sharing low sensitivity and PPV but relatively high NPV and specificity.

A Novel Thyroid Phantom for Ultrasound Volumetry: Determination of Intraobserver and Interobserver Variability

A novel thyroid ultrasound phantom with tissue-equivalent characteristics was designed consisting of two lobes with three lesions each. One set of lesions is manufactured with a -5 dB echo difference to the surrounding tissue, the other with -10 dB. This phantom was used as a standardized measuring object for reproducibility of two-dimensional and three-dimensional ultrasound volumetry and for an interobserver and intraobserver variability study. For the variability study, nine experienced physicians scanned all specimen three times. Each time the volumes were calculated using the ellipsoid method. A three-dimensional ultrasound scan of each specimen was performed to evaluate all volumes by multiplanar volume approximation. The results of these volume data were compared to the known true volumes. The interobserver variability ranged from -13.4% to 11.9% (median, 0.7%); the intraobserver variability from -9.1% to 16.4% (median, 3.6%). The systematic error as calculated from the total mean of all specimens is 0.5% for the interobserver variability and 4.1% for the intraobserver variability. The phantom can be used for training purposes, to improve the skills of the examining physicians by simulating real thyroid morphology, to provide a standardized reference object for long-term quality control of conventional ultrasound scanners, and the determination of the accuracy and reproducibility of volumetry using three-dimensional ultrasound systems.

Comparison of optic nerve head assessment with a digital stereoscopic camera (discam), scanning laser ophthalmoscopy, and stereophotography

Purpose To compare computer-assisted planimetry using the Discam system (Marcher Enterprises Ltd., Hereford, UK), confocal scanning laser ophthalmoscopy (CSLO), and stereoscopic disc photography with respect to optic nerve head (ONH) measurements and glaucoma status. Design Comparative, observational case series and interobserver variability study. Methods Three hundred eighty-six eyes of 233 consecutive subjects were imaged with the Discam, and a subset underwent ONH evaluation with CSLO (n = 297), stereoscopic photography (n = 233), or both. Subjects were classified into normal, glaucoma suspect, and glaucoma groups based on clinical findings of slit-lamp disc examination and visual field testing. Agreement of cup-to-disc ratio measurements among the three technologies was assessed by the intraclass correlation coefficient (ICC) and areas under the receiver operator characteristic (AROC) curves. Main outcome measures Cup-to-disc ratio as measured by Discam, CSLO, and stereography. Results Intraclass correlation coefficients calculated using the two-way random effects model for comparing Discam, CSLO, and stereography among cup-to-disc area ratio, vertical cup-to-disc ratio, and horizontal cup-to-disc ratio were 0.46 to 0.53. The ICC was higher for eyes with larger optic discs (0.51) than those with smaller ones (0.32). The ICC calculated using the fixed effects model for the Discam and CSLO comparison was 0.72. Areas under the receiver operator characteristic curves were 0.67 to 0.80 among the three technologies comparing normal with glaucomatous eyes. In general, there was no statistically significant difference between techniques with respect to sensitivity and specificity of glaucoma detection. Conclusions There is good agreement between ONH measurements obtained by Discam, CSLO, and stereography; however, the results are not interchangeable. Similar AROC curve values among all three techniques imply that the Discam, CSLO, and stereography perform equally for the determination of glaucoma status.

Emergency Ultrasound Optic Nerve Sheath Measurement to Detect Increased Intracranial Pressure in Head Injury Patients: Preliminary Study of Interobserver Variability in Normal Human Subjects

Emergency Ultrasound Optic Nerve Sheath Measurement to Detect Increased Intracranial Pressure in Head Injury Patients: Preliminary Study of Interobserver Variability in Normal Human Subjects

Emergency ultrasound optic nerve sheath measurement to detect increased intracranial pressure in head injury patients: preliminary study of interobserver variability in normal human subjects

Objective: The potential use of ultrasound to rapidly detect intracranial hypertension in head trauma patients is a novel concept. It has been shown that patients with increased intracranial pressure have increased ONSD (optic nerve sheath diameter). As part of a multi-center study, optic nerve sheath diameter will be determined as a reflection of increased intracranial pressure confirmed by cranial CT or neurosurgical intervention. Prior study data agree that the normal ONSD is between 4.5 and 5.0 mm. We hypothesized that interobserver variability would not be significant between investigators in a series of normal patients.

Defining dysplasia in Barrett esophagus.

Histologic grading of esophageal biopsies remains the cornerstone of management in patients with Barrett esophagus. This remains true despite ongoing research into the genetic abnormalities that occur in the setting of intestinal metaplasia, dysplasia, and adenocarcinoma of the esophagus. As of this writing, there are no objective tests that can replace the pathologist's eyes in predicting prognosis in this setting. However, pathologists and clinicians are well aware that the histologic interpretation of biopsies is an inherently subjective practice. The validity of this exercise depends entirely on the experience of the pathologist. Having said that, two interobserver studies on the grading of dysplasia in Barrett esophagus suggest that, if published criteria are adhered to, the histologic grading of dysplasia is a useful tool in patient management. In this review, the updated pathologic criteria for each category of dysplasia are presented, and the results of two large interobserver variability studies are reviewed.

Reproducibility of evaluation of optic disc change for glaucoma with stereo optic disc photographs

Purpose To determine the reproducibility of the assessment for glaucomatous change in serial optic disc stereo-slides. Design Masked interobserver variability study. Participants Serial optic disc stereo-slides from 40 patients. Methods Three independent ophthalmologists evaluated for change a set of two serial 20° optic disc color stereo-slides of 40 patients. This test set was not from European Glaucoma Prevention Study (EGPS) patients. Each observer performed two evaluations at least 30 days apart and was masked from the temporal sequence of the slides and his or her previous evaluation. Each patient was graded as changed or stable by two-out-of-three agreement. A κ statistic was used to calculate the intra- and interobserver reproducibility as well as the assignment reproducibility (first consensus versus second consensus). The same procedure was followed to test the reproducibility when another experienced ophthalmologist was added to one of the three reading centers. Main outcome measures Reproducibility in evaluating glaucomatous optic disc change. Results The intraobserver reproducibility (95% confidence interval [CI]) in the evaluation of change ranged between 0.79 (0.45–1.14) and 1.00 (0.69–1.31). The interobserver reproducibility (95% CI) in the evaluation of change ranged between 0.45 (0.15–0.75) and 0.75 (0.44–1.06). The assignment reproducibility (first consensus versus second consensus in the evaluation of change) between the senior EGPS readers was 0.94 (0.63–1.25). The assignment reproducibility when another experienced ophthalmologist replaced one of the readers was 0.94 (0.63–1.25). Conclusions The assignment reproducibility of three expert readers looking for glaucomatous change in serial optic disc stereo-slides was excellent. It remained so when one of the three experts was replaced by another experienced reader.

Duplex ultrasound as first-line screening test for patients suspected of renal artery stenosis: prospective evaluation in high-risk group.

Our objective was to assess how far the progress in ultrasound devices has increased feasibility and accuracy of Duplex ultrasound (DUS) for the diagnosis of renal artery stenosis (RAS), in a population with high prevalence of atherosclerotic renovascular lesions. Ninety-one hypertensive patients with atherosclerotic disease were prospectively evaluated by both DUS and digital subtraction angiography (DSA) of the renal arteries. Only proximal criteria (peak systolic velocity >180 mm/s or renal-to-aortic ratio >3.5) were used for the diagnosis of significant RAS (>60% narrowing). For both techniques, two readers were involved for interobserver variability study. Two hundred one arteries were demonstrated by DSA on 182 available kidneys. The prevalence of RAS among the study group was 37%. Sixteen of the 19 accessory arteries were not seen at DUS; in 8 patients, one renal artery was not seen at DUS (feasibility 91%). On the 177 arteries assessed, in comparison with DSA, DUS yielded 96, 91, and 97% mean values of accuracy, sensitivity, and specificity, respectively. Kappa for interobserver agreement was 0.95 for DUS and 0.92 for DSA. Although still unreliable for the detection of accessory arteries, DUS is in our experience an accurate and reproducible diagnostic test for RAS.

Computed tomography determination of prostate volume and maximum dimensions: a study of interobserver variability

Objective: (1) To evaluate the reproducibility of prostate volume, maximum dimensions and geometrical center coordinates determination using computed tomography (CT) and (2) to identify patterns of interobserver variability. Materials and methods: Forty patients, suitable for our brachytherapy program, were selected for the study. All patients underwent CT scanning and the prostate volumes were determined by three radiation oncologists. Measurements of geometrical center coordinates, maximum organ dimensions in the anterior–posterior (AP), lateral (Lat) and longitudinal (Long) axes as well as prostate volumes were recorded. This yielded 840 measurements of seven variables for analysis. The means and corresponding standard deviations (SD) of each variable were calculated for each patient. The SDs were then averaged and presented as indices of dispersion. Average variations from the mean were also calculated for each observer along with the SDs. Results: Analysis of the geometrical center coordinates revealed acceptable variability amongst observers. For the AP, Lat and Long coordinates the SDs were 0.78, 0.89 and 1.72 mm, respectively. The corresponding values for the maximum organ dimensions were 2.54, 2.72 and 4.43 mm, respectively. While the volumes outlined by observer B were less than or equal to the mean in 95% of cases and those of observer C were greater than or equal to the mean in 93% of cases, the volumes of observer A were equally distributed above and below the mean (48% in both cases). Conclusion: The determination of the geometrical center coordinates was reproducible amongst observers. The largest variations were seen with the Long axis. The volume determination is more variable. However, a characteristic trend was seen amongst observers when their volumes were compared to the mean volumes of the group.

Nasopharyngeal carcinoma volume measurements determined with computed tomography: study of intraobserver and interobserver variability.

To investigate the intraobserver and interobserver variability of computed tomography-based volume measurements of nasopharyngeal carcinoma. Prospective study. Tertiary care centre. The primary tumour volume of 13 nasopharyngeal carcinomas was repeatedly measured by two trained observers independently in two different sessions, using the summation of area technique. Mean tumour volume and its standard deviation were calculated for each tumour. Statistical analysis was done with multivariate analysis, linear regression, and a two-way analysis of variance (ANOVA) random effects model. The coefficient of variation was less than 20% in 11 volume measurements, but a large discrepancy between observers was noted in two tumours with involvement of the paranasal sinuses. A good linear correlation was found between mean tumour volume and its standard deviation: standard deviation = 0.26 volume - 2.48 (r = .80). When the two tumours with a large coefficient of variation were excluded, the two-way ANOVA random effects model revealed that both the interobserver (p = .83) and the intraobserver (p = .90) effect are not statistically significant; interobserver variability was the major component of total variability (71.0%). Total variability in the computed tomography-based measurement of nasopharyngeal carcinoma volume is small by having the measurements done by a trained observer, except in tumours with involvement of the paranasal sinuses.

Lower eyelid medial canthal tendon laxity grading: an interobserver study of normal subjects

Objective To describe a simple grading system for medial canthal tendon (MCT) laxity and measure its reproducibility. Study design Observational case series and interobserver variability study. Participants Fifty subjects (100 lower eyelids) without eyelid disease. The age range was 19 to 98 years. Methods Subjects without eyelid pathology or previous surgery were selected. Two ophthalmologists assessed the position of the inferior punctum in relation to the cornea with the patient in primary gaze. Two measurements were made, the first with the lower eyelid at rest and then with the lateral distraction test. Both observers were masked to the other observer’s measurements. Main outcome measures Kappa statistics to show the strength of agreement between the two observers for the resting and lateral distraction eyelid positions. Results High kappa values were found for resting position (agreement in 92 of 100 eyelids) and for lateral distraction testing (agreement in 85 of 100 eyelids). The range of resting positions was from position –1 to position 1. The range of laterally distracted positions was from position 0 to position 5. Overall, a very high level of agreement was achieved between observers. Conclusions This grading system for MCT laxity is simple, easy to learn and reproducible, easier than measuring in millimeters, and has an advantage over merely saying the tendon is “lax” or “not lax.” We propose this standardized grading system in the evaluation of patients with entropion and ectropion to identify those patients who may benefit from MCT stabilization/plication.

Dysplasia as a predictive marker for invasive carcinoma in Barrett esophagus: A follow-up study based on 138 cases from a diagnostic variability study

The objective of endoscopic surveillance in Barrett esophagus (BE) is to assess the risk of subsequent development of invasive carcinoma. Criteria for morphologic evaluation of dysplasia, the presumed precursor lesion, have been established, although there are surprisingly few data in the literature correlating biopsy diagnosis of dysplasia with outcome. We collected follow-up information on 138 patients with BE whose initial endoscopic biopsy specimens had been selected for submission in an interobserver variability study performed by 12 pathologists with special interest in gastrointestinal pathology and reviewed blindly twice each by all the participants. Cases were scored as BE with no dysplasia, atypia indefinite for dysplasia (IND), low-grade dysplasia (LGD), high-grade dysplasia (HGD), intramucosal carcinoma, and frankly invasive carcinoma, thus generating 24 scores on each biopsy specimen. Clinical follow-up was obtained and correlated with both the submitting diagnoses and majority diagnoses. Kaplan-Meier statistics were used to compare both the submitting and majority diagnoses with outcome using detection or documentation of invasive carcinoma as the endpoint. Using the submitting diagnoses, no invasive carcinomas were detected in 44 cases diagnosed as BE (median follow-up, 38.5 months). Carcinomas were detected in 4 of 22 (18%) cases submitted as IND (median progression-free survival of 62 months), in 4 of 25 (15%) cases of LGD (median progression-free survival of 60 months), in 20 of 33 cases of HGD (median progression-free survival, 8 months), and all 13 (100%) cases submitted as adenocarcinoma. Grade on initial biopsy correlated significantly with progression to invasive carcinoma (log-rank P =.0001). Majority diagnosis was achieved in 99 of the cases. Using the majority diagnoses, no invasive carcinomas were found in 50 cases of BE (median follow-up, 48 months), and carcinomas were detected in 1 of 7 (14%) IND cases (80% progression-free survival at 2 months), 3 of 15 (20%) LGD (median progression-free survival, 60 months), 9 of 15 (60%) HGD (median progression-free survival, 7 months), and all 12 (100%) carcinoma. Initial grading again correlated significantly with progression to invasive carcinoma (log-rank P =.0001). However, there were 39 cases without a majority diagnosis. Among these, no carcinomas developed in 8 cases with an average score between BE and IND. Carcinomas were detected in 9 of 21 (43%) cases with an average score between IND and LGD, and 7 of 10 (70%) cases with an average score between LGD and HGD. There were ulcers in 8 of 39 cases (20%) of the “no-majority” group and in 13 of 99 (13%) of the majority cases. Of 21 total ulcerated cases, cancer was demonstrated in 15 (71%) of these on follow-up. These data support combining the IND and LGD categories for surveillance purposes. Cases without dysplasia may be followed up conservatively. The data obtained from submitted diagnoses as opposed to those from blind review suggest that knowledge of the clinical findings aids in diagnosis. The data also support the assertion that HGD is strongly associated with invasive carcinoma. Rebiopsy of ulcerated areas should be considered because they may harbor malignancy. Histologic grading of dysplasia using established criteria is a powerful prognosticator in BE. HUM PATHOL 32:379-388. Copyright © 2001 by W.B. Saunders Company

Computed tomography (CT) determination of prostate volume and maximum dimensions: A study of interobserver variability

Computed tomography (CT) determination of prostate volume and maximum dimensions: A study of interobserver variability