Interobserver Variation In Assessment Research Articles

Variability in Synoptic Reporting of Colorectal Cancer pT4a Category and Lymphovascular Invasion.

Serosal involvement (pT4a category) and lymphovascular invasion have prognostic significance in colorectal carcinoma, but are subject to interobserver variation in assessment. To provide the first large-scale assessment of interobserver variability in pT4a category and lymphovascular invasion reporting in real-world practice and to explore the impact of information from guidelines on variability in reporting these features. Analysis of 1555 consecutive synoptic reports of colorectal carcinoma was performed using multivariate logistic regression. Interobserver variability before and after the presentation of guideline information was assessed using an image-based survey. Significant differences in the odds of reporting pT4a versus pT3 category, detecting lymphovascular invasion of any type, and detecting large vessel invasion were identified among hospital sites and for individual pathologists compared with the median pathologist at the same site. Consistent with these results, interobserver agreement was only moderate in the image-based survey regarding T4a staging and lymphovascular invasion (all κ ≤ 0.57). The provision of information from guidelines did not tend to increase interobserver agreement in the survey, though responses in favor of using an elastic stain increased following recommendations for their use. However, when observers were provided with elastic-stained images, interobserver agreement remained only moderate (κ = 0.55). Real-world reporting of pT4a category and lymphovascular invasion shows substantial variability at both local and regional levels. Our study underscores the need to address these features in quality initiatives, and provides a novel method through which existing synoptic data can be harnessed to monitor reporting patterns and provide individualized feedback.

Repeated biopsies in patients with prostate cancer on active surveillance: clinical implications of interobserver variation in histopathological assessment.

To investigate the clinical implications of interobserver variation in the assessment of re-biopsies obtained during active surveillance (AS) of prostate cancer. In all, 107 patients with low-risk prostate cancer with 93 diagnostic biopsy sets and 109 re-biopsy sets were included. The International Society of Urological Pathology 2005 Gleason scoring system was used for the histopathological assessment of all biopsies. Three different definitions of histopathological progression were applied. Unweighted and linear weighted Kappa (κ) statistics were used to compare the interobserver agreement. The overall Gleason score agreement was 68.8% with a weighted κ of 0.670. The interobserver agreement was 79.6% for meeting the AS selection criteria. According to the three progression definitions applied, overall agreement was between 80.7% and 89.0% with weighted κ values of 0.746-0.791. Treatment recommendations would have changed in up to 10.1% (95% confidence interval 5.4-17.7%) of the 109 re-biopsy sets. Kappa statistics showed strong agreement between the histological evaluations. However, up to 10% of patients on AS would receive a different treatment recommendation depending upon which histopathological evaluation of re-biopsies was used for treatment planning.

How many patients fulfil the surface electrocardiogram criteria for subcutaneous implantable cardioverter-defibrillator implantation?

To determine the number of patients with a primary or secondary prevention implantable cardioverter-defibrillator (ICD) indication who are eligible for subcutaneous ICD (S-ICD) implantation according to the S-ICD manufacturer's surface electrocardiogram (ECG) screening template. One hundred and ninety-six ICD patients with a non-paced ventricle were assessed using erect and supine ECG limb lead recordings to simulate the three S-ICD sensing vectors. Each ECG lead was scrutinized by two independent observers. Subcutaneous ICD eligibility required two or more leads to satisfy the S-ICD screening template in both erect and supine positions. Overall, 85.2% of patients [95% confidence interval (CI): 80.2-90.2%] fulfilled surface ECG screening criteria. The proportion of patients with 3, 2, 1, and 0 qualifying leads were 37.2% (95% CI: 30.4-44.0%), 48.0% (95% CI: 41.0-55.0%), 11.2% (95% CI: 6.8-15.6%), and 3.6% (95% CI: 1.0-6.2%). The S-ICD screening template was satisfied more often by Lead III (primary vector, 83.7%, 95% CI: 78.5-88.9%) and Lead II (secondary vector, 82.7%, 95% CI: 77.4-88.0%) compared with Lead I (alternate vector, 52.6%, 95% CI: 45.6-59.6%). A prolonged QRS duration was the only baseline characteristic independently associated with ineligibility for S-ICD implantation. There was 92.9% agreement between the two independent observers in assessment of eligibility using the S-ICD screening template. About 85.2% of patients with an indication for a primary or secondary prevention ICD have a surface ECG that is suitable for S-ICD implantation when assessed with an S-ICD screening template. There is minor inter-observer variation in assessment of eligibility using the S-ICD screening template.

Interobserver variation in assessment of gastric premalignant lesions

Either atrophy or intestinal metaplasia of the gastric mucosa are considered premalignant lesions. The new operative link for gastritis assessment staging system is based on the detection of atrophy, and the operative link for assessment of intestinal metaplasia staging system is based on the detection of intestinal metaplasia. Good interobserver agreement is necessary for identification of any premalignant condition. The aim of this study was to compare the agreement between findings of gastric atrophy and intestinal metaplasia by expert and general pathologists and to analyze the possible reasons behind any possible disagreement. Patients with dyspeptic symptoms, aged 55 years and above, without previous Helicobacter pylori eradication were enrolled and analyzed according to the updated Sydney Classification by two expert pathologists and an experienced general pathologist; the results were compared with the consensus driven by the two experts. Gastric biopsy specimens from 121 patients (91 women) were included in the analysis; the mean age of the patients was 67.4 years. H. pylori infection was present in 61.2% of patients. The level of agreement between the general pathologist and the two experts (κ-value) was 0.12, 0.46, and 0.87, respectively, for detecting atrophy in the corpus; 0.77, 0.77, and 0.65, respectively, for detecting intestinal metaplasia in the corpus; 0.06, 0.51, and 0.54, respectively, for detecting atrophy in the antrum; and 0.69, 0.85, and 0.79, respectively, for detecting metaplasia in the antrum. The agreement was substantially higher for intestinal metaplasia than for atrophy. This could result in discrepancies when the operative link for gastritis assessment and operative link for assessment of intestinal metaplasia staging systems are applied and can be caused by differences in the criteria used to define atrophy.

Accuracy of Cryptorchidism Diagnoses and Corrective Surgical Treatment Registration in the Danish National Patient Registry

In recent years several Danish studies of the etiology, time trends and long-term health consequences of cryptorchidism have relied on diagnoses and surgical treatments registered in the National Patient Registry. We evaluated the diagnostic accuracy of these registry data. According to the Danish National Patient Registry, 16,168 males were diagnosed with cryptorchidism and 9,244 surgical treatments for cryptorchidism were performed between January 1, 1995 and October 10, 2009. We randomly selected 500 diagnosed cases, of which 284 had been managed surgically. We requested the medical records from the departments making the diagnoses and performing the surgery. We successfully retrieved medical records for 452 diagnosed cases (90%) and 249 operations (88%). Overall positive predictive value of a registry diagnosis of cryptorchidism was 80% (95% CI 77-84) using the testicular position described by the physician performing the clinical examination as the gold standard. Similarly the positive predictive value of the surgical treatment registration was 99% (95% CI 98-100) using the type of procedure performed. The data on cryptorchidism in the Danish National Patient Registry are quite accurate. In etiological research the limited misclassification will in most cases only slightly attenuate estimates of the true relative association. Thus, the registry has the potential to serve as a valuable research tool, although caution should be exercised when studying time trends or geographical differences.

1048 Which Activity Index for Ulcerative Colitis (UC)? Evaluation of Inter-Observer Variation in Clinical, Endoscopic and Composite Indices

1048 Which Activity Index for Ulcerative Colitis (UC)? Evaluation of Inter-Observer Variation in Clinical, Endoscopic and Composite Indices

Tumor cell type can be reproducibly diagnosed and is of independent prognostic significance in patients with maximally debulked ovarian carcinoma

Ovarian surface epithelial carcinomas are routinely subclassified by pathologists based on tumor cell type and grade. It is controversial whether cell type or grade is superior in predicting patient response to treatment or survival, in patients stratified by stage of disease. The aim of this study was to uniformly apply updated criteria for cell-type and grade assignment to a series of 575 cases of ovarian surface epithelial carcinoma. All patients were optimally surgically debulked, with no macroscopic residual disease after primary surgery. Slides from these cases were reviewed by a single pathologist, who was blinded to patient outcomes. In 50 cases, 2 additional pathologists reviewed the slides independently to determine interobserver variation in assessment of cell type and grade. The distribution of tumor stage was as follows: stage I--233 cases, stage II--246 cases, stage III--96 cases. The most common cell type encountered was serous carcinoma (229/575, 40%), followed by clear cell (149/575, 26%), endometrioid (139/575, 24%), and mucinous (36/575, 6%). Serous carcinomas were significantly more likely to present with advanced stage disease (76/229 [33.2%] were stage III, and 82% of all stage III tumors were serous), whereas all nonserous cell types were stage I or II at diagnosis in greater than 90% of cases. Both FIGO grade and Silverberg grade stratified patients into groups with significantly different risks of relapse and survival, but the Silverberg grading system was a more powerful prognosticator. In multivariate analysis, stage was the most powerful prognostic indicator (P < .0001), followed by tumor cell type (P = .015), but grade was not of independent significance. Interobserver variation in assignment of cell type was very good (kappa = 0.77) with moderate reproducibility in assignment of Silverberg grade (kappa = 0.40) and minimal reproducibility in assignment of FIGO grade (kappa = 0.27). Thus, in this series of cases of ovarian surface epithelial carcinomas with no macroscopic residual disease after primary debulking surgery, assignment of tumor cell type was both more reproducible and provided superior prognostic information compared with assignment of tumor grade. As tumor cell type also correlates with underlying molecular abnormalities and may predict response to chemotherapy, this suggests that tumor cell type could be used to guide treatment decisions for patients with ovarian surface epithelial carcinoma.

Ki67 and p53 immunohistochemistry reduces interobserver variation in assessment of Barrett's oesophagus

To devise clinically applicable methods for assessing p53 and Ki67 immunohistochemical (IHC) reactivity in Barrett's oesophagus (BE) and to compare the interobserver agreement between these methods and routine haematoxylin and eosin (H&E) evaluation. One hundred and fifteen biopsies diagnosed as BE, selected from the files of the University Hospital MAS, Malmo, were re-evaluated for dysplasia by three pathologists. For IHC analysis areas with the most prominent positivity were evaluated. The mean of p53+ epithelial nuclei/high-power field (HPF) was obtained by counting between 1 and 5 HPFs/biopsy. A proliferation quotient (PQ) was obtained by dividing the number of Ki67+ epithelial nuclei in the upper half by the lower half of the mucosa, using two HPFs. Mean kappa values were 0.24, 0.71 and 0.52 for H&E, p53 and Ki67 evaluations, respectively. There was a correlation between increasing severity of dysplasia, IHC measurable overexpression of p53 and shift of the mucosal proliferation zone towards the surface, measured as PQ. The described methods for p53 and Ki67 evaluation are more reproducible than routine H&E evaluation of BE. Furthermore, the IHC methods correlate with the severity of dysplasia and are useful supplementary prognostic markers.

Inter-observer Variation in Assessment of 845 Labor Admission Tests: Comparison Between Midwives and Obstetricians in the Clinical Setting and Two Experts

The labor admission tests entails 20-30 minutes of cardiotocographic (CTG) recording on entry to the maternity ward. Because observer variability reflects the reliability of any clinical test, this study was done to determine the degree of agreement between midwives and obstetricians in evaluating labor admission tests. Two expert consultant obstetricians from other centers also participated. The study group included 845 high- and low-risk women who were admitted in labor and gave birth within 24 hours of the test. Test findings were rated as reactive, equivocal, or ominous. The kappa (κ) value is the difference between observed and expected agreement, or the proportion of potential agreement beyond change. Values exceeding 0.75 reflect excellent agreement, and values between 0.4 and 0.75 reflect fair to good agreement. Weighted kappa (κW) was used in this analysis. The proportion of agreement (Pa) also was calculated. Agreement between the two experts was κw 0.38; Pa for a reactive labor admission test, 0.86; and Pa for an equivocal or ominous test, 0.33. The respective values for agreement between one expert and midwives/obstetricians were 0.25, 0.89, and 0.18. For the other expert, the respective figures were 0.28, 0.85, and 0.20. Nearly 6% of newborn infants had fetal distress. At a cutoff of an equivocal test, sensitivity values for the 2 expert observers were 0.22 and 0.31. Positive predictive values were 0.13 and 0.11. The likelihood ratios for a positive test were 2.3 and 1.92, and for a negative test, 0.86 and 0.83. Despite little evidence that it is beneficial, labor admission testing remains routine practice in most obstetrical units in the Western world. The present findings should prompt reconsideration of this practice, and they suggest the relevance of educational programs for maternity ward staff so that they will more accurately interpret CTG recordings.

Inter-observer variation in assessment of 845 labour admission tests: comparison between midwives and obstetricians in the clinical setting and two experts

Objective To assess the inter-observer agreement in assessment of the labour admission test between midwives and obstetricians in the clinical setting and two experts in the non-clinical setting, the inter-observer agreement between two experts in the non-clinical setting and to what degree fetal distress in labour could be predicted by the two experts. Design Observational study. Setting The maternity unit of Hammerfest Hospital, Norway. Population Eight hundred and forty-five high and low risk women. Method The labour admission test was first assessed by the midwife or obstetrician in the clinical setting, and was later assessed by two experts. The traces were assessed as normal, equivocal or ominous. Weighted kappa (κw), proportion of agreement (Pa) and predictive values were calculated. Main outcome measures Weighted kappa, proportion of agreement, sensitivity, positive predictive value and likelihood ratios. Results Inter-observer agreement between Expert 1 and Expert 2: κw 0.38 (CI 0.31–0.46), Pa for reactive labour admission test 0.86 (CI 0.83–0.88) and Pa for equivocal/ominous test 0.33 (CI 0.26–0.40). Agreement between Expert 1 and midwives/obstetricians: κw 0.25 (CI 0.15–0.36), Pa for reactive labour admission test 0.89 (CI 0.87–0.91) and Pa for equivocal/ominous labour admission test 0.18 (CI 0.11–0.25). Agreement between Expert 2 and midwives/obstetricians: κw 0.28 (CI 0.20–0.37), Pa for reactive labour admission test 0.85 (CI 0.82–0.88) and Pa for equivocal/ominous test 0.20 (CI 0.14–0.26). Totally 5.9% of the newborns had fetal distress. At cutoff equivocal test, sensitivity was 0.22 and 0.31 in the two observers. Positive predictive values were 0.13 and 0.11. Likelihood ratio for a positive test was 2.30 and 1.92 and likelihood ratio for a negative test 0.86 and 0.83. Conclusion A labour admission test is still routine practice in most obstetric units in the Western world when there is little evidence on its benefits. The results from this study may provide some reconsideration for such practice, and for more research.

Spiral CT of pulmonary nodules: interobserver variation in assessment of lesion size.

The aim of this study was to assess interobserver variability in size determination of pulmonary nodules at spiral CT. Twenty-three patients with known pulmonary nodules (diameter 2-40 mm, mean diameter 7 mm) underwent spiral chest CT (collimation 5 mm, pitch 1). Images were reconstructed at 3- and 5-mm intervals (RI). Hard copies were analyzed by two radiologists who recorded every nodule with regard to location, diagnostic confidence ("definite," "probable") and nodule size in increments of 1 mm with specific attention to correct classification into one of three size classes (< or = 5 mm, 6-10 mm, > 10 mm). Interobserver variability was determined with Pearson's correlation coefficient and kappa measure. Of a total of 286 nodules, 103 nodules were found accordingly by both readers at 3 mm RI, and 96 at 5 mm RI. There was a good correlation of measurements (in millimeters) between both readers (Pearson's correlation coefficient: 0.89-0.95). Interobserver variability in categories was good at both reconstruction intervals (kappa: 0.61 at 3 mm, 0.74 at 5 mm RI) and very good (0.81) at 5 mm RI when uncertain nodules were excluded. Spiral CT allows reproducible size determination of pulmonary nodules as shown by good interobserver agreement in exact size measurement and categorization into three size classes.

Interobserver variation in cell selection for DNA image cytometry.

To describe a systematic investigation of interobserver differences in interpretation of nuclear morphology in preparations of small cell lung cancer (SCLC). The screening/reviewing facility on the highly optimised microscope environment was used to individually tag 127 nuclei, chosen to reflect the spectrum of morphological appearances in nuclear preparations from three biopsy specimens of SCLC. Each nucleus was reviewed and labelled as control (lymphocyte), malignant or unsatisfactory by each of four observers. DNA histograms were plotted for each specimen using the nuclei identified as malignant by each participant. The histograms were compared in terms of identification of DNA stemlines and by calculation of a 5c exceeding rate (5cER). Interobserver variation in assessment of morphology was seen in 55.1% of nuclei. Disagreement occurred most frequently in the malignant/unsatisfactory category. Differences in morphological classification had little influence on histogram assessment by means of visual inspection but did show an effect on 5cER. There are significant interobserver differences in subjective assessment of nuclear morphology in cytometric preparations. This effect may seriously influence cytometric measurements.

Interobserver variation in assessment of gastroduodenal lesions associated with non-steroidal anti-inflammatory drugs.

Video endoscopic images were used to investigate whether gastroenterologists could agree on the definition of lesions within the stomach seen at endoscopy, with particular reference to those seen in patients taking non-steroidal anti-inflammatory drugs. Seven experienced endoscopists, unaware of the patients' clinical history or drug consumption, recorded their classification for 93 randomised video images of gastric lesions. There was complete agreement in the diagnosis of ulceration for nine images from patients who were not taking non-steroidal anti-inflammatory drugs; eight of nine were classified as deep ulcers, with 86% agreement for this subclassification. By contrast, the overall agreement for lesions in patients taking non-steroidal anti-inflammatory drugs was only 55%. Only nine of 44 ulcers were subclassified as deep, and there was considerable cross classification of non-haemorrhagic erosions and ulcers. In conclusion, ulcers that occur in patients taking non-steroidal anti-inflammatory drugs differ from those in patients who are not taking these drugs in that they are often more superficial and difficult to distinguish from erosions. The prognostic importance of these lesions is, therefore, uncertain.

Proteinuria and its assessment in normal and hypertensive pregnancy

The purposes of this study were (1) to determine 24-hour urinary protein excretion rates in normal human pregnancy and (2) to assess the reliability of assessment of proteinuria by dipstick measurement. At 17 to 20 and 33 to 36 weeks of pregnancy, 174 normal volunteers collected a 24-hour urine sample; the volume and the protein and creatinine concentrations were measured. The result for protein was compared with dipstick analysis of an early morning midstream urine sample collected at the conclusion of the 24-hour period. Sixty-eight consecutive inpatients admitted to the antenatal ward with hypertension and positive urine dipstick tests for protein underwent the same procedure. The interobserver variability in dipstick analyses of urine samples of known protein content was assessed with the aid of 66 volunteers from the hospital staff. The upper 95% confidence limit of the normal population was less than 200 mg per 24 hours, at both stages of pregnancy investigated. In these women and in the hypertensive inpatients a high proportion of false-positive and false-negative results was found with dipstick analyses. Interobserver variation in assessment of proteinuria by dipstick was high, with an 18% false-positive rate and a false-negative rate approaching 40% for samples with 30 mg/dl. Even in the presence of 100 mg/dl the false-negative rate was 7%, whereas the concentration of protein was significantly underestimated in 20% of samples with 500 mg/dl. Dipstick urinalysis cannot be relied on either to detect or to exclude the presence of proteinuria in pregnant women.

Interobserver variation in assessment of respiratory signs. Physicians' guesses as to interobserver variation.

Interobserver variation was studied for 11 respiratory signs and for the diagnostic conclusions made on the basis of these signs. Two physicians each, with and without knowledge of the patient's history, performed physical examination of the lungs in a total of 202 unselected patients from a department of internal medicine. The interobserver variation was calculated as the kappa coefficient, using a modified method that made it possible to calculate the interobserver variation for a pair of observers made up by varying physicians. No difference was demonstrated between the kappa coefficients obtained for examinations performed with and without knowledge of the history. The kappa values for the signs ranged from -0.04 to 0.58, and for the diagnoses from 0.15 to 0.68, i.e., expressed by kappa values, poor agreement. In a questionnaire study we asked a number of physicians from departments of internal medicine to guess the size of the interobserver variation for the 11 respiratory signs examined. A total of 47 physicians (90%) returned the questionnaire and the answers showed that the physicians markedly underestimated the interobserver variation.