Observer Disagreement Research Articles

Relative Precision and Sources of Bias in Pronghorn Sex and Age Composition Surveys

We compared pronghorn (Antilocapra americana) composition survey estimates among observers and examined factors that may influence classifications at 4 sites in Wyoming. We also compared fixed-wing aerial to ground survey estimates. Observer disagreement was related to size of pronghorn group (P < 0.01) during ground surveys. Fixed-wing aerial surveys yielded significantly lower adult buck ratios than ground surveys in 3 of 6 (P < 0.05) comparisons. Observers' classifications differed in 3 of 12 ground, and 5 of 12 aerial comparisons (P < 0.05) with half the disagreements being fawn ratios. Neither aerial nor ground surveys was consistently more precise. Precision of individual observers was similar at most sites within methods.

Measuring Eye Movements in Graves?? Ophthalmopathy

Background: The current grading of class IV (extraocular muscle involvement) of the NO SPECS classification for Graves ophthalmopathy is subjective and, in practice, virtually limited to three grades (grade c rarely occurs). The subjectiveness results in a high degree of observer disagreement, as illustrated by kappa values of 0.48 to 0.64 for the agreement between the authors in grading of class IV changes in 13 patients. Methods: The authors investigated the significance of a quantitative approach in assessing eye motility and modified a hand perimeter to measure the monocular eye movements in degrees. Results: The mean maximal excursions in the four main directions of gaze, obtained with this device in 40 healthy control subjects (5 men and 5 women from each decade between 20 and 60 years of age), were as follows: elevation=34°, depression=58°, adduction=48°, and abduction=46°. The ductions showed an age-related decline. The intraobserver and interobserver variations were assessed in 12 and 18 patients with Graves ophthalmopathy, respectively, by calculating the coefficients of repeatability, which varied from 5° to 8°. The authors compared the current grading of NO SPECS class IV with quantitative measurements of elevation in 11 patients by calculating the kappa values for agreement. These were found to be 0.57 and 0.80, respectively. Conclusions: The quantitative measurement of eye movements with a modified perimeter appears to be precise and reproducible. In addition, it should replace the current grading of the extraocular muscle movements of Graves ophthalmopathy patients in the NO SPECS classification, which is less reliable, as evidenced by a higher degree of observer disagreement.

Information and observer disagreement in histopathology.

Concepts from information theory are used to assess the process of ordinal classification of histopathological continua such as degree of dysplasia or degree of differentiation. A working model is used to show that reducing the number of categories leads to an increase in inter-observer agreement, as measured by kappa statistics, but a decrease in information transmitted. It is argued that the prime role of histopathologists is to pass information, obtained from visual patterns, to the clinician and therefore more categories, rather than fewer, should be used. It is suggested that judgements of continua should be made on 100 point scales and quoted together with 95% confidence limits; and that the judgements of different observers should be averaged to form a reference range.

Measuring Eye Movements in Graves Ophthalmopathy

The current grading of class IV (extraocular muscle involvement) of the NO SPECS classification for Graves ophthalmopathy is subjective and, in practice, virtually limited to three grades (grade c rarely occurs). The subjectiveness results in a high degree of observer disagreement, as illustrated by kappa values of 0.48 to 0.64 for the agreement between the authors in grading of class IV changes in 13 patients. The authors investigated the significance of a quantitative approach in assessing eye motility and modified a hand perimeter to measure the monocular eye movements in degrees. The mean maximal excursions in the four main directions of gaze, obtained with this device in 40 healthy control subjects (5 men and 5 women from each decade between 20 and 60 years of age), were as follows: elevation = 34 degrees, depression = 58 degrees, adduction = 48 degrees, and abduction = 46 degrees. The ductions showed an age-related decline. The intraobserver and interobserver variations were assessed in 12 and 18 patients with Graves ophthalmopathy, respectively, by calculating the coefficients of repeatability, which varied from 5 degrees to 8 degrees. The authors compared the current grading of NO SPECS class IV with quantitative measurements of elevation in 11 patients by calculating the kappa values for agreement. These were found to be 0.57 and 0.80, respectively. The quantitative measurement of eye movements with a modified perimeter appears to be precise and reproducible. In addition, it should replace the current grading of the extraocular muscle movements of Graves ophthalmopathy patients in the NO SPECS classification, which is less reliable, as evidenced by a higher degree of observer disagreement.

Observer Disagreement in Histological Classification of Ovarian Tumors in Japan

Histological slides of 789 cases of ovarian tumor registered by the Japanese Gynecologic Oncology Group, sponsored by the Ministry of Health and Welfare, Japan, were reviewed by three certified pathologists in order to evaluate the observer disagreement among the pathologists and the reproducibility of the histological classification. In 53.0% of the cases, all observers agreed upon the diagnosis. At least two of them agreed in 88.4% of the cases. The observer disagreement was highest for low potential malignancy (borderline malignancy) tumors (44.4%), including serous, mucinous, and endometrioid tumors. The results of the study clearly indicate the necessity of establishing a slide review system in order to obtain reliable data for comparative research. Moreover, the histological criteria, especially of tumors of borderline malignancy, should be more clearly defined to obtain high reproducibility of ovarian tumor diagnosis.

Follow‐up after periapical surgery: the value of the one‐year control

A total of 474 teeth treated with periapical surgery were examined after one year. Complete healing was unequivocally demonstrated for 250 teeth. Two hundred and fourteen teeth were scheduled for further follow-up, and of these, 194 (91%) could be followed until a stable situation was recorded. Of 41 cases judged as completely healed after one year, but with initial observer disagreement, only 2 (5%) failed later. The 76 cases showing incomplete healing (scars) after one year with few exceptions ended as completely healed or persisted as incomplete healings. The uncertain group (n = 72) was equally re-distributed as successes (completely or incompletely healed) or failures at the end control. Five cases showing no healing after one year all ended as failures. Based on these findings it is concluded that the one-year control will provide a valid diagnosis for the majority of cases. Only a minor number, the uncertain healings, need further follow-up.

Observer Disagreement in Rating Neurologic Impairment in Multiple Sclerosis: Facts and Consequences

The interobserver variability of the expanded disability status scale (EDSS) was studied in 59 patients with multiple sclerosis (MS). Interrater agreement was measured by the kappa coefficient. Agreement was low in patients with mild disability (EDSS less than 5); it was higher in patients with EDSS equal to or greater than 5. The difference between ratings of 2 independent examiners was equal to at least 1 point in 34% of the MS cases. This variability must be taken into consideration in designing clinical trials in MS.

Relation of observer agreement to accuracy according to a two-receiver signal detection model of diagnosis.

A long-standing problem of medical diagnosis is why rates of observer disagreement are so high. To study this question a model of test interpretation is defined which accounts for disagreement when observers use identical diagnostic strategies. Test subjects are assigned to one of two disease classes on the basis of two parameters: the test score and lambda, the diagnostic certainty, a parameter analogous to signal-to-noise ratio. The rate of disagreement and the mean error rate of independent ideal observers are derived as a function of diagnostic strategy. It follows from the model that: 1) Maximization of the ratio of disagreement to error defines an optimal diagnostic strategy with advantages over maximum likelihood and Bayes diagnostic strategies. 2) Disagreement exceeds error if the sensitivity and specificity of independent observers exceed one-half for all lambda. 3) The ratio of disagreement to error can be no greater than two if test interpretation is dichotomous but can approach infinity for trichotomous interpretation, i.e., if observers are allowed to interpret cases as indeterminate.

Differential diagnosis of jaundice: junior staff experience with the Copenhagen pocket chart.

Originally published in 1984, the Copenhagen Pocket Chart for early differentiation between causes of jaundice has been tested with success in centres outside Denmark. Using a logistic discrimination model, it estimates probabilities of obstruction and non-obstruction in each case (and provides a further subdivision if desired). Here we evaluate its performance in the hands of young clinicians on a consecutive series of 173 jaundiced patients from two Danish hospitals. The chart performed as well as in the original series: confident diagnoses (probability greater than or equal to 0.80) were assigned to 124 patients; of these 115 proved correct (93%). In 46 patients diagnostic probabilities were less than 0.80, and 3 patients had an unknown cause of jaundice. There were 108 cases in which physician and chart were in agreement, both with a confident diagnosis, and only one of these cases was wrong. In one hospital, contributing 107 cases, each patient was independently examined by a medical student in addition to the physician's examination. Student performance was equally good, practically speaking, in particular when taking the scores on the chart into consideration. As to observer disagreement, the student and the physician typically differed on 0-2 of the chart's 21 items. In no case, however, did this lead to a confident obstructive diagnosis being changed into a confident diagnosis of non-obstruction, or vice versa.

Reliability of selective pulmonary arteriography in the diagnosis of pulmonary embolism.

To test the reliability of conventional selective pulmonary arteriography in the diagnosis of pulmonary embolism, three angiographers reviewed the arteriograms of a series of 60 patients retrospectively, independently, and without benefit of additional data. Pulmonary arteriograms had been interpreted as positive for pulmonary embolism in 25 of these patients during their hospitalizations. Angiographers A, B, and C judged the arteriograms of 24, 29, and 25 patients, respectively, as positive for pulmonary embolism. Mean interobserver agreement was 86%. Interobserver agreement was not associated significantly with the quality of the arteriogram or with selective injection of a lobar vs a pulmonary artery, but was associated strongly with the magnitude of thromboembolism. All angiographers agreed that the arteriograms were positive in 18 cases of pulmonary embolism graded as massive, lobar, or segmental, but agreed in only two of 15 cases graded as subsegmental. We conclude that conventional selective pulmonary arteriography is reliable in the detection of embolus in segmental or larger pulmonary arteries. Observer disagreement becomes considerable for embolus limited to subsegmental pulmonary arteries, indicating that emboli of this size are at the resolution limit of the technique.

Digital subtraction angiography in coronary artery bypass graft assessment: clinical applicability.

Application of electrocardiogram gated digital subtraction angiography to the assessment of coronary artery bypass graft function was studied one week to eight years after bypass operation in ten unselected patients with recurrent chest pain. For the digital method, contrast was injected into the ascending aorta via a 4 or 5 French gauge catheter. The results of this technique were compared with those of selective graft and coronary angiography in the same patients by two independent observers. Of twenty six grafts in the series, patency was confirmed in twenty one by both selective and digital angiography. The quality of graft run off, graded by each observer using a simple scoring system, demonstrated six points of inter observer disagreement when standard cineangiograms were used, compared with nine points of disagreement when digital images were used. Digital subtraction angiography provided useful graft visualisation, but was less good than conventional angiography at defining the native coronary circulation. The role of this promising new technique has yet to be established.

Assessing Teacher Performance from Observed Competency Indicators Defined by Classroom Teachers

AbstractThis study was concerned with the measurement of a set of indicators of teacher competence, defined by the teachers themselves as observable in their classroom behavior. The question it sought to answer was whether scoring keys for existing low-inference observation schedules could be developed that would measure any or all of these indicators objectively and reliably. Multiple observations were made with four such instruments in 100 classrooms in a single rural school system to provide data relevant to the question. Forty-two scoring keys were developed to measure one or another of 26 indicators identified by the teachers and used to score the records made in the 100 classrooms. An analysis of variance was made of the scores on each key to estimate its reliability and to isolate and assess errors of measurement due to lack of internal consistency and to instability confounded with observer disagreement. It was concluded that keys could be constructed to yield stable and consistent measures of mos...

A GRAPHICAL JUDGMENTAL AID WHICH SUMMARIZES OBTAINED AND CHANCE RELIABILITY DATA AND HELPS ASSESS THE BELIEV ABILITY OF EXPERIMENTAL EFFECTS

Interval by interval reliability has been criticized for "inflating" observer agreement when target behavior rates are very low or very high. Scored interval reliability and its converse, unscored interval reliability, however, vary as target behavior rates vary when observer disagreement rates are constant. These problems, along with the existence of "chance" values of each reliability which also vary as a function of response rate, may cause researchers and consumers difficulty in interpreting observer agreement measures. Because each of these reliabilities essentially compares observer disagreements to a different base, it is suggested that the disagreement rate itself be the first measure of agreement examined, and its magnitude relative to occurrence and to nonoccurrence agreements then be considered. This is easily done via a graphic presentation of the disagreement range as a bandwidth around reported rates of target behavior. Such a graphic presentation summarizes all the information collected during reliability assessments and permits visual determination of each of the three reliabilities. In addition, graphing the "chance" disagreement range around the bandwidth permits easy determination of whether or not true observer agreement has likely been demonstrated. Finally, the limits of the disagreement bandwidth help assess the believability of claimed experimental effects: those leaving no overlap between disagreement ranges are probably believable, others are not.

Reliability and reproducibility of interpretation of 99m technetium pyrophosphate myocardial scintigrams

The interpretations of 156 99mtechnetium pyrophosphate myocardial scintigrams by four observers were analyzed in order to determine the reliability and reproducibility of the subjective process of reading scintigrams. The scintigrams were scored on an integral scale from 0 to 4, depending upon the degree of myocardial radionuclide accumulation, and the site and nature of uptake were specified. Exact agreement upon score was generally poor but approximate concurrence of interpretation was good (90.4 and 92.5% inter- and intra-observer agreement, respectively). There was somewhat less agreement on scintigrams with the higher scores of 3 and 4 (83.3 and 78.0%, respectively). A high level of concurrence upon the differentiation between diffuse and localized uptake, and upon the site of uptake, was found. We conclude that only approximate rather than exact agreement of individual readers' interpretations can be expected in this subjective technique, that scintigrams with higher degrees of radionuclide accumulation produce slightly greater observer disagreement, and that variability of interpretation could account for some of the diagnostic inaccuracy of 99mtechnetium pyrophosphate myocardial scintigraphy.

Observer variation in histologic classification of thyroid cancer.

Histologic slides of 696 cases of thyroid cancer reported to the national cancer registries of Finland, Iceland, Norway and Sweden were reviewed by 5 Nordic pathologists in order to determine the observer variation between the pathologists, as well as the reproducibility of the WHO classification of thyroid tumours. In 58% of the cases all observers agreed upon the diagnosis and in 82% at least three of them agreed. The observer disagreement was lowest for papillary carcinoma (7%) and highest for follicular carcinoma (27%). The corresponding figures for anaplastic and medullary carcinomas were 18% and 23%, respectively. The most common diverging diagnosis for cases finally interpreted as papillary carcinoma was follicular carcinoma, and for cases finally interpreted as follicular carcinoma, a benign thyroid lesion. The results of the present study clearly indicate the necessity of having all cases reviewed by the same pathologist or group of pathologists in order to obtain reliable results for comparative studies.

International Kveim histology trial.

An histological analysis of Kveim-test biopsies performed on a total of 306 patients including those with sarcoidosis, Crohn's disease, tuberculosis, and collagen disease is presented. Five main Kveim suspensions, "antigens," were used in the survey, Colindale K 12, Lot 5, Marburg 2 and 3, Commonwealth Serum Laboratory (CSL) early batches, 001/2/3 and late batches, 005/6. The biopsies were read blind by two observers (overall agreement 75%), the nature of the antigen and disease later disclosed, and the finding statistically analysed. It was agreed that focal epithelioid-cell collections remain as the essential criteria for the diagnosis of Kveim positivity. Diffuse epithelioid cells, Langhans' giant cells, lymphocytes, and necrosis are associated with focal epithelioid cells and hence Kveim positivity. Steroids were found to inhibit Kveim positivity. Observer disagreements were mainly due to difficulties in distinguishing macrophages from true epithelioid cells. We conclude that doubtful cells should be ignored in deciding Kveim positivity. Lymph-node-derived Kveim antigens were the most sarcoid-specific. They were more granulogenic and produced more necrosis, despite also producing the most foreign material. It is recommended that validation of new Kveim antigens must include adequate testing on patients with Crohn's disease, as the more nonspecific antigens were shown to give equal positive rates in sarcoidosis and Crohn's disease.

The reliability of clinical methods, data and judgments (second of two parts).

Interpreting Diagnostic ProceduresElectrocardiograph (Table 3)Efforts are under way to minimize sources of observer disagreement in electrocardiographic interpretation by use of codes, averaging of readings, computer analysis, and other methods.35 36 37 Even with completely reliable computerized reading, the diagnostic implications of electrocardiographic changes depend on the patient's age, sex, and clinical situation. In the following studies, the clinical information available to the readers varied.Gorman and his associates30 compared two independent interpretations of an unselected series of 561 electrocardiograms of adult inpatients. The tracings were read first by one of three electrocardiographers and then by one . . .

Observer Disagreement in the Classification and Histology of Hodgkin's Disease&lt;xref ref-type="fn" rid="FN2"&gt;2&lt;/xref&gt;

Observer Disagreement in the Classification and Histology of Hodgkin's Disease&lt;xref ref-type="fn" rid="FN2"&gt;2&lt;/xref&gt;

The Measurement of Observer Disagreement in the Recording of Signs

The Measurement of Observer Disagreement in the Recording of Signs

Observer disagreements in the interpretation of hilar shadows in chest radiographs

Summary As part of an investigation of patients with haemoptysis and a ‘normal’ chest radiograph, 18 observers accustomed to the interpretation of chest radiographs reviewed the radiographs of 69 patients examined by bronchoscopy to confirm or exclude a bronchogenic carcinoma. They were asked to classify the 138 hilar shadows as either ‘normal’, ‘suspicious’ or ‘definitely enlarged’. The clinicians in charge of these patients had previously come to an agreement about the classification of these same hilar shadows. In no case was there any abnormal shadow in the lung field. The majority of the observers were in agreement about the appearance of the majority of the hilar shadows as defined by the clinicians. Each observer's disagreements with the clinicians, whether many or few, commonly received support from only a minority of the other observers. There were wide differences between the observers in their assessment of the hilar shadows; this emphasises the difficulty in assessing a normal hilar shadow in a chest radiograph in routine clinical work. There appeared to be a general average of suspiciousness for these observers in their assessment of the hilar shadows and differences from this general level occurred in the direction of both greater and lesser suspiciousness. There was no evidence that any particular observer differed from the average in his ability to discriminate between normal and abnormal hilar shadows, according either to the clinician's assessment of abnormality or the findings at bronchoscopy.