Interobserver Kappa Research Articles

Value of fat suppression in the MRI evaluation of suspected arrhythmogenic right ventricular dysplasia.

Arrhythmogenic right ventricular dysplasia (ARVD) is characterized by intramyocardial fibrofatty change. Fat suppression performed during conventional spin-echo imaging has been used to confirm fatty infiltration. The utility of fat suppression for enhancing the interpretation of studies of suspected ARVD has not previously been formally tested. We investigated the value of fat suppression for enhancing the interpretation of intramyocardial fatty infiltration. Twenty-six consecutive patients clinically referred for evaluation of possible ARVD underwent cardiac MRI. Two independent observers reviewed the images retrospectively. Intramyocardial areas (n = 101) that had increased signal intensity relative to normal surrounding myocardium on T1-weighted conventional spin-echo images ("index areas") were identified. The index areas were interpreted for presence of fatty infiltration using two sets of images: The first set was obtained without fat suppression, and the second set was obtained with fat suppression. Agreement between reviewers and confidence of interpretation were determined and compared. Interobserver agreement was measured using a 5-point scale: 1, definitely not fat; 2, probably not fat; 3, equivocal; 4, probably fat; and 5, definitely fat. The resulting kappa values were 0.35 for non-fat-suppressed images and 0.55 for fat-suppressed images. Interobserver kappa increased from 0.67 without fat suppression to 0.90 with fat suppression using a 3-point scale: 1, not fat; 2, equivocal; and 3, fat. Confidence in the diagnosis increased from 7.2 without fat suppression to 8.8 with fat suppression (p < 0.0001) on a 10-point scale ranging from 1, not confident, to 10, very confident. The use of fat-suppressed in addition to non-fat-suppressed conventional T1-weighted spin-echo imaging increased interobserver agreement and confidence in diagnosis and evaluation of intramyocardial fatty infiltration in patients who were suspected to have ARVD.

Sonography of Fetal Choroid Plexus Cysts

To establish diagnostic criteria for the diagnosis of fetal choroid plexus cysts across gestation. Prenatal sonographic images of 166 fetuses without choroid plexus cysts were prospectively collected from 13 weeks to term. Texture characteristics of the choroid plexus regions were analyzed to quantify changes as a function of gestational age. A set of 20 fetal sonographic images with choroid plexus cysts were used to create cyst prototypes (1.5-2.5 mm), which were randomly embedded into normal choroid plexus images from varying gestational ages. A test set of 544 images was created, which included 408 images with choroid plexus cysts and 136 images without choroid plexus cysts. Four observers following a blinded study design evaluated the presence of choroid plexus cysts in the images. The influence of cyst size and gestational age on the detection of cysts was measured with receiver operating characteristic analysis and analysis of variance. Observer agreement was characterized by agreement kappa statistics. Texture analysis indicated greater echo texture heterogeneity at earlier gestational ages. The receiver operating characteristic analysis showed a corresponding decrease in diagnostic accuracy for cyst detection at earlier gestational ages. The cyst detection threshold (area under the receiver operating characteristic curve, 0.8) was 2.4 mm (SE, 0.2 mm) for 13 to 21 weeks' gestation and decreased to 1.9 mm (SE, 0.17 mm) for 22 to 38 weeks' gestation. Average interobserver and intraobserver kappa statistics were 0.37 and 0.53, respectively. Because of the changing echo texture of the choroid plexus through gestation, choroid plexus cysts must be at least 2.5 mm in diameter for confident diagnosis before 22 weeks' gestation and at least 2 mm after 22 weeks.

Reproducibility and reliability of the outerbridge classification for grading chondral lesions of the knee arthroscopically.

Few studies have investigated the accuracy and reproducibility of the Outerbridge classification system for classification of chondral damage in the knee. Arthroscopically assigned Outerbridge grades are accurate, reliable, and reproducible. Cadaver study. Six cadaveric knees underwent diagnostic arthroscopy, which was videotaped. An arthrotomy was then performed and the arthroscopically identified lesions were measured with calipers. Nine orthopaedic surgeons reviewed each video and graded each chondral lesion two separate times. Accuracy of observations was calculated based on the percentage of agreement between the grades determined during arthroscopy and arthrotomy. The overall accuracy was 68% but varied by location. The kappa coefficient between the two scores was 0.602; the arthroscopy grade was higher than the arthrotomy grade 63% of the time. The intraobserver and interobserver kappa coefficients were 0.80 and 0.52, respectively. The mean interobserver kappa between the two physicians in practice 5 years or more was 0.72, compared with 0.50 for physicians in practice less than 5 years. The Outerbridge classification was moderately accurate when used to grade chondral lesions arthroscopically. Orthopaedic surgeons can accurately grade chondral lesions of the knee with the Outerbridge classification, regardless of their level of experience.

Accuracy and reliability of the endoscopic classification of portal hypertensive gastropathy

Background: There is no consensus regarding the endoscopic classification of the severity of portal hypertensive gastropathy. This study compared the accuracy and reproducibility of the 2-category classification system (2-CCS) with the 3-category classification system (3-CCS). Methods: Ninety-eight endoscopic pictures of portal hypertensive gastropathy and 22 of nonspecific gastritis were selected. Eight duplicate sets were generated, each in a different random order. These were shown to 6 experienced endoscopists during 2 sessions 1 week apart with 4 slide sets at each session. Each picture was scored by using either the 2-CCS or 3-CCS. Kappa statistics and percent agreement were used to estimate the reproducibility and agreement. Results: The mean percentage agreement among the 4 separate readings for each observer was significantly lower for the 3-CCS compared with the 2-CCS (mean [standard deviation] = 33.5% [8.9%] vs. 64.9% [9.1%]; p = 0.0001). The mean (SD) interobserver kappa values were 0.44 (0.03) for the 3-CCS and 0.52 (0.04) for the 2-CCS ( p = 0.02), and the respective intraobserver kappa values were 0.43 (0.1) and 0.63 (0.06) ( p = 0.002). Conclusions: Even though both the 2-CCS and 3-CCS have substantial limitations with regard to specificity and reliability, there were better agreement and reproducibility with the simpler classification system for portal hypertensive gastropathy. (Gastrointest Endosc 2002;56:675-80.)

Modality-specific spike identification in simultaneous magnetoencephalography/electroencephalography: a methodological approach.

Epileptiform spikes may have a different morphology and signal-to-noise ratio in simultaneously recorded EEGs and magnetoencephalograms (MEGs) that may lead to differences in the identification of spikes if both the modalities are presented separately. Moreover, there are no criteria for MEG spikes. It is unknown to which extent the visual assessment of MEG data yields consistent and meaningful results. Nineteen patients were selected with mesial temporal lobe epilepsy who underwent whole-head simultaneous MEG/EEG. These data were split into MEG and EEG files and were assessed independently by three observers for the occurrence of spikes. Interobserver kappa values were calculated. A mean kappa value greater than 0.5 was taken as a criterion for the presence of unequivocal spikes. Index cases from the resulting four subgroups were studied further. One patient had unequivocal spikes in both modalities, one in EEG only, one in MEG only, and one did not show any unequivocal spike. Spikes on which at least two observers agreed were then subjected to a template match algorithm to test for equal morphology and distribution. Equal spikes were averaged and electrical and magnetic field maps were plotted. Unequivocal spikes were found in both MEG and EEG in one patient, in MEG only in two patients, in EEG only in two patients, and no spikes in either modality were seen in 14 patients. In the four index patients, MEG showed 50 to 80% more spikes than EEG. After averaging identical consensus spikes, MEG spikes revealed a concomitant spike in the EEG, but the reverse was not always true. Even in the patient with MEG and EEG spikes that met all selection criteria, simultaneous field maps showed unexpected inconsistencies. In most patients with mesial temporal lobe epilepsy, there are no unequivocal spikes during MEG/EEG. In some cases, however, experienced electroencephalographers can identify MEG spikes reliably. Because of a better signal-to-noise ratio, more spikes could be identified in MEG than in EEG. Simultaneous MEG/EEG recordings do not simply ensure the best of both, but one modality may improve the identification of spikes in the other. In addition, different aspects of a complex source can be revealed. Our three-step approach to combined data ensures a reproducible selection of spikes for source modeling.

Observer variability in assessing lumbar spinal stenosis severity on magnetic resonance imaging and its relation to cross-sectional spinal canal area.

Magnetic resonance image grading of lumbar spinal stenosis severity was analyzed retrospectively using a common clinical format. To assess the interobserver and intraobserver reliability of magnetic resonance image used to grade patients with lumbar spinal stenosis, as compared with cross-sectional spinal canal area. Physicians currently classify the degree of lumbar spinal stenosis on magnetic resonance imaging as mild, moderate, or severe. Unfortunately, there is no consensus on criteria for these definitions. The magnetic resonance image scans of 15 patients with lumbar stenosis were blindly rated by seven observers for the degree of central, lateral recess, and foraminal stenosis between L1-L2 and L5-S1. Weighted kappa statistics were performed to analyze the inter- and intraobserver agreement. Digitized spinal canal area measurements were calculated. Linear regression models were used to assess the reliability of the grading system in predicting the cross-sectional area. The average interobserver kappa score was 0.26. Within different specialties, the interobserver reliability was higher among radiologists (0.40), followed by neurosurgeons (0.21) and orthopedic surgeons (0.15). The average intraobserver kappa score was 0.11, rising to 0.43 after categories were combined (P = 0.001). The classification of central stenosis highly predicted spinal canal area (P < 0.001). The findings indicate only a fair level of agreement among all observers. However, the ability of the various readers to predict the degree of central stenosis was high. Further studies should evaluate a consensus-based, standardized magnetic resonance image classification aimed at improved agreement among observers.

Agreement among clinicians in the recognition of patterns of optic disk damage in glaucoma

PURPOSE: To determine the interobserver and intraobserver agreement in the recognition of different patterns of glaucomatous optic disk damage and evaluate if these patterns changed over time in patients followed for a number of years. METHODS: Patients with early to moderate glaucoma (n = 105) were consecutively enrolled to participate in a prospective observational study. In the first part of the present study, optic disk photographs obtained closest to patient’s entry date in the prospective study were classified in a masked fashion by three observers according to the pattern of optic disk damage into one of the following: (1) focal, (2) myopic, (3) senile sclerotic, (4) concentric cup enlargement, (5) normal appearance, or (6) miscellaneous (those disks that did not qualify for any of the other groups). The observers were also asked to assign a confidence score for each classification, ranging from 1 (low confidence) to 5 (highest confidence). The three observers reclassified the photographs after a minimal period of 2 months, in order to assess intraobserver agreement. In a second part of the study, one observer reviewed, in a masked fashion, all the optic disk photographs that had been taken during the routine follow-up of the 105 patients in order to evaluate whether the classification of disk pattern changed over time. RESULTS: Intraobserver agreement yielded kappa values (95% confidence interval [CI]) from 0.51 (CI, 0.40 to 0.62) to 0.85 (CI, 0.77 to 0.93) depending on the observer. Interobserver agreement kappa values between all three observers was 0.40 (CI, 0.35 to 0.46), but it improved if photographs classified with moderate or higher degrees of confidence were included (0.52 [CI, 0.44 to 0.60]). During a mean follow-up period of 8.2 ± 4.8 years, 41.6% of the eyes were always classified into the same group and 23.6% of the eyes were classified at least once into two or more of the four patterns of damage (groups 1 to 4) CONCLUSIONS: Intraobserver agreement on the pattern of optic disk damage was generally very good, with some variation among the observers. Interobserver agreement was reasonable and increased proportionally with the confidence in the classification. During long-term follow-up, the pattern of optic disk damage usually did not change. This type of classification can probably be used accurately in clinical practice.

Magnetic resonance classification of lumbar intervertebral disc degeneration.

A reliability study was conducted. To develop a classification system for lumbar disc degeneration based on routine magnetic resonance imaging, to investigate the applicability of a simple algorithm, and to assess the reliability of this classification system. A standardized nomenclature in the assessment of disc abnormalities is a prerequisite for a comparison of data from different investigations. The reliability of the assessment has a crucial influence on the validity of the data. Grading systems of disc degeneration based on state of the art magnetic resonance imaging and corresponding reproducibility studies currently are sparse. A grading system for lumbar disc degeneration was developed on the basis of the literature. An algorithm to assess the grading was developed and optimized by reviewing lumbar magnetic resonance examinations. The reliability of the algorithm in depicting intervertebral disc alterations was tested on the magnetic resonance images of 300 lumbar intervertebral discs in 60 patients (33 men and 27 women) with a mean age of 40 years (range, 10-83 years). All scans were analyzed independently by three observers. Intra- and interobserver reliabilities were assessed by calculating kappa statistics. There were 14 Grade I, 82 Grade II, 72 Grade III, 68 Grade IV, and 64 Grade V discs. The kappa coefficients for intra- and interobserver agreement were substantial to excellent: intraobserver (kappa range, 0.84-0.90) and interobserver (kappa range, 0.69-0.81). Complete agreement was obtained, on the average, in 83.8% of all the discs. A difference of one grade occurred in 15.9% and a difference of two or more grades in 1.3% of all the cases. Disc degeneration can be graded reliably on routine T2-weighted magnetic resonance images using the grading system and algorithm presented in this investigation.

Adolescent Idiopathic Scoliosis

The lack of a reliable, universally acceptable system for classification of adolescent idiopathic scoliosis has made comparisons between various types of operative treatment an impossible task. Furthermore, long-term outcomes cannot be determined because of the great variations in the description of study groups. We developed a new classification system with three components: curve type (1 through 6), a lumbar spine modifier (A, B, or C), and a sagittal thoracic modifier (-, N, or +). The six curve types have specific characteristics, on coronal and sagittal radiographs, that differentiate structural and nonstructural curves in the proximal thoracic, main thoracic, and thoracolumbar/lumbar regions. The lumbar spine modifier is based on the relationship of the center sacral vertical line to the apex of the lumbar curve, and the sagittal thoracic modifier is based on the sagittal curve measurement from the fifth to the twelfth thoracic level. A minus sign represents a curve of less than +10 degrees, N represents a curve of 10 degrees to 40 degrees, and a plus sign represents a curve of more than +40 degrees. Five surgeons, members of the Scoliosis Research Society who had developed the new system and who had previously tested the reliability of the King classification on radiographs of twenty-seven patients, measured the same radiographs (standing coronal and lateral as well as supine side-bending views) to test the reliability of the new classification. A randomly chosen independent group of seven surgeons, also members of the Scoliosis Research Society, tested the reliability and validity of the classification as well. The interobserver and intraobserver kappa values for the curve type were, respectively, 0.92 and 0.83 for the five developers of the system and 0.740 and 0.893 for the independent group of seven scoliosis surgeons. In the independent group, the mean interobserver and intraobserver kappa values were 0.800 and 0.840 for the lumbar modifier and 0.938 and 0.970 for the sagittal thoracic modifier. These kappa values were all in the good-to-excellent range (>0.75), except for the interobserver reliability of the independent group for the curve type (kappa = 0.74), which fell just below this level. This new two-dimensional classification of adolescent idiopathic scoliosis, as tested by two groups of surgeons, was shown to be much more reliable than the King system. Additional studies are necessary to determine the versatility, reliability, and accuracy of the classification for defining the vertebrae to be included in an arthrodesis.

Detection of lung cancer on the chest radiograph: a study on observer performance

Study objectives: to study the validity and observers consistency in the detection of lung cancer on the chest radiograph. Materials and methods: the chest radiographs of 100 clinical cases were interpreted by 14 observers. The radiographs were obtained from 30 patients with initially missed but histopathologically proven non-small cell lung cancer (NSCLC), 35 patients with other cardiopulmonary diseases and 35 patients with no abnormalities. The observers consisted of ten experienced radiologists, two-experienced chest physicians and two residents in radiology. All observers were unaware of the study design. The validity and observer consistency was determined for each observer. Results: the mean sensitivity and specificity of the ten radiologists were 0.36 and 0.90. For the two chest physicians, the mean sensitivity and specificity were 0.29 and 0.96. For the two residents in radiology, mean sensitivity and specificity were 0.25 and 0.94. The mean interobserver kappa and mean intraobserver kappa for the radiologists were 0.38 and 0.54. For the two chest physicians, the mean interobserver kappa was 0.43, while the intraobserver kappa was 0.59. For the two residents in radiology, mean interobserver kappa was 0.35 and the intraobserver kappa was 0.42. There was no significant relation between the consistency parameters and validity parameters. The interobserver and intraobserver kappa values showed good correlation. Conclusion: the validity of the chest radiograph and observers consistency in the detection of nodular lung cancer varies widely. The level of experience is likely to influence the diagnostic performance.

Classification of osteonecrosis of the femoral head. Reliability, reproducibility, and prognostic value.

The purpose of the current investigation was to determine interobserver and intraobserver reliability of the classification system of Steinberg et al for osteonecrosis of the femoral head. Sixty-five anteroposterior and lateral radiographs of hips were selected randomly from a pool of patients with confirmed osteonecrosis of the femoral head. Six fellowship-trained observers viewed the radiographs (Reading 1). The observers used six main stages of the classification excluding A, B, and C subgroups. The same observers viewed the radiographs 4 months later in reverse order (Reading 2). Reading 1 was used to calculate interobserver kappa values. Reading 2 was used to calculate intraobserver kappa values. Stage-specific kappa values for interobserver variation between all viewers were as follows: Stage I, k = 0.64; Stage II, k = 0.51; Stage III, k = 0.21; Stage IV, k = 0.49; Stage V, k = 0.36; and Stage VI, k = 0.80. Stage-specific kappa values for intraobserver variation between all viewers were as follows: Stage I, k = 0.74; Stage II, k = 0.60; Stage III, k = 0.46; Stage IV, k = 0.59; Stage V, k = 0.27; and Stage VI, k = 0.78. An average of 10 of 21 (48%) of these errors involved Stage III. An average of 6.3 of 21 (30%) intraobserver errors involved Stage V. The presence of the crescent sign in Stage III and joint space narrowing in Stage V markedly diminished the overall reliability of any four- to six-stage classification system. Based on the authors' experience and analysis of the current classifications of osteonecrosis of the femoral head, an easy and reproducible Pittsburgh classification system is proposed.

The accuracy and reproducibility of radiographic assessment of stress-shielding. A postmortem analysis.

Although periprosthetic bone loss remains a major concern in total hip arthroplasty, radiographic assessment of such loss is both difficult and subjective. In the present study, we assessed the ability of orthopaedic surgeons to reproducibly recognize changes in periprosthetic bone density on radiographs. We hypothesized that assessment of periprosthetic bone loss on plain radiographs is not reliable enough to justify its use in outcomes research. Twenty-nine unilateral total hip replacements and the surrounding bone were retrieved at autopsy, and radiographs were made; radiographs of the contralateral, normal femur were also made after implantation of an identical prosthesis and used as a control. Three orthopaedic surgeons independently examined the specimen radiographs and classified bone loss in each of sixteen femoral zones. Bone loss was recorded as present if the bone of the femur that had had in vivo implantation showed evidence of cortical thinning, increased porosity, or decreased density (either cortical or trabecular) when compared with the control femur. The kappa coefficient was used to quantify interobserver and intraobserver reproducibility in determining bone loss for the 464 zones examined and in determining the Engh and Bobyn stress-shielding classification of each femur. In fourteen femoral pairs, bone loss was also quantified with dual-energy x-ray absorptiometry, and the resulting value was then compared with the bone-loss classification that had been determined radiographically. First, the surgeons agreed on the presence or absence of bone loss in 73 percent (337) of the 464 zones. The interobserver kappa value of 0.58 denoted only good reproducibility. The intraobserver reproducibility was better; the surgeon's initial evaluation of bone loss agreed with his second evaluation for 90 percent of the zones (kappa = 0.74). Second, the three surgeons agreed on the degree of stress-shielding, according to the Engh and Bobyn classification, in 66 percent (nineteen) of the twenty-nine femora. The kappa value for this comparison was only 0.27, indicating marginal reproducibility. Third, although there was some agreement among reviewers when there was 20 to 60 percent reduction in bone-mineral content as determined with dual-energy x-ray absorptiometry, excellent agreement among the examiners (kappa = 0.85) was not achieved until bone loss averaged 70 percent. On the basis of these results, we suggest caution in interpreting results from studies of femoral bone loss that have used plain radiographic analysis if the authors have not provided interobserver reliability data. We question the utility of evaluating periprosthetic bone loss on radiographs, since the loss is not reproducibly recognized until 70 percent of the bone is gone.

Radiographic fracture assessments: which ones can we reliably make?

To identify the fracture characteristics that can be reliably assessed by analysis of plain radiographs of tibial plateau fractures. Radiographic review study. Five orthopaedic traumatologists served as observers. Observers made assessments based on the radiographs of fifty-six tibial plateau fractures. Precise definitions of the assessments to be made were agreed on by all observers. The tested assessments included raters' abilities to identify and locate fracture lines, identify the presence of fracture displacement and comminution, make quantitative measurements of displacement, and characterize qualitative features of fractures. For thirty-eight of the fractures that had a computed tomography (CT) scan available, assessments were repeated using both radiographs and CT scans. To characterize interobserver reliability, percentage agreement and kappa statistics were calculated for categorical variables, and intraclass correlation coefficients (ICC) were calculated for noncategorical variables. Reliability of the assessments varied widely. Determining the location of fracture lines had the greatest reliability, whereas the subjective assessments of fracture stability and energy showed the poorest reliability. Although the ICCs for quantitative measurements approached acceptable levels, the tolerance limits were extremely wide. The addition of a CT scan improved the reliability of most assessments, but not to a statistically significant degree. Many basic radiographic interpretations relied on in making treatment decisions are made variably by observers. Using experienced raters and precise definitions of fracture assessments does not guarantee a high level of agreement. Discrete assessments have higher interrater agreements than do more qualitative assessments. Quantitative measures have wide tolerance limits and, therefore, probably cannot be used reproducibly to classify fractures or make treatment decisions. We conclude the reliability of fracture classification is limited by raters' abilities to agree on basic radiographic assessments.

Breast Imaging Reporting and Data System: inter- and intraobserver variability in feature analysis and final assessment.

We sought to evaluate the use of the Breast Imaging Reporting and Data System (BI-RADS) standardized mammography lexicon among and within observers and to distinguish variability in feature analysis from variability in lesion management. Five experienced mammographers, not specifically trained in BI-RADS, used the lexicon to describe and assess 103 screening mammograms, including 30 (29%) showing cancer, and a subset of 86 mammograms with diagnostic evaluation, including 23 (27%) showing cancer. A subset of 13 screening mammograms (two with malignant findings, 11 with diagnostic evaluation) were rereviewed by each observer 2 months later. Kappa statistics were calculated as measures of agreement beyond chance. After diagnostic evaluation, the interobserver kappa values for describing features were as follows: breast density, 0.43; lesion type, 0.75; mass borders, 0.40; special cases, 0.56; mass density, 0.40; mass shape, 0.28; microcalcification morphology, 0.36; and microcalcification distribution, 0.47. Lesion management was highly variable, with a kappa value for final assessment of 0.37. When we grouped assessments recommending immediate additional evaluation and biopsy (BI-RADS categories 0, 4, and 5 combined) versus follow-up (categories 1, 2, and 3 combined), five observers agreed on management for only 47 (55%) of 86 lesions. Intraobserver agreement on management (additional evaluation or biopsy versus follow-up) was seen in 47 (85%) of 55 interpretations, with a kappa value of 0.35-1.0 (mean, 0.60) for final assessment. Inter- and intraobserver variability in mammographic interpretation is substantial for both feature analysis and management. Continued development of methods to improve standardization in mammographic interpretation is needed.

Local cardiac wall stabilization influences the reproducibility of regional wall motion during off-pump coronary artery pass surgery.

Myocardial ischemia is a risk factor during off-pump coronary artery bypass procedures. The development of new regional wall motion abnormalities assessed by transesophageal echocardiography (TEE) is a very sensitive sign of myocardial ischemia. To facilitate anastomosis, the epicardial area of the anastomosis site is often immobilized by a "stabilizer." This study was designed to investigate whether cardiac wall stabilization with an epicardial stabilizer could affect the interpretation of wall motion during coronary anastomosis without cardiopulmonary bypass. The TEE videotapes of 15 adult patients were investigated. Left ventricular (LV) transgastric short and long axis views were divided according to a modified 16-segment method. LV wall motion was scored using a 5-grade scale by two independent blinded investigators during pre-occlusion, occlusion, and reperfusion of anastomosed coronary arteries. The wall motion scores of a stabilized segment combined with two adjacent segments were compared with those of non-stabilized segments. Interobserver agreement was assessed using the weighted kappa statistic. A total of 216 segments were analyzed by two investigators. The interobserver kappa coefficient in pre-occlusion and reperfusion periods was 0.87, 0.87 and 0.86, 0.87, respectively, indicating high agreements without stabilizer. During the occlusion period in stabilized and non-stabilized segments, it was 0.59 and 0.76, respectively, showing significantly less reproducibility in the presence of stabilizer. Cardiac wall stabilization affects the reproducibility in the interpretation of regional wall motion during off-pump coronary artery bypass surgery. Caution should be used when monitoring for myocardial ischemia using TEE during coronary artery bypass surgery with epicardial stabilizer.

Visual analysis of serial T2-weighted MRI in multiple sclerosis: intra- and interobserver reproducibility.

We evaluated the effect of consensus formation and training on the agreement between observers in scoring the number of new and enlarging multiple sclerosis (MS) lesions on serial T2-weighted MRI studies. The baseline and month 9 MRI studies of 16 patients with a range of MRI activity were used (dual-echo conventional spin-echo sequence, TR 2000, TE 34 and 90 ms, 5 mm contiguous slices, inplane resolution 1 mm). First, the serial studies were visually analysed for the presence of new and enlarging lesions, on two occasions, by five experienced observers, without adopting any consensus strategy and in isolation. Next, the observers met to identify the common sources of inconsistencies in reporting between observers and formulate consensus rules. Finally, a further independent reading session was performed on the same MRI dataset, this time applying the consensus rules. Agreement between observers was assessed using kappa scores. Without the consensus rules, interobserver kappa scores for the first and second reading sessions for new lesions were only 0.51 and 0.39 respectively; agreement for enlarging lesions was even worse. The mean intraobserver kappa score for new lesions was higher at 0.72, reflecting the fact that the observers were consistently applying their individual assessment strategies. Application of the consensus rules did not lead to a significant improvement in inter observer kappas; the kappa scores adopting the guidelines were 0.46 and 0.21 for new and enlarging lesions respectively. Consensus guidelines thus did not improve the reproducibility of visual analysis of serial T2-weighted MRI, and the level of agreement between observers remained only moderate. Suboptimal repositioning is likely to be a major source of residual variability and this suggests a future role for image registration strategies; until then, a single observer, or pair of observers working in consensus, should be used in MS studies.

A quantitative scale of acanthosis nigricans.

To develop and validate a scale for acanthosis nigricans (AN). Subjects were participants from the San Antonio Family Diabetes Study and the San Antonio Family Heart Study. A total of 406 subjects were independently examined for AN by at least two observers. Five locations were examined: the neck, axilla, elbows, knuckles, and knees. Interobserver concordance and kappa statistics were calculated to determine replicability of the scale. Comparisons of diabetes-related risk factors by AN score were also calculated. Only the neck had consistently high kappa statistics, and thus, other locations were excluded from further analyses. Elevated AN was strongly associated with elevated fasting insulin and BMI in both diabetic and nondiabetic subjects. Elevated AN was also strongly associated with elevated fasting glucose, systolic blood pressure, and diastolic blood pressure, and with decreased HDL in nondiabetic subjects. In diabetic subjects, elevated AN was associated with elevated total cholesterol. We have developed a scale for AN that is easy to use, has high interobserver reliability in Mexican Americans, and correlates well with fasting insulin and BMI. This scale will permit longitudinal and cross-sectional evaluation of AN and will permit the evaluation of AN as a trait in genetic studies.

Sonography of solid breast lesions: observer variability of lesion description and assessment.

The purpose of this study was to measure the level of inter- and intraobserver agreement and to evaluate the causes of variability in radiologists' descriptions and assessments of sonograms of solid breast masses. Sixty sonograms of solid masses were evaluated independently by five radiologists. Observers used the lexicon of a recently published benchmark report on sonographic appearances of breast masses to determine mass shape, margin, echogenicity, echo texture, presence of echogenic pseudocapsule, and acoustic transmission. Final diagnostic assessments were determined by applying the rule-based model of the same benchmark report to the radiologists' descriptions. In addition, one observer interpreted each case twice to evaluate intraobserver variability. Inter- and intraobserver variability were measured using Cohen's kappa statistic. We also investigated causes of variability in radiologists' descriptions. Interobserver agreement ranged from lowest for determining the presence of an echogenic pseudocapsule (kappa = .09) to highest for determining mass shape (kappa = .8). Intraobserver agreement was lowest for mass echo texture (kappa = .24) and greatest for mass shape (kappa = .79). Variability in descriptions of lesions contributed to interobserver (kappa = .51) and some intraobserver (kappa = .66) inconsistency in assessing the likelihood of malignancy. Lack of uniformity among observers' use of descriptive terms for solid breast masses resulted in inconsistent diagnoses. The need for improved definitions and additional illustrative examples could be addressed by developing a standardized lexicon similar to that of the Breast Imaging Reporting and Data System.

Radiolucent lines and component stability in knee arthroplasty

The radiolucent lines and the stability of the components of 66 knee arthroplasties were assessed by six orthopaedic surgeons on conventional anteroposterior and lateral radiographs and on fluoroscopic views which had been taken on the same day. The examiners were blinded as to the patients and clinical results. The interpretation of the radiographs was repeated after five months. On fluoroscopically-assisted radiographs four of the six examiners identified significantly more radiolucent lines for the femoral component (p &lt; 0.05) and one significantly more for the tibial implant. Five examiners rated more femoral components as radiologically loose on fluoroscopically-assisted radiographs (p = 0.0008 to 0.0154), but none did so for the tibial components. The mean intra- and interobserver kappa values were higher for fluoroscopically-assisted radiographs for both components. We have shown that fluoroscopically-assisted radiographs allow more reproducible, and therefore reliable, detection of radiolucent lines in total knee arthroplasty. Assessment of the stability of the components is significantly influenced by the radiological technique used. Conventional radiographs are not adequate for evaluation of the stability of total knee arthroplasty and should be replaced by fluoroscopically-assisted films.

Measurement of endometrial thickness with transvaginal ultrasonography: is it reproducible?

To assess the intraobserver and interobserver variability of transvaginal ultrasonographic measurement of endometrial thickness, two investigators (M.V., M.-F. D.) independently evaluated two successive measurements in 55 patients with a 5.0 MHz transvaginal transducer. Each observer was blinded to his or her own and to the other's results. Measurements were taken at the maximal thickness in the longitudinal uterine axis. The intraclass coefficients evaluating the agreement between two measures of a same observer were 94% and 97%; the coefficient was 94% between the two investigators. Intraobserver kappa values (agreement on endometrium < or = 5 mm and > 5 mm) were 0.70 and 0.81, respectively; the interobserver kappa value was 0.74. Transvaginal sonographic measurement of endometrial thickness has excellent intraobserver and good interobserver agreement. However, in clinical practice, an endometrial thickness measuring +/- 1 mm of a cut-off value should be interpreted with caution, especially when this may modify the clinical management.