Mean Interobserver Variation Research Articles

Use of Crowd Innovation to Develop an Artificial Intelligence–Based Solution for Radiation Therapy Targeting

Radiation therapy (RT) is a critical cancer treatment, but the existing radiation oncologist work force does not meet growing global demand. One key physician task in RT planning involves tumor segmentation for targeting, which requires substantial training and is subject to significant interobserver variation. To determine whether crowd innovation could be used to rapidly produce artificial intelligence (AI) solutions that replicate the accuracy of an expert radiation oncologist in segmenting lung tumors for RT targeting. We conducted a 10-week, prize-based, online, 3-phase challenge (prizes totaled $55 000). A well-curated data set, including computed tomographic (CT) scans and lung tumor segmentations generated by an expert for clinical care, was used for the contest (CT scans from 461 patients; median 157 images per scan; 77 942 images in total; 8144 images with tumor present). Contestants were provided a training set of 229 CT scans with accompanying expert contours to develop their algorithms and given feedback on their performance throughout the contest, including from the expert clinician. The AI algorithms generated by contestants were automatically scored on an independent data set that was withheld from contestants, and performance ranked using quantitative metrics that evaluated overlap of each algorithm's automated segmentations with the expert's segmentations. Performance was further benchmarked against human expert interobserver and intraobserver variation. A total of 564 contestants from 62 countries registered for this challenge, and 34 (6%) submitted algorithms. The automated segmentations produced by the top 5 AI algorithms, when combined using an ensemble model, had an accuracy (Dice coefficient = 0.79) that was within the benchmark of mean interobserver variation measured between 6 human experts. For phase 1, the top 7 algorithms had average custom segmentation scores (S scores) on the holdout data set ranging from 0.15 to 0.38, and suboptimal performance using relative measures of error. The average S scores for phase 2 increased to 0.53 to 0.57, with a similar improvement in other performance metrics. In phase 3, performance of the top algorithm increased by an additional 9%. Combining the top 5 algorithms from phase 2 and phase 3 using an ensemble model, yielded an additional 9% to 12% improvement in performance with a final S score reaching 0.68. A combined crowd innovation and AI approach rapidly produced automated algorithms that replicated the skills of a highly trained physician for a critical task in radiation therapy. These AI algorithms could improve cancer care globally by transferring the skills of expert clinicians to under-resourced health care settings.

Classification of Pediatric Supracondylar Fractures: Comparison between the Gartland and the Lagrange & Rigault Classifications

Background The supracondylar fracture of the humerus is one of the most common fractures in pediatric age. Many classifications were proposed, with Gartland being the most widely used one and Lagrange and Rigault (L&R) the most used in French-speaking countries. Objectives The goal of this study was to compare the Wilkins-modified Gartland classification with the L&R classification in terms of reproducibility. Methods Three observers with similar training levels classified 35 fractures according to both classifications twice to evaluate both intra and interobserver variation. Results The mean intraobserver variation was 0.78 and 0.77 for Gartland and L&R classifications, respectively, and the mean interobserver variation was 0.55 and 0.62 for Gartland and L&R classifications, respectively. Conclusions Both classifications reveal adequate to usage in clinical and investigational practices, which is consistent with the literature.

Reliability of Estimates of ACL Attachment Locations in 3-Dimensional Knee Reconstruction Based on Routine Clinical MRI in Pediatric Patients

Background: Current techniques of anterior cruciate ligament (ACL) reconstruction focus on the placement of femoral and tibial tunnels at anatomic ACL attachments, which can be difficult to identify intraoperatively. Purpose: To determine whether the 3-dimensional (3D) center of ACL attachments can be reliably detected from routine magnetic resonance imaging (MRI) in patients with intact ACLs and whether the reliability of this technique changes if the ACL is torn. Study Design: Cohort study (diagnosis); Level of evidence, 3. Methods: A computer technique was developed in which users identify points along ACL attachments on routine clinical MRI of preoperative knees. These attachments are then displayed on a 3D MRI reconstruction, which can be used as a visual guide for the surgeon during arthroscopic surgery. Thirty-seven pediatric patients (age range, 10-17 years) with ACL tears and 37 controls with intact ACLs were examined. Two blinded observers identified cruciate ligament attachments on routine clinical 1.5-T MRI of knees. From the resulting 3D model, the location of the center of each ligament attachment site and its area were calculated and reliability assessed. Results: Mean interobserver variation of the centers of ACL attachments for the intact versus torn ACL was 1.7 ± 0.9 mm versus 1.8 ± 1.1 mm (femoral) and 1.4 ± 0.9 mm versus 1.7 ± 1.0 mm (tibial), respectively (P > .05). The 95% confidence interval for the center location was at most 4 mm. The identified ACL attachment areas were more variable, with interobserver reliability ranging from fair to excellent by the intraclass correlation coefficient. Overlap of ligament areas between observers for the intact versus torn ACL was 70% ± 15% versus 73% ± 12% (femoral) and 79% ± 9% versus 78% ± 10% (tibial), respectively (P > .05). In all cases, intraobserver reliability was superior to interobserver reliability. Conclusion: The 3D locations of ACL tibial and femoral attachment centers were identified from routine clinical MRI with variability averaging less than 2 mm between 2 observers. The margin of error was at most 4 mm, representing the thickness of a single axial MRI slice, whether the ACL was intact or torn. Remnant tissue at attachments allows a reliable assessment even of torn ligaments. Identification of the ligament attachment areas was more user dependent than was identification of the attachment centers.

Magnetic Resonance Angiography Shows Dilatation of the Middle Cerebral Artery after Infusion of Glyceryl Trinitrate in Healthy Volunteers

Previous studies have reported dilatation of the middle cerebral artery (MCA) during acute glyceryl trinitrate (GTN)-induced headache, using imaging techniques such as transcranial Doppler (TCD), positron emission tomography (PET) and single photon emission computerized tomography (SPECT). In the present study we aimed to evaluate whether magnetic resonance angiography (MRA) may be used to examine the effect of GTN on the MCA, with respect to changes in diameter and cross-sectional area in healthy volunteers. In addition, we wanted to determine the intra- and inter-observer variation of the method. In a randomized, double blind, crossover study 12 healthy volunteers received intravenous infusion of GTN (0.5 microg/kg/min for 20 min) or placebo. Using 1.5 Tesla MRA, we recorded changes in the diameter and cross-sectional area of MCA before, during and after infusion of GTN. The MRA images were evaluated by two blinded, independent observers/neuroradiologists. The primary endpoints were the differences in the AUC for diameter and cross-sectional area of the MCA between the two experimental conditions and the intra- and inter-observer variation. The areas under the curve (AUC) of the MCA diameter and cross-sectional area were significantly greater after GTN than after placebo (P < 0.05). The intra-observer variation (day-to-day) at baseline was 8.3% and 10.9% for the two observers. The mean inter-observer variation of the cross-sectional MCA area was 15.5% and for the diameter measurements 8%. The present study shows that the MRA method gives a reliable semi-quantitative index of the vascular changes in the intra-cerebral arteries after infusion of GTN and may be useful for headache research.

Reproducibility of intra-operative measurement of the mechanical axes of the lower limb during total knee replacement with a non-image-based navigation system

Objective: The restoration of a normal mechanical axis of the lower limb following total knee prosthesis (TKP) depends on the accuracy of the intra-operative measurement of the femoro-tibial angle. We have studied the reproducibility of intra-operative measurement of the coronal mechanical femoro-tibial axis with the OrthoPilot® (Aesculap, Tuttlingen, Germany) non-image-based navigation system.Material and Methods: A consecutive series of 20 TKP (Aesculap SEARCH Evolution® prosthesis) implanted by the same surgical team of two senior orthopedic surgeons was analyzed. They used a non-image-based navigation system that allows the mechanical axes of the femur and tibia to be defined with a kinematic analysis. The operating surgeon and assistant surgeon performed the kinematic analysis twice and once, respectively, and measured coronal mechanical femoro-tibial angles in maximal extension and at 90° flexion without varus or valgus stress.Results: The mean intra-observer variation in the measurement of the coronal mechanical femoro-tibial angle in maximal extension was 0.1° (SD=0.7°). The mean intra-observer variation in the measurement of the coronal mechanical femoro-tibial angle at 90° of knee flexion was 0.2° (SD=0.6°). The mean inter-observer variation in the measurement of the coronal mechanical femoro-tibial angle in maximal extension was 0.1° (SD=0.7°). The mean inter-observer variation in the measurement of the coronal mechanical femoro-tibial angle in maximal extension was 0.0° (SD=0.6°). There were no significant differences and a high correlation between all paired intra- and inter-observer measurements.Conclusion: This system allows high reproducibility of the intra-operative measurement of the mechanical axes of the lower limb by a non-image-based kinematic registration of the hip, knee and ankle centers.

Toward a consensus on reference values for thyroid volume in iodine-replete schoolchildren: results of a workshop on inter-observer and inter-equipment variation in sonographic measurement of thyroid volume.

Interpretation of thyroid ultrasonography for assessing goiter prevalence requires valid reference criteria from iodine-sufficient populations. Reports have suggested the current reference criteria for thyroid volume (T(vol)) of WHO/ICCIDD (International Council for the Control of Iodine Deficiency Disorders) may be too high. Our objective was to determine if inter-observer and/or inter-equipment variability contributes to the disagreement in sonographic T(vol) in children reported from iodine-sufficient areas. A 2-day workshop in which four experienced ultrasound examiners from around Europe measured T(vol) in 45 6--12-year-old Swiss schoolchildren using four different portable ultrasound machines. One of the participating examiners (observer A) had generated the T(vol) data in European children that are the basis for the WHO/ICCIDD reference criteria. Sonographic T(vol) was measured in each child by all four examiners on all four machines. Six hundred and eighty-four examinations were completed, with examiners having no knowledge of one another's results. Inter-observer and inter-equipment variation was calculated. Mean inter-equipment variation in T(vol) was 15.2% (95% CI: 14.1, 16.3%). There were no significant differences in T(vol) between equipment (P=0.51). For all observers, the mean inter-observer variation in T(vol) was 25.6% (95% CI: 23.9, 27.2%). At all ages and all body surface areas, there was a large systematic measurement bias (+30% volume) between the mean T(vol) of observer A and the mean Tvol of observers B, C and D. Reanalysis using data from observers B, C and D reduced the mean inter-observer variation in T(vol) to 13.3% (95% CI: 11.9, 14.7%). A correction factor for the systematic difference of operator A for the P50 and P97 of T(vol) was estimated using analysis of covariance. When applied to the WHO/ICCIDD reference data, it sharply reduced the discrepancy between the WHO/ICCIDD criteria and those from other iodine-sufficient children around the world. Inter-equipment error contributes minimally to reported differences in sonographic T(vol). Even among experienced examiners, inter-observer variation in sonographic T(vol) in children can be high, and probably contributes to the current disagreement on normative values in iodine-sufficient children. A systematic bias at least partially explains why the WHO/ICCIDD reference data differ from those reported from other iodine-sufficient children around the world. The findings argue strongly for the standardization of methods used for sonographic measurement of T(vol) in children.

Automatic Border Detection for Assessment of Left Ventricular Diastolic Function Among Normal Neonates: Comparison with Doppler Echocardiography.

Doppler echocardiography is the standard noninvasive method to assess left ventricular (LV) diastolic function. Recently, automatic border detection (ABD), a method based on analysis of integrated ultrasonic backscatter, has been introduced permitting real-time, on-line assessment of LV diastolic function. A comparison of these methods in normal, full-term neonates has not been performed. Therefore, the objectives of this study were to evaluate the usefulness of ABD in the assessment of LV diastolic function among normal neonates, to compare parameters obtained with the ABD method with standard Doppler-derived indexes of diastolic function, and to assess the reproducibility of ABD measurements. We studied 17 consecutive normal neonates during natural sleep with both methods shortly after birth (mean 17.4 +/- 3.9 h) and approximately 2 weeks later (mean 14.8 +/- 2.2 days). An average of five consecutive cardiac cycles were performed. Similar to Doppler indexes, no significant change in any ABD parameter of diastolic function occurred between the early and later studies. A complete ABD study could be performed within 5 minutes. Mean interobserver variation for individual ABD measurements ranged from 0% to 11%. Compared with Doppler, rapid filling fraction was greater and atrial filling fraction was less with ABD. Regression analysis showed poor correlation of these parameters between methods, but their ratio by each method remained constant between studies. A similar poor correlation existed between peak E wave velocity by Doppler and peak rapid filling rate by ABD and between peak A wave velocity by Doppler and peak atrial filling rate by ABD. These differences may be explained by technical factors and different aspects of diastolic filling assessed by each method. This study indicated that ABD was a feasible and reproducible method compared with Doppler echocardiography for serial evaluation of LV diastolic function among neonates.

Abnormal fetal cardiac axis in the detection of intrathoracic anomalies and congenital heart disease.

In a 14-month period, 409 women with singleton gestations referred for perinatal ultrasound consultation underwent evaluation of the fetal cardiac axis. Cardiac and intrathoracic anomalies were confirmed either by neonatal echocardiography or autopsy. Overall, 32 fetuses had an abnormal axis (nine, smaller axis than normal; 23, larger axis than normal). Of the 29 found to have cardiac (n = 24) or intrathoracic (n = 5) anomalies, 23 had an abnormal axis. The median cardiac axis of the normal group (44.0 degrees) was significantly smaller than that of the cardiac/intrathoracic anomaly group(60.0 degrees) (p = 0.002). The cardiac axis was independent of gestational age. The mean interobserver variation was 1.3 +/- 1.8 degrees. The sensitivity of an abnormal axis (< 28 degrees or > 59 degrees) in detecting congenital heart disease or intrathoracic anomalies was 23/29 (79.3%), with specificity of 371/380 (97.5%), positive predictive value of 23/32 (71.9%), and negative predictive value of 371/377 (98.4%). Of those with a cardiac anomaly and an abnormal axis (n = 18), five were felt to have an initial normal four-chamber view. An abnormal fetal cardiac axis, either larger or smaller than normal, is suggestive of a cardiac or intrathoracic anomaly and requires further investigation, such as fetal echocardiography. The cardiac axis should be considered with the four-chamber view in fetal ultrasound evaluation.

Diurnal variation of Cobb angle measurement in adolescent idiopathic scoliosis.

To investigate the possible influence of gravity on daily variations of curve magnitude in adolescent idiopathic scoliosis, standardized anteroposterior standing radiographs of the spine were taken at 8:00 AM and repeated in the same conditions at 8:00 PM on the same day, 19 girls with idiopathic scoliosis, aged 10-16 years were included. The Cobb angle of the primary thoracic curve was measured blindly by four skilled orthopaedic surgeons. The mean interobserver variation in measuring Cobb angles on the 38 radiographs was 1.6 degrees. The average Cobb angle on morning radiographs was 60 degrees (range, 42-91 degrees) as compared to 65 degrees (range, 47-89 degrees) in the afternoon, a significant (P < 0.001) difference using a paired two-sided Student t test. It was concluded that there is a statistically and clinically significant daily increase of curve severity in moderate to severe idiopathic scoliosis.

Leg length inequality measured by ultrasound and clinical methods

Leg length inequality (LLI) was determined by ultrasound and two clinical methods in 100 subjects without previous disorders of the lower extremities. The mean LLI by ultrasound was 4.1 mm (range: 0–16 mm). An LLI of 10 mm or more was found in 4% of the subjects. The mean inter-observer variation by ultrasound was 0.9 mm (range: ÷5–8 mm), which was significantly less than that obtained by clinical methods. The 95% confidence interval of inter-observer variation (±2 SD) by ultrasound was ± 5 mm and this interval is assumed to represent the accuracy of the method. Because LLI measurement by ultrasound is more reliable than clinical methods, ultrasound is recommended for routine use in clinical practice.

Interobserver and intraobserver variations in sonographic renal length measurements in children.

Sonographic measurement of renal length is used commonly to evaluate growth of the kidneys in children. However, no previous studies have been performed to determine the degree of interobserver and intraobserver variability in such determinations. We measured the interobserver and intraobserver error in sonographic measurements of renal length obtained independently by three experienced imagers in 21 children (41 kidneys). The mean interobserver variation between any two imagers ranged from 3.87 to 5.49 mm. The mean intraobserver variation was 0.87 to 3.61 mm. The observed variability in sonographic measurement of renal length is comparable to the expected annual increase in length of the kidneys during childhood (2.2-5.7 mm per year). Therefore, caution is suggested when using sonography to evaluate renal growth in children during a year's time.

Variability and reproducibility of quantitative left ventricular angiography.

To evaluate the reproducibility of left ventricular angiography for the assessment of left ventricular (LV) function and regional wall motion, two ventriculographies were performed in the 30 degrees right anterior oblique (RAO) projection, at 15-minute intervals, in 19 patients undergoing coronary angiography. Heart rate, left ventricular systolic pressure, and end-diastolic pressure were measured 15 minutes after the first angiography returned to the baseline values (71.0 +/- 14.1 vs. 72.2 +/- 15.5 beats/minute, 153.6 +/- 18.0 vs. 152.8 +/- 19.9 mm Hg, 21.7 +/- 8.6 vs. 20.9 +/- 7.3 mm Hg, respectively). Global and regional LV performance was analyzed by two observers with a computer-assisted technique. Intraobserver mean variation of end-diastolic volume and ejection fraction was less than 3% of the control value. Interobserver mean variations for the same parameters were less than 4% of control values. For both observers, there was no significant variation of LV end-diastolic volume and ejection fraction from one study to the other. Under stable hemodynamic conditions, the mean observed variations were, depending on the observer, 5-6% of the control value for LV end-diastolic volume and 5% for ejection fraction. Analysis of segmental wall motion was also highly reproducible. The mean intraobserver variation (% of control value) of wall motion ranged from 4.4% to 9.2%, depending on the sectors studied. The mean interobserver variation, whatever the sector, ranged from 6.9% to 13.5%.(ABSTRACT TRUNCATED AT 250 WORDS)