Gold Standard Interpretation Research Articles

Identification of treatment-warranted retinopathy of prematurity by neonatal nurse specialist

ObjectiveTo determine the accuracy in the identification of infants with treatment-warranted retinopathy of prematurity (ROP) by a trained and experienced ROP neonatal nurse specialist compared with skilled ophthalmologists.MethodsA single-centre, prospective,...

Machine learning for endoleak detection after endovascular aortic repair

Diagnosis of endoleak following endovascular aortic repair (EVAR) relies on manual review of multi-slice CT angiography (CTA) by physicians which is a tedious and time-consuming process that is susceptible to error. We evaluate the use of a deep neural network for the detection of endoleak on CTA for post-EVAR patients using a novel data efficient training approach. 50 CTAs and 20 CTAs with and without endoleak respectively were identified based on gold standard interpretation by a cardiovascular subspecialty radiologist. The Endoleak Augmentor, a custom designed augmentation method, provided robust training for the machine learning (ML) model. Predicted segmentation maps underwent post-processing to determine the presence of endoleak. The model was tested against 3 blinded general radiologists and 1 blinded subspecialist using a held-out subset (10 positive endoleak CTAs, 10 control CTAs). Model accuracy, precision and recall for endoleak diagnosis were 95%, 90% and 100% relative to reference subspecialist interpretation (AUC = 0.99). Accuracy, precision and recall was 70/70/70% for generalist1, 50/50/90% for generalist2, and 90/83/100% for generalist3. The blinded subspecialist had concordant interpretations for all test cases compared with the reference. In conclusion, our ML-based approach has similar performance for endoleak diagnosis relative to subspecialists and superior performance compared with generalists.

A comprehensive artificial intelligence–enabled electrocardiogram interpretation program

BackgroundAutomated computerized electrocardiogram (ECG) interpretation algorithms are designed to enhance physician ECG interpretation, minimize medical error, and expedite clinical workflow. However, the performance of current computer algorithms is notoriously inconsistent. We aimed to develop and validate an artificial intelligence–enabled ECG (AI-ECG) algorithm capable of comprehensive 12-lead ECG interpretation with accuracy comparable to practicing cardiologists.MethodsWe developed an AI-ECG algorithm using a convolutional neural network as a multilabel classifier capable of assessing 66 discrete, structured diagnostic ECG codes using the cardiologist’s final annotation as the gold-standard interpretation. We included 2,499,522 ECGs from 720,978 patients ≥18 years of age with a standard 12-lead ECG obtained at the Mayo Clinic ECG laboratory between 1993 and 2017. The total sample was randomly divided into training (n = 1,749,654), validation (n = 249,951), and testing (n = 499,917) datasets with a similar distribution of codes. We compared the AI-ECG algorithm’s performance to the cardiologist’s interpretation in the testing dataset using receiver operating characteristic (ROC) and precision recall (PR) curves.ResultsThe model performed well for various rhythm, conduction, ischemia, waveform morphology, and secondary diagnoses codes with an area under the ROC curve of ≥0.98 for 62 of the 66 codes. PR metrics were used to assess model performance accounting for category imbalance and demonstrated a sensitivity ≥95% for all codes.ConclusionsAn AI-ECG algorithm demonstrates high diagnostic performance in comparison to reference cardiologist interpretation of a standard 12-lead ECG. The use of AI-ECG reading tools may permit scalability as ECG acquisition becomes more ubiquitous.

The learning curve for interpretation of oesophageal high‐resolution manometry: a prospective interventional cohort study

High-resolution manometry has become the preferred choice of oesophagologists for oesophageal motor assessment, but the learning curve among trainees remains unclear. To determine the learning curve of high-resolution manometry interpretation. A prospective interventional cohort study was performed on 18 gastroenterology trainees, naïve to high-resolution manometry (median age 32 ± 4.0 years, 44.4% female). An intake questionnaire and a 1-h standardised didactic session were performed at baseline. Multiple 1-h interpretation sessions were then conducted periodically over 15 months where 10 studies were discussed; 5 additional test studies were provided for interpretation, and results were compared to gold standard interpretation by the senior author. Hypothetical management decisions based on trainee interpretation were separately queried. Accuracy was compared across test interpretations and sessions to determine the learning curve, with a goal of 90% accuracy. Baseline accuracy was low for abnormal body motor patterns (53.3%), but higher for achalasia/outflow obstruction (65.9%). Recognition of achalasia reached 90% accuracy after six sessions (P = 0.01), while overall accurate management decisions reached this threshold by the 4th session (P < 0.001). Based on our data, the threshold of 90% accuracy for recognition of any abnormal from normal pattern was reached after 30 studies (3rd session) but fluctuated. Diagnosis of oesophageal body motor patterns remained suboptimal; accuracy of advisability of fundoplication improved, but did not reach 90%. High-resolution manometry has a steep learning curve among trainees. Achalasia recognition is achieved early, but diagnosis of other abnormal motor patterns and management decisions require further supervised training.

Can emergency physicians accurately rule out clinically important cervical spine injuries by using computed tomography?

Emergency physicians are expected to rule out clinically important cervical spine injuries using clinical skills and imaging. Our objective was to determine whether emergency physicians could accurately rule out clinically important cervical spine injuries using computed tomographic (CT) imaging of the cervical spine. Fifteen emergency physicians were enrolled to interpret a sample of 50 cervical spine CT scans in a nonclinical setting. The sample contained a 30% incidence of cervical spine injury. After a 2-hour review session, the participants interpreted the CT scans and categorized them into either a suspected cervical spine injury or no cervical spine injury. Participants were asked to specify the location and type of injury. The gold standard interpretation was the combined opinion of two staff radiologists. Emergency physicians correctly identified 182 of the 210 abnormal cases with cervical spine injury. The sensitivity of emergency physicians was 87% (95% confidence interval [CI] 82-91), and the specificity was 76% (95% CI 74-77). The negative likelihood ratio was 0.18 (95% CI 0.12-0.25). Experienced emergency physicians successfully identified a large proportion of cervical spine injuries on CT; however, they were not sufficiently sensitive to accurately exclude clinically important injuries. Emergency physicians should rely on a radiologist review of cervical spine CT scans prior to discontinuing cervical spine precautions.

Diagnosing pulmonary edema

Diagnosing the underlying cause of acute dyspnea can be challenging. Lung ultrasound may help to identify pulmonary edema as a possible cause. To evaluate the ability of residents to recognize pulmonary edema on lung ultrasound using chest radiographs as a comparison standard. This is a prospective, blinded, observational study of a convenience sample of resident physicians in the Departments of Emergency Medicine (EM), Internal Medicine (IM), and Radiology. Residents were given a tutorial on interpreting pulmonary edema on both chest radiograph and lung ultrasound. They were then shown both ultrasounds and chest radiographs from 20 patients who had presented to the emergency department with dyspnea, 10 with a primary diagnosis of pulmonary edema, and 10 with alternative diagnoses. Cohen's κ values were calculated to describe the strength of the correlation between resident and gold standard interpretations. Participants included 20 EM, 20 IM, and 20 Radiology residents. The overall agreement with gold standard interpretation of pulmonary edema on lung ultrasound (74%, κ = 0.51, 95% confidence interval 0.46-0.55) was superior to chest radiographs (58%, κ = 0.25, 95% confidence interval 0.20-0.30) (P < 0.0001). EM residents interpreted lung ultrasounds more accurately than IM residents. Radiology residents interpreted chest radiographs more accurately than did EM and IM residents. Residents were able to more accurately identify pulmonary edema with lung ultrasound than with chest radiograph. Physicians with minimal exposure to lung ultrasound may be able to correctly recognize pulmonary edema on lung ultrasound.

Establishing a Gold Standard for Test Sets: Variation in Interpretive Agreement of Expert Mammographers

Test sets for assessing and improving radiologic image interpretation have been used for decades and typically evaluate performance relative to gold standard interpretations by experts. To assess test sets for screening mammography, a gold standard for whether a woman should be recalled for additional workup is needed, given that interval cancers may be occult on mammography and some findings ultimately determined to be benign require additional imaging to determine if biopsy is warranted. Using experts to set a gold standard assumes little variation occurs in their interpretations, but this has not been explicitly studied in mammography. Using digitized films from 314 screening mammography exams (n = 143 cancer cases) performed in the Breast Cancer Surveillance Consortium, we evaluated interpretive agreement among three expert radiologists who independently assessed whether each examination should be recalled, and the lesion location, finding type (mass, calcification, asymmetric density, or architectural distortion), and interpretive difficulty in the recalled images. Agreement among the three expert pairs for recall/no recall was higher for cancer cases (mean 74.3±6.5) than for noncancers (mean 62.6±7.1). Complete agreement on recall, lesion location, finding type and difficulty ranged from 36.4% to 42.0% for cancer cases and from 43.9% to 65.6% for noncancer cases. Two of three experts agreed on recall and lesion location for 95.1% of cancer cases and 91.8% of noncancer cases, but all three experts agreed on only 55.2% of cancer cases and 42.1% of noncancer cases. Variability in expert interpretive is notable. A minimum of three independent experts combined with a consensus should be used for establishing any gold standard interpretation for test sets, especially for noncancer cases.

A decision support system for stress only myocardial perfusion scintigraphy may save unnecessary rest studies

Aim: It is often a practical question whether to continue with the rest study after termination of a fairly normal stress myocardial perfusion scintigraphy (MPS), in particular for physicians with limited experience. The purpose of this study was to analyze the value of using a decision support system (DSS) to guide less experienced physicians in this situation. Methods: Nine residents from eight different nuclear medicine departments interpreted 100 MPS stress studies, first without and then with access to the advice of a DSS. Each study was interpreted regarding the necessity of adding a rest study for correct interpretation of the MPS. The patients had undergone a gated stress and rest MPS, using a Tc-99m sestamibi protocol. Interpretations made by three nuclear cardiology experts, having access to all available clinical and image information, were used as the gold standard. Results: In the cases where the gold standard interpretation wanted a rest study the 9 residents asked for it in 94% and 95% before and after having access to the DSS, respectively ( p &gt;0.05). The residents did not want a rest study in 57% (without) and 69% (with the advice from the decision support system), in the patients, considered to have a normal stress study by the experts ( p &lt;0.005). The DSS significantly reduced interobserver variation among the residents. Conclusion: The present study shows that with the support of a DSS less experienced physicians get closer to the decisions of highly experienced nuclear cardiologists regarding the need of adding a rest study to a stress MPS.

Predicting a single HIV drug resistance measure from three international interpretation gold standards

ObjectiveTo investigate the possibility of combining the interpretation of three gold standard interpretation algorithms using weighted heuristics in order to produce a single resistance measure. MethodsThe outputs of HIVdb, Rega, ANRS were combined to obtain a single resistance profile using the equally weighted voting algorithm, accuracy based weighing voting algorithm and the Bayesian based weighted voting algorithm techniques. ResultsThe Bayesian based voting combination increased the accuracy of the resistance profile prediction compared to phenotype, from 58% to 69%. The equal weighted voting algorithm and the accuracy based algorithm both increased the prediction accuracy to 60%. ConclusionFrom the result obtained it is evident that combining the gold standard interpretation algorithms may increase the predictive ability of the individual interpretation algorithms.

What is the effect of reporting all emergency department radiographs?

Objectives: To evaluate the effect of formal radiological reporting of all emergency department (ED) radiographs on clinical practice and patient outcome, and to consider whether a selective reporting policy might...

Validity and Reliability of the Thrombolysis in Brain Infarction (TIBI) Flow Grades

48 Background: In patients eligible for thrombolytic therapy, the TIBI grading system defines residual transcranial Doppler (TCD) flow signals at the site of clot location. In addition, TIBI flow grades correlate with angiography and predict short-term improvement following TPA therapy. The aim of this study was to develop a valid and reliable instrument to assess how accurately health professionals determine TIBI flow grades. Methods: Two expert sonographers developed a 60 question computerized examination of the 6 TIBI flow grades. Gold standard interpretation was determined by a consensus interpretation by a pool of expert sonographers. All items were equally weighted and written in a standard format. A computerized TIBI grading tutorial and standard interpretation rules were available for all test-takers. To assess content validity and reliability, a blinded panel of 3 experts independently took the examination. Inter-rater agreement was determined among experienced sonographers, stroke neurologists, and other health professionals (N=11). Results: Expert assessment of each test item measured objective congruence at 1.0, overall test validity coefficient of 0.996 (95%CI 0.993–0.997), and an item/TIBI flow grade validity coefficient of 0.992 (95%CI 0.996–0.997). Overall test and individual item reliability had coefficients of 0.98(95%CI 0.97–0.99) and 0.97 (95%CI 0.95–0.98, p Conclusions: Our computerized examination is a valid and reliable tool to assess an individuals ability to score TIBI flow grades. In addition, TIBI flow grade evaluation has a high degree of inter-rater reliability, even among those with limited experience in ultrasound.

Validity and Reliability of the Thrombolysis in Brain Infarction (TIBI) Flow Grades

48 Background: In patients eligible for thrombolytic therapy, the TIBI grading system defines residual transcranial Doppler (TCD) flow signals at the site of clot location. In addition, TIBI flow grades correlate with angiography and predict short-term improvement following TPA therapy. The aim of this study was to develop a valid and reliable instrument to assess how accurately health professionals determine TIBI flow grades. Methods: Two expert sonographers developed a 60 question computerized examination of the 6 TIBI flow grades. Gold standard interpretation was determined by a consensus interpretation by a pool of expert sonographers. All items were equally weighted and written in a standard format. A computerized TIBI grading tutorial and standard interpretation rules were available for all test-takers. To assess content validity and reliability, a blinded panel of 3 experts independently took the examination. Inter-rater agreement was determined among experienced sonographers, stroke neurologists, and other health professionals (N=11). Results: Expert assessment of each test item measured objective congruence at 1.0, overall test validity coefficient of 0.996 (95%CI 0.993–0.997), and an item/TIBI flow grade validity coefficient of 0.992 (95%CI 0.996–0.997). Overall test and individual item reliability had coefficients of 0.98(95%CI 0.97–0.99) and 0.97 (95%CI 0.95–0.98, p&lt;0.0001) respectively. Inter-rater agreement was 0.89 (coefficient of contingency) and 0.86 (kappa) for all participants. Kappa range included 0.98 for an experienced interpreter, 0.81–0.89 for stroke neurologists with beginner experience in TCD and 0.65 for those without previous TCD experience (66% of correct answers after 1 hour tutorial). Conclusions: Our computerized examination is a valid and reliable tool to assess an individuals ability to score TIBI flow grades. In addition, TIBI flow grade evaluation has a high degree of inter-rater reliability, even among those with limited experience in ultrasound.

Video-capture teleradiology for the after-hours reading of computerized tomography scans.

We evaluated the after-hours support that radiologists could provide from their homes to an urban hospital for reading emergency computerized tomography (CT) scans. During a six-month study period, 36 CT scans were acquired using a video-capture card in a PC and transmitted via an ordinary telephone line. The teleradiology interpretation and the formal report (from the hard copy) were compared with a reference or gold standard interpretation made by a radiologist and an emergency physician for 31 of the 36 cases. In comparison with the gold standard, there were 26 correct diagnoses by teleradiology (84%) and five incorrect (16%). Head CT scans accounted for 74% of the transmitted scans and all five of the cases with discrepant diagnoses. The cause of the single clinically relevant discrepancy was found to be unrelated to the use of teleradiology. In comparison with the formal report, the accuracy of the teleradiologist was not significantly different for all categories combined or for head CT scans alone. Video-capture technology provided a reliable means of transmitting CT scans for after-hours interpretation in emergency cases.

GIBSON'S PARADOX AND POLICY REGIMES: A COMPARISON OF THE EXPERIENCE IN THE US, UK AND ITALY

AbstractThis paper compares the behaviour of long‐term interest rates and prices in Italy, the UK and the USA, and seeks to shed light into what has become known as the ‘Gibson Paradox’. We compare the various theoretical explanations for the observed positive correlation of interest rates and prices in the USA and the UK. Using both regression and frequency domain techniques, we demonstrate that there is little evidence for the occurrence of the paradox in the case of Italy. The key conclusion of the paper is that the comparative evidence from these three countries supports a gold standard interpretation of the Paradox.

An assessment of the radiological module of NEONATE as an aid in interpreting chest X-ray findings by nonradiologists

NEONATE is a prototype of an expert system for the Newborn Intensive Care Unit developed at the Primary Children's Medical Center in Salt Lake City, Utah. A pilot study was undertaken to see if the addition of radiological frames to the NEONATE software could aid attending Neonatologists to interpret chest X-ray films. A set of radiological frames was created from rules generated by a radiologist. The performance of these radiological frames was compared to the performance of other radiologists using the Kappa statistic to measure agreement. There is a good agreement between the computer's decisions and the radiologists' decisions. The radiological frames were also tested to see if they help physicians who are not trained in radiology. A system that compares the residents' interpretation and the computer's interpretation to a gold standard interpretation was developed. It shows that the computer helps the first and second year residents, but not the third year residents. This article suggests that NEONATE's interpretation of chest X-ray findings are close to the radiologists' interpretations. While NEONATE's radiological frames help novice physicians in reaching better chest X-ray interpretation, the current study suggests that they are not likely to help a Neonatologist.