Radiology Resident Performance Research Articles

Training on contrast-enhanced ultrasound LI-RADS classification for resident radiologists: a retrospective comparison of performance after training

ObjectivesTo evaluate the effects and benefits of training radiology residents on contrast-enhanced ultrasound (CEUS) according to the Liver Imaging Reporting and Data System (LI-RADS).MethodsIn total, 234 patients at high risk of hepatocellular carcinoma (HCC) who underwent CEUS were enrolled, including 27 lesions in the education set and 207 lesions in the test sets (a–d). Forty-five radiology residents and 4 radiology experts involved in CEUS LI-RADS training individually reviewed the test sets before, immediately after, and 3-months after training. The consistency with kappa values of the description of CEUS features, the classification of focal liver lesions (FLLs), and the diagnostic performance were evaluated.ResultsThe level of agreement between the radiology experts and residents improved after training (all p < 0.05), while there were no significant differences between the post-training and 3-months post-training results (all p > 0.05). The sensitivity, specificity, positive predictive value, and area under the curve (AUC) based on the CEUS LI-RADS classification of the radiology experts in the diagnosis of HCC were 62.9%, 96.4%, 96.3%, and 0.796, respectively. The diagnostic performance of the radiology residents significantly improved after training (all p < 0.05). Misunderstanding of definitions and subjective interpretation of images were the main reasons for disagreement with multiple responses.ConclusionDedicated CEUS LI-RADS training improved the performance of radiology residents in diagnosing FLLs and their agreement with radiology experts on CEUS features. Images and videos to explain typical features of the training were essential to improve agreement between the radiology experts and residents.Critical relevance statementAgreement on lesion descriptors between radiology experts and residents can improve with training.Key PointsThe diagnostic performance of less experienced radiologists for diagnosing HCC could be improved by training.Images and videos to explain typical features during training were essential.Agreement on lesion descriptors between radiology experts and residents improved after training.Graphical

Improving the diagnostic performance of inexperienced readers for thyroid nodules through digital self-learning and artificial intelligence assistance.

Data-driven digital learning could improve the diagnostic performance of novice students for thyroid nodules. To evaluate the efficacy of digital self-learning and artificial intelligence-based computer-assisted diagnosis (AI-CAD) for inexperienced readers to diagnose thyroid nodules. Between February and August 2023, a total of 26 readers (less than 1 year of experience in thyroid US from various departments) from 6 hospitals participated in this study. Readers completed an online learning session comprising 3,000 thyroid nodules annotated as benign or malignant independently. They were asked to assess a test set consisting of 120 thyroid nodules with known surgical pathology before and after a learning session. Then, they referred to AI-CAD and made their final decisions on the thyroid nodules. Diagnostic performances before and after self-training and with AI-CAD assistance were evaluated and compared between radiology residents and readers from different specialties. AUC (area under the receiver operating characteristic curve) improved after the self-learning session, and it improved further after radiologists referred to AI-CAD (0.679 vs 0.713 vs 0.758, p<0.05). Although the 18 radiology residents showed improved AUC (0.7 to 0.743, p=0.016) and accuracy (69.9% to 74.2%, p=0.013) after self-learning, the readers from other departments did not. With AI-CAD assistance, sensitivity (radiology 70.3% to 74.9%, others 67.9% to 82.3%, all p<0.05) and accuracy (radiology 74.2% to 77.1%, others 64.4% to 72.8%, all p <0.05) improved in all readers. While AI-CAD assistance helps improve the diagnostic performance of all inexperienced readers for thyroid nodules, self-learning was only effective for radiology residents with more background knowledge of ultrasonography. Online self-learning, along with AI-CAD assistance, can effectively enhance the diagnostic performance of radiology residents in thyroid cancer.

Deep Learning to Differentiate Benign and Malignant Vertebral Fractures at Multidetector CT.

Background Differentiating between benign and malignant vertebral fractures poses diagnostic challenges. Purpose To investigate the reliability of CT-based deep learning models to differentiate between benign and malignant vertebral fractures. Materials and Methods CT scans acquired in patients with benign or malignant vertebral fractures from June 2005 to December 2022 at two university hospitals were retrospectively identified based on a composite reference standard that included histopathologic and radiologic information. An internal test set was randomly selected, and an external test set was obtained from an additional hospital. Models used a three-dimensional U-Net encoder-classifier architecture and applied data augmentation during training. Performance was evaluated using the area under the receiver operating characteristic curve (AUC) and compared with that of two residents and one fellowship-trained radiologist using the DeLong test. Results The training set included 381 patients (mean age, 69.9 years ± 11.4 [SD]; 193 male) with 1307 vertebrae (378 benign fractures, 447 malignant fractures, 482 malignant lesions). Internal and external test sets included 86 (mean age, 66.9 years ± 12; 45 male) and 65 (mean age, 68.8 years ± 12.5; 39 female) patients, respectively. The better-performing model of two training approaches achieved AUCs of 0.85 (95% CI: 0.77, 0.92) in the internal and 0.75 (95% CI: 0.64, 0.85) in the external test sets. Including an uncertainty category further improved performance to AUCs of 0.91 (95% CI: 0.83, 0.97) in the internal test set and 0.76 (95% CI: 0.64, 0.88) in the external test set. The AUC values of residents were lower than that of the best-performing model in the internal test set (AUC, 0.69 [95% CI: 0.59, 0.78] and 0.71 [95% CI: 0.61, 0.80]) and external test set (AUC, 0.70 [95% CI: 0.58, 0.80] and 0.71 [95% CI: 0.60, 0.82]), with significant differences only for the internal test set (P < .001). The AUCs of the fellowship-trained radiologist were similar to those of the best-performing model (internal test set, 0.86 [95% CI: 0.78, 0.93; P = .39]; external test set, 0.71 [95% CI: 0.60, 0.82; P = .46]). Conclusion Developed models showed a high discriminatory power to differentiate between benign and malignant vertebral fractures, surpassing or matching the performance of radiology residents and matching that of a fellowship-trained radiologist. © RSNA, 2024 See also the editorial by Booz and D'Angelo in this issue.

A Visual Aid Tool for Detection of Pancreatic Tumour-Vessel Contact on Staging CT: A Retrospective Cohort Study.

Purpose: In pancreatic adenocarcinoma, the difficult distinction between normal and affected pancreas on CT studies may lead to discordance between the pre-surgical assessment of vessel involvement and intraoperative findings. We hypothesize that a visual aid tool could improve the performance of radiology residents when detecting vascular invasion in pancreatic adenocarcinoma patients. Methods: This study consisted of 94 pancreatic adenocarcinoma patient CTs. The visual aid compared the estimated body fat density of each patient with the densities surrounding the superior mesenteric artery and mapped them onto the CT scan. Four radiology residents annotated the locations of perceived vascular invasion on each scan with the visual aid overlaid on alternating scans. Using 3 expert radiologists as the reference standard, we quantified the area under the receiver operating characteristic curve to determine the performance of the tool. We then used sensitivity, specificity, balanced accuracy ((sensitivity + specificity)/2), and spatial metrics to determine the performance of the residents with and without the tool. Results: The mean area under the curve was 0.80. Radiology residents' sensitivity/specificity/balanced accuracy for predicting vascular invasion were 50%/85%/68% without the tool and 81%/79%/80% with it compared to expert radiologists, and 58%/85%/72% without the tool and 78%/77%/77% with it compared to the surgical pathology. The tool was not found to impact the spatial metrics calculated on the resident annotations of vascular invasion. Conclusion: The improvements provided by the visual aid were predominantly reflected by increased sensitivity and accuracy, indicating the potential of this tool as a learning aid for trainees.

Impact of deep learning on radiologists and radiology residents in detecting breast cancer on CT: a cross-vendor test study

To investigate the effect of deep learning on the diagnostic performance of radiologists and radiology residents in detecting breast cancers on computed tomography (CT). In this retrospective study, patients undergoing contrast-enhanced chest CT between January 2010 and December 2020 using equipment from two vendors were included. Patients with confirmed breast cancer were categorised as the training (n=201) and validation (n=26) group and the testing group (n=30) using processed CT images from either vendor. The trained deep-learning model was applied to test group patients with (30 females; mean age=59.2±15.8 years) and without (19 males, 21 females; mean age=64±15.9 years) breast cancer. Image-based diagnostic performance of the deep-learning model was evaluated with the area under the receiver operating characteristic curve (AUC). Two radiologists and three radiology residents were asked to detect malignant lesions by recording a four-point diagnostic confidence score before and after referring to the result from the deep-learning model, and their diagnostic performance was evaluated using jackknife alternative free-response receiver operating characteristic analysis by calculating the figure of merit (FOM). The AUCs of the trained deep-learning model on the validation and test data were 0.976 and 0.967, respectively. After referencing with the result of the deep learning model, the FOMs of readers significantly improved (reader 1/2/3/4/5: from 0.933/0.962/0.883/0.944/0.867 to 0.958/0.968/0.917/0.947/0.900; p=0.038). Deep learning can help radiologists and radiology residents detect breast cancer on CT.

The moderating role of resilience in the association between workload and depressive symptoms among radiology residents in China: results from a nationwide cross-sectional study.

The current study aimed to explore the moderating role of psychological resilience in the association between workload and depressive symptoms among radiology residents during standardized residency training (SRT) in China. A nationwide cross-sectional online survey was conducted among radiology residents in China. Workload was measured by working hours per week and the frequency of frontline nightwork in the last month. Resilience was assessed by the 2-item Connor-Davidson Resilience Scale. Depressive symptoms were measured by the Depression Anxiety Stress Scales. The hierarchical regression and simple slope analyses were performed to examine the moderating effect of resilience. Among 3666 radiology residents, the mean age was 27.3years (SD = 2.6) and 58% were female. About 24.4% of the participants reported medium to severe depressive symptoms. The hierarchical regression showed that working hours (ba = 0.11, 95%CI: 0.08, 0.14) and having frontline nightwork more than once (ba = 1.22, 95%CI: 0.67, 1.78) were positively associated with depressive symptoms; the moderating effect of resilience was significant in the association of depressive symptoms with working hours (ba = - 0.02, 95%CI: - 0.03, - 0.01) and having frontline nightwork more than once (ba = - 0.28, 95%CI: - 0.49, - 0.07). The simple slope test showed the association between workload-related variables and depressive symptoms was only significant in those with a relatively lower level of resilience. The study found that resilience was an important modifier buffering the positive association between workload and depressive symptoms among radiology residents in China. Future medical training programs are suggested to include effective intervention components to increase personal resilience. Heavy workload in clinical setting may pose adverse effect on mental health and job performance of radiology residents. The study investigated whether psychological resilience would mitigate the association between workload and depressive symptoms among Chinese radiology residents. • Radiology residents with a heavier workload presented a higher level of depressive symptoms in China. • Psychological resilience mitigated the positive association between workload and depressive symptoms. • The association between workload and depressive symptoms was only statistically significant in radiology residents with a relatively lower level of resilience.

Structured report improves radiology residents' performance in reporting chest high-resolution computed tomography: a study in patients with connective tissue disease.

PURPOSE To evaluate the performance of radiology residents (RRs) when using a dedicated structured report (SR) template for chest HRCT in patients with suspected connective tissue disease-interstitial lung disease (CTD-ILD), compared to the traditional narrative report (NR). METHODS We retrospectively evaluated 50 HRCT exams in patients with suspected CTD-ILD. A chest-devoted radiologist reported all the HRCT exams as the reference standard, pointing out pulmonary fibrosis findings (i.e., honeycombing, traction bronchiectasis, reticulation, and volume loss), presence and pattern of ILD, and possible other diagnoses. We divided four RRs into two groups according to their expertise level. In each group, RRs reported all HRCT examinations alternatively with NR or SR, noting each report's reporting time. The Cohen's Kappa, Wilcoxon, and McNemar tests were used for statistical analysis. RESULTS Regarding the pulmonary fibrosis findings, we found higher agreement between RRs and the reference standard reader when using SR than NR, regardless of their expertise level, except for volume loss.RRs' accuracy for "other diagnosis" was higher when using SR than NR, moving from 0.48 to 0.66 in the novel group (p = 0.035) and from 0.44 to 0.80 in the expertise group (p < 0.001). No differences in accuracy were found between ILD presence and ILD pattern. The reporting time was significantly lower (p = 0.001) when using SR than NR. CONCLUSION SR is of value in increasing the reporting of critical chest HRCT findings in the complex CTD-ILD scenario and should be used early and systematically during the residency.

Interpretation of computed tomography of the cervical spine by non-radiologists: a systematic review and meta-analysis.

After trauma, clearance of the cervical spine refers to the exclusion of underlying serious injuries. Accurate assessment of computed tomography (CT) is commonly required prior to clearance of the cervical spine. Delays to clearance can lead to prolonged immobilization with associated patient discomfort and adverse effects. This systematic review aimed to determine performance of non-radiologists to evaluate cervical spine CT. MEDLINE, EMBASE, Cochrane library with sources of grey literature and reference lists of selected articles were appraised from inception to April 2021. We included manuscripts that reported discordance in CT cervical spine interpretation between non-radiologists and radiologists. The Newcastle-Ottawa scale (NOS) was used to assess quality of included studies and statistical heterogeneity was assessed using the I2 statistic. There were 43 studies identified for eligibility and 4 manuscripts included in the final analysis. There were two studies that reported on the performance of radiology residents, one study on the performance of surgical residents and one on emergency physicians. The pooled discordance was 0.25 (95%CI 0.21-0.28) but was lower for radiology residents (range 0.007-0.05). There was significant statistical heterogeneity (I2 = 99.6%, P< 0.001) among studies. There is a paucity of evidence documenting the ability of non-radiologists in accurately interpreting CT of the cervical spine. A number of discordant findings suggest that studies with larger sample sizes are indicated to accurately ascertain the ability of non-radiologists in this area.

Resident-attending discrepancy rates for two consecutive versus nonconsecutive weeks of overnight shifts.

Recent Accreditation Council for Graduate Medical Education policy changes no longer limit the number of consecutive night shifts allowed for trainees. Few studies have examined radiology resident overnight performance over time. This study aimed to compare significant resident-attending discrepancy rates for residents working 2 nonconsecutive versus consecutive weeks of overnight shifts. The authors hypothesized significantly increased week-two discrepancies in the consecutive group. For 2020, a retrospective analysis of significant overnight resident-attending discrepancy rates over a 24-week period using database searches was performed for residents self-selecting 2 nonconsecutive versus consecutive weeks. The nonconsecutive group typically had a 7-day mix of days off and day shifts between their night shift weeks. Paired and unpaired t tests were performed with p < 0.05 considered significant. For the 24 sets of 2weeks covered by two residents at a time, eight were nonconsecutive and 16 were consecutive. The nonconsecutive group had 75.0% R4 coverage compared to 37.5% for the consecutive group. There were no significant study volume differences between the groups. A total of 27,906 studies (35.3% cross-sectional [CT and MR], 54.9% radiograph plus fluoroscopy, 9.8% US) were performed with 223 discrepancies (0.80%). Overall discrepancies for the nonconsecutive versus consecutive groups were 39/4505 (0.87%) versus 59/9462 (0.62%; p = 0.32) for week one and 46/4732 (1.0%) versus 79/9207 (0.86%; p = 0.60) for week two with no significant differences between the groups by modality. Residents self-selecting 2 consecutive weeks of overnight shifts do not have increased resident-attending discrepancy rates compared to 2 nonconsecutive weeks.

Evaluation of neuroradiology emergency MRI interpretations: low discrepancy rates between on-call radiology residents' preliminary interpretations and neuroradiologists' final reports.

To evaluate the performance of on-call radiology residents in interpreting alone brain and spine MRI studies performed after hours, to describe their mistakes, and to identify influencing factors that increased the occurrence of errors. A total of 328 MRI examinations performed during a 13-month period (from December 1, 2019, to January 1, 2021) were prospectively included. Discrepancies between the preliminary interpretation of on-call radiology residents and the final reports of attending neuroradiologists were noted and classified according to a three-level score: level 1 (perfect interpretation or minor correction), level 2 (important correction without immediate change in patient management), or level 3 (major correction with immediate change in patient management). Categorical data were compared using Fisher's exact test. The overall discrepancy rate (level-2 and level-3 errors) was 16%; the rate of major discrepancies (only level-3 errors) was 5.5%. The major-discrepancy rate of second-year residents, when compared with that of senior residents, was significantly higher (p = 0.02). Almost all of the level-3 errors concerned cerebrovascular pathology. The most common level-2 errors involved undescribed aneurysms. We found no significant difference in the major-discrepancy rate regarding time since the beginning of the shift. The great majority of examinations were correctly interpreted. The rate of major discrepancies in our study was comparable to the data in the literature, and there was no adverse clinical outcome. The level of residency has an effect on the rate of serious errors in residents' reports. • The rate of major discrepancies between preliminary MRI interpretations by on-call radiology residents and final reports by attending neuroradiologists is low, and comparable to discrepancy rates reported for head CT interpretations. • The youngest residents made significantly more serious errors when compared to senior residents. • There was no adverse clinical outcome in patient morbidity as a result of an initial misdiagnosis.

Diagnostic Performance of Radiology Residents in Thoracic CT Imaging in Emergency Radiology During The COVID-19 Pandemic

During the COVID-19 pandemic, the workload of all radiologists and radiology residents, as well as other clinical physicians, has increased. This study aims to determine the diagnostic performance of radiology residents, who effectively contribute to the diagnosis of COVID-19. The thoracic Computed Tomography (CT) images of 135 patients aged 20-83 diagnosed with COVID-19 were evaluated retrospectively by five radiology residents and a radiologist with 10 years of experience. The diagnostic performance of the radiology residents in evaluating COVID-19 was assessed according to their year of residency and the patients' age and gender. Receiver Operating Characteristic (ROC) curve analysis was performed to determine the sensitivity and specificity of radiology residents. The radiology residents' performance in determining COVID-19 using CT findings was evaluated as follows: sensitivity 97.22%, specificity 88.89%, positive predictive value 90.91%, negative predictive value 96.55%, and accuracy 93.33%. According to the year of residency, the sensitivity and specificity of the radiology residents in determining COVID-19 using CT images were between 92.3% and 100%, and 71.43% and 100%, respectively. The high sensitivity and specificity of radiology residents in evaluating thoracic CT images for COVID-19 diagnosis indicate that radiologists are as important as clinical physicians in the diagnosis of COVID-19.

Radiology resident selection factors predict resident performance

PurposeTo determine selection factors that predict radiology resident performance. Methods59 consecutive radiology residents from 2002 to 2015 were ranked on performance during residency. Correlations and multiple regression analyses were performed to predict resident performance from the following selection factors: United States Medical Licensing Exam (USMLE) Step 1 score, medical school rank, Alpha Omega Alpha (AOA) membership, honors in clinical rotations, Medical Student Performance Evaluation (MSPE), and interview score. Results were compared against predictions from Match rank position. ResultsFive selection factors showed significant or marginally significant correlations with resident performance (r = 0.2 to 0.3). The interview score was not significantly correlated. A multiple regression model comprised of the USMLE Step 1 score, medical school rank, AOA membership, and interview score predicted resident performance, with an adjusted R2 of 0.19. The interview score was included in the model but did not achieve statistical significance. Match rank did not predict resident performance, with an R2 of 0.01. ConclusionsA multiple regression model comprised of the USMLE Step 1 score, medical school rank, and AOA membership predicted radiology resident performance and may assist with resident selection.

Brain MRI Deep Learning and Bayesian Inference System Augments Radiology Resident Performance.

Automated quantitative and probabilistic medical image analysis has the potential to improve the accuracy and efficiency of the radiology workflow. We sought to determine whether AI systems for brain MRI diagnosis could be used as a clinical decision support tool to augment radiologist performance. We utilized previously developed AI systems that combine convolutional neural networks and expert-derived Bayesian networks to distinguish among 50 diagnostic entities on multimodal brain MRIs. We tested whether these systems could augment radiologist performance through an interactive clinical decision support tool known as Adaptive Radiology Interpretation and Education System (ARIES) in 194 test cases. Four radiology residents and three academic neuroradiologists viewed half of the cases unassisted and half with the results of the AI system displayed on ARIES. Diagnostic accuracy of radiologists for top diagnosis (TDx) and top three differential diagnosis (T3DDx) was compared with and without ARIES. Radiology resident performance was significantly better with ARIES for both TDx (55% vs 30%; P < .001) and T3DDx (79% vs 52%; P = 0.002), with the largest improvement for rare diseases (39% increase for T3DDx; P < 0.001). There was no significant difference between attending performance with and without ARIES for TDx (72% vs 69%; P = 0.48) or T3DDx (86% vs 89%; P = 0.39). These findings suggest that a hybrid deep learning and Bayesian inference clinical decision support system has the potential to augment diagnostic accuracy of non-specialists to approach the level of subspecialists for a large array of diseases on brain MRI.

Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents

Chest radiography is the most common diagnostic imaging examination performed in emergency departments (EDs). Augmenting clinicians with automated preliminary read assistants could help expedite their workflows, improve accuracy, and reduce the cost of care. To assess the performance of artificial intelligence (AI) algorithms in realistic radiology workflows by performing an objective comparative evaluation of the preliminary reads of anteroposterior (AP) frontal chest radiographs performed by an AI algorithm and radiology residents. This diagnostic study included a set of 72 findings assembled by clinical experts to constitute a full-fledged preliminary read of AP frontal chest radiographs. A novel deep learning architecture was designed for an AI algorithm to estimate the findings per image. The AI algorithm was trained using a multihospital training data set of 342 126 frontal chest radiographs captured in ED and urgent care settings. The training data were labeled from their associated reports. Image-based F1 score was chosen to optimize the operating point on the receiver operating characteristics (ROC) curve so as to minimize the number of missed findings and overcalls per image read. The performance of the model was compared with that of 5 radiology residents recruited from multiple institutions in the US in an objective study in which a separate data set of 1998 AP frontal chest radiographs was drawn from a hospital source representative of realistic preliminary reads in inpatient and ED settings. A triple consensus with adjudication process was used to derive the ground truth labels for the study data set. The performance of AI algorithm and radiology residents was assessed by comparing their reads with ground truth findings. All studies were conducted through a web-based clinical study application system. The triple consensus data set was collected between February and October 2018. The comparison study was preformed between January and October 2019. Data were analyzed from October to February 2020. After the first round of reviews, further analysis of the data was performed from March to July 2020. The learning performance of the AI algorithm was judged using the conventional ROC curve and the area under the curve (AUC) during training and field testing on the study data set. For the AI algorithm and radiology residents, the individual finding label performance was measured using the conventional measures of label-based sensitivity, specificity, and positive predictive value (PPV). In addition, the agreement with the ground truth on the assignment of findings to images was measured using the pooled κ statistic. The preliminary read performance was recorded for AI algorithm and radiology residents using new measures of mean image-based sensitivity, specificity, and PPV designed for recording the fraction of misses and overcalls on a per image basis. The 1-sided analysis of variance test was used to compare the means of each group (AI algorithm vs radiology residents) using the F distribution, and the null hypothesis was that the groups would have similar means. The trained AI algorithm achieved a mean AUC across labels of 0.807 (weighted mean AUC, 0.841) after training. On the study data set, which had a different prevalence distribution, the mean AUC achieved was 0.772 (weighted mean AUC, 0.865). The interrater agreement with ground truth finding labels for AI algorithm predictions had pooled κ value of 0.544, and the pooled κ for radiology residents was 0.585. For the preliminary read performance, the analysis of variance test was used to compare the distributions of AI algorithm and radiology residents' mean image-based sensitivity, PPV, and specificity. The mean image-based sensitivity for AI algorithm was 0.716 (95% CI, 0.704-0.729) and for radiology residents was 0.720 (95% CI, 0.709-0.732) (P = .66), while the PPV was 0.730 (95% CI, 0.718-0.742) for the AI algorithm and 0.682 (95% CI, 0.670-0.694) for the radiology residents (P < .001), and specificity was 0.980 (95% CI, 0.980-0.981) for the AI algorithm and 0.973 (95% CI, 0.971-0.974) for the radiology residents (P < .001). These findings suggest that it is possible to build AI algorithms that reach and exceed the mean level of performance of third-year radiology residents for full-fledged preliminary read of AP frontal chest radiographs. This diagnostic study also found that while the more complex findings would still benefit from expert overreads, the performance of AI algorithms was associated with the amount of data available for training rather than the level of difficulty of interpretation of the finding. Integrating such AI systems in radiology workflows for preliminary interpretations has the potential to expedite existing radiology workflows and address resource scarcity while improving overall accuracy and reducing the cost of care.

USMLE Step 3 Scores Have Value in Predicting ABR Core Examination Outcome and Performance: A Multi-institutional Study

We analyzed multi-institutional data to determine if Step 3 performance tiers can identify radiology residents with increased risk of Core examination failure and submean performance. We collected Step 3 scores (USMLE Step 3 or COMLEX Level 3) and American Board of Radiology (ABR) Core examination outcomes and scores for anonymized residents from 13 different Diagnostic Radiology residency programs taking the ABR Core examination between 2013 and 2019. Step 3 scores were converted to percentiles based on Z-score, with Core outcome and performance analyzed for Step 3 groups based on 50th percentile and based on quintiles. Core outcome was scored as fail when conditionally passed or failed. Core performance was measured by the percent of residents with scores below the mean. Differences between Step 3 groups for Core outcome and Core performance were statistically evaluated. Data were available for 342 residents. The Core examination failure rate for 121 residents with Step 3 scores <50th percentile was 19.8% (fail relative risk = 2.26), significantly higher than the 2.7% failure rate for the 221 other residents. Of 42 residents with Step 3 scores in the lowest quintile, the Core failure rate increased to 31.0% (fail relative risk = 3.52). Core performance improved with higher Step 3 quintiles. Step 3 licensing scores have value in predicting radiology resident performance on the ABR Core examination, enabling residency programs to target higher risk residents for early assessment and intervention.

Impact of Simulation Training on Radiology Resident Performance in Neonatal Head Ultrasound

The aim of this study was to determine whether resident performance in head ultrasound on neonates improves following brain phantom simulation training. Ten junior radiology residents with at least one year of radiology training were divided into two equal groups. Both groups received a detailed head ultrasound protocol sheet and observed a technologist perform a head ultrasound on a neonatal patient at the beginning of their first pediatric radiology rotation. Both groups of residents also received teaching with a brain phantom model. Group A residents independently performed one head ultrasound exam, subsequently received phantom simulation training, and then performed a post-training head ultrasound exam. Group B residents received phantom simulation training prior to their first head ultrasound exam. Three pediatric radiologists independently and blindly reviewed the ultrasound images of each head ultrasound exam for proficiency of image acquisition using a validated scoring system. Scores of Group A residents prior to phantom training were compared to their scores after phantom training as well as to scores of Group B residents using simple linear regression. There was a statistically significant improvement in the performance of head ultrasound on neonates when comparing the same residents pre- and postphantom training (p = 0.003). Residents who initially trained with the phantom performed significantly better on their first head ultrasound examination on a neonate than those residents who did not (p = 0.005). Our novel head ultrasound phantom training model significantly improves radiology resident performance of head ultrasound on neonates and may, therefore, be beneficial for residency education.

The Relationship Between US Medical Licensing Examination Step Scores and ABR Core Examination Outcome and Performance: A Multi-institutional Study.

We analyzed multi-institutional data to understand the relationship of US Medical Licensing Examination (USMLE) Step scores to ABR Core examination performance to identify Step score tiers that stratify radiology residents into different Core performance groups. We collected USMLE Step scores and ABR Core examination outcomes and scores for anonymized residents from 13 different diagnostic radiology residency programs taking the ABR Core examination between 2013 and 2019. USMLE scores were grouped into noniles using z scores and then aggregated into three tiers based on similar Core examination pass-or-fail outcomes. Core performance was grouped using standard deviation from the mean and then measured by the percent of residents with scores below the mean. Differences between Step tiers for Core outcome and Core performance were statistically evaluated (P < .05 considered significant). Differences in Step 1 terciles Core failure rates (45.9%, 11.9%, and 3.0%, from lowest to highest Step tiers; n= 416) and below-mean Core performance (83.8%, 54.1%, and 21.1%, respectively; n= 402) were significant. Differences in Step 2 groups Core failure rates (30.0%, 10.6%, and 2.0%, from lowest to highest Step tiers; n= 387) and below-mean Core performance (80.0%, 43.7%, and 14.0%, respectively; n= 380) were significant. Step 2 results modified Core outcome and performance predictions for residents in Step 1 terciles of varying statistical significance. Tiered scoring of USMLE Step results has value in predicting radiology resident performance on the ABR Core examination; effective stratification of radiology resident applicants can be done without reporting numerical Step scores.

Performance of On-Call Radiology Residents in Interpreting Total Spine MRI Studies for the Detection of Spinal Cord Compression or Cauda Equina Compression.

OBJECTIVE. Accurate diagnosis of spinal cord compression (SCC) or cauda equina compression (CEC) is important in the emergency setting so management decisions may be made promptly. The purpose of this study is to evaluate the performance of on-call radiology residents in interpreting total spine MRI studies for the detection of SCC or CEC. MATERIALS AND METHODS. On-call radiology residents' preliminary interpretation of total spine MRI studies performed over a period of two and half years were evaluated. Agreements and disagreements were determined for SCC (defined as severe spinal stenosis with compression of the spinal cord and lack of surrounding CSF), CEC (defined as > 75% narrowing of the lumbar canal and lack of CSF in the thecal sac), and the presence of other significant findings that might impact clinical management. Studies with true-positive, true-negative, false-positive, and false-negative findings were identified. RESULTS. SCC or CEC was present in 72 of the 295 studies with preliminary interpretations done by on-call residents. The sensitivity and specificity for detecting SCC or CEC were 93.1% and 98.2%, respectively. Sensitivity was higher for 3rd- and 4th-year residents (97.7%) than for 2nd-year residents (86.2%). Other significant findings were present in 34 of the 295 studies. The overall sensitivity and specificity for the detection of other significant findings were 85.3% and 100.0%, respectively. The level of training did not affect the detection of other significant findings. CONCLUSION. The overall sensitivity of on-call radiology residents' interpretation of total spine MR images for the detection of SCC or CEC was high, improving with their level of training. However, residents' sensitivity was slightly less for the detection of other significant findings. Resident performance can be further improved with focused training and the use of a preliminary interpretation template.

Training with simulated lung nodules in X-rays can improve the localization performance of radiology residents

Training with simulated lung nodules in X-rays can improve the localization performance of radiology residents

Assessing Competence in Emergency Radiology Using an Online Simulator

Traditional assessments in radiology residency focus on the Medical Expert CanMEDS role and typically rely upon a single or limited static images. We designed an Emergency Radiology Simulator that aimed to assess the breadth of competencies required across Medical and NonMedical Expert domains. An online simulator with typical emergency cases was administered in October 2015 to Post Graduate Year (PGY) 2-5 residents in Radiology. Residents provided preliminary reports, which were graded for style and content. The simulation also included prioritization, protocoling, counseling, and handover exercises geared to assess NonMedical Expert roles. Fourty eight residents participated in the simulation. Level of resident was 11 PGY-2, 17 PGY-3, 13 PGY-4, and 7 PGY-5. There was a significant difference in resident performance between PGY-2 residents and those more senior in terms of the Medical Expert role (findings, diagnosis, recommendations, and clinical relevance of reports). Differences in performance between PGY levels were not seen in the NonMedical Expert roles (prioritization, protocoling, counseling, and handover). Simulation provides an opportunity to assess radiology resident performance across multiple domains. PGY-2 residents performed worse on the Medical Expert domains, although performance did not significantly vary between the other years. This may suggest that competence in Emergency Radiology is achieved early in residency, possibly related to the importance placed on developing skills related to on-call performance during the PGY-2 year. The simulator should be extended to other areas of Radiology, in order to assess the ability to discriminate performance in other subspecialties.