Clinical Knowledge Examination Research Articles

Pediatric residency milestone performance is not predicted by the United States Medical Licensing Examination Step 2 Clinical Knowledge

Objectives: This study aims to show whether correlation exists between pediatric residency applicants’ quantitative scores on the United States Medical Licensing Exam Step 2 Clinical Knowledge examination and their subsequent performance in residency training based on the Accreditation Council for Graduate Medical Education Milestones, which are competency-based assessments that aim to determine residents’ ability to work unsupervised after postgraduate training. No previous literature has correlated Step 2 Clinical Knowledge scores with pediatric residency performance assessed by Milestones. Methods: In this retrospective cohort study, the United States Medical Licensing Exam Step 2 Clinical Knowledge Scores and Milestones data were collected from all 188 residents enrolled in a single categorical pediatric residency program from 2012 - 2017. Pearson correlation coefficients were calculated amongst available test and milestone data points to determine correlation between test scores and clinical performance. Results: Using Pearson correlation coefficients, no significant correlation was found between quantitative scores on the Step 2 Clinical Knowledge exam and average Milestones ratings (r = -0.1 for post-graduate year 1 residents and r = 0.25 for post-graduate year 3 residents). Conclusions: These results demonstrate that Step 2 scores have no correlation to success in residency training as measured by progression along competency-based Milestones. This information should limit the importance residency programs place on quantitative Step 2 scores in their ranking of residency applicants. Future studies should include multiple residency programs across multiple specialties to help make these findings more generalizable.

Pediatric residency milestone performance is not predicted by the United States Medical Licensing Examination Step 2 Clinical Knowledge.

This study aims to show whether correlation exists between pediatric residency applicants' quantitative scores on the United States Medical Licensing Exam Step 2 Clinical Knowledge examination and their subsequent performance in residency training based on the Accreditation Council for Graduate Medical Education Milestones, which are competency-based assessments that aim to determine residents' ability to work unsupervised after postgraduate training. No previous literature has correlated Step 2 Clinical Knowledge scores with pediatric residency performance assessed by Milestones. In this retrospective cohort study, the United States Medical Licensing Exam Step 2 Clinical Knowledge Scores and Milestones data were collected from all 188 residents enrolled in a single categorical pediatric residency program from 2012 - 2017. Pearson correlation coefficients were calculated amongst available test and milestone data points to determine correlation between test scores and clinical performance. Using Pearson correlation coefficients, no significant correlation was found between quantitative scores on the Step 2 Clinical Knowledge exam and average Milestones ratings (r = -0.1 for post-graduate year 1 residents and r = 0.25 for post-graduate year 3 residents). These results demonstrate that Step 2 scores have no correlation to success in residency training as measured by progression along competency-based Milestones. This information should limit the importance residency programs place on quantitative Step 2 scores in their ranking of residency applicants. Future studies should include multiple residency programs across multiple specialties to help make these findings more generalizable.

The Validity of MCAT Scores in Predicting Students' Performance and Progress in Medical School: Results From a Multisite Study.

This is the first multisite investigation of the validity of scores from the current version of the Medical College Admission Test (MCAT) in clerkship and licensure contexts. It examined the predictive validity of MCAT scores and undergraduate grade point averages (UGPAs) for performance in preclerkship and clerkship courses and on the United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge examinations. It also studied students' progress in medical school. Researchers examined data from 17 U.S. and Canadian MD-granting medical schools for 2016 and 2017 entrants who volunteered for the research and applied with scores from the current MCAT exam. They also examined data for all U.S. medical schools for 2016 and 2017 entrants to regular-MD programs who applied with scores from the current exam. Researchers conducted linear and logistic regression analyses to determine whether MCAT total scores added value beyond UGPAs in predicting medical students' performance and progress. Importantly, they examined the comparability of prediction by sex, race and ethnicity, and socioeconomic status. Researchers reported medium to large correlations between MCAT total scores and medical student outcomes. Correlations between total UGPAs and medical student outcomes were similar but slightly lower. When MCAT scores and UGPAs were used together, they predicted student performance and progress better than either alone. Despite differences in average MCAT scores and UGPAs between students who self-identified as White or Asian and those from underrepresented racial and ethnic groups, predictive validity results were comparable. The same was true for students from different socioeconomic backgrounds, and for males and females. These data demonstrate that MCAT scores add value to the prediction of medical student performance and progress and that applicants from different backgrounds who enter medical school with similar ranges of MCAT scores and UGPAs perform similarly in the curriculum.

Perceptions of USMLE Step 1 Pass/Fail Score Reporting Among Orthopedic Surgery Residency Program Directors

The United States Medical Licensing Examination (USMLE) Step 1 examination will transition from graded to pass/fail scoring starting no earlier than January 2022. Orthopedic surgery residency programs will need to adapt to these changes. The goal of this study was to investigate the perceptions of orthopedic surgery residency program directors on the change of Step 1 from a graded to a pass/fail examination. We also investigated how the change would affect the other factors that are typically considered in the selection of orthopedic surgery residents. A survey was distributed to 161 directors of allopathic orthopedic surgery programs. Contact information was obtained from a national database. Of those contacted, 75 (46.6%) program directors responded. Most (85.3%) did not support the pass/fail change. Most believe that greater importance will be placed on the Step 2 Clinical Knowledge examination (96.0%), audition elective with their department (84.0%), personal knowledge of the applicant (78.7%), grades (74.7%), letters of recommendation from recognizable orthopedic surgeons (74.7%), and Alpha Omega Alpha status (69.3%). Most also believe that this change will advantage allopathic students who attend highly regarded schools (58.7%). Most of the program directors support a graded preclinical curriculum (69.3%) and caps on the number of orthopedic surgery residency applications (70.7%). Although most orthopedic surgery program directors disagree with the change to a pass/fail Step 1 examination, residency programs will need to reevaluate how they screen applicants for an interview once the scored Step 1 is no longer available. With this change, other factors, such as Step 2 score, audition rotations, and grades in clerkships, will be emphasized more heavily. [Orthopedics. 2022;45(1):e30-e34.].

Once in the Door, Grit May Matter More: An Evaluation of Grit in Medical Students.

Health professions educators seek understanding of the process of learner performance and achievement. Grit, defined as passion and perseverance for long-term goals, has been implicated in this process. Research suggests that effort counts twice in achievement: skill times effort equals skill and, subsequently, skill times effort equals achievement. Research also suggests that an individual's interest influences job performance and satisfaction. Grit, composed of Perseverance of Effort and Consistency of Interest, has been shown to positively correlate with performance and achievement in various disciplines. Limited research reveals Grit negatively correlates with resident attrition and physician burnout. This study evaluates relationships between Grit and medical student performance measures. We gather validity evidence for Grit scale use among medical students and, subsequently, use this data to evaluate for differences in Grit score by gender and assess whether there is a relationship between Grit and standardized exam scores. We hypothesize that Grit will not vary with gender and that it will be positively associated with standardized examination scores. Enrolled students receive the 12-item Grit survey. An exploratory factor analysis evaluates validity evidence of Grit. Descriptive statistical analysis, Pearson correlation, and moderation analysis evaluate if Grit score differs by gender and if there are relationships with standardized exams. The survey response rate is 59% (412/698) and exploratory factor analysis replicates the two-factor structure of Grit found in other fields-Perseverance of Effort and Consistency of Interest. Average Grit scores do not significantly differ by gender. Grit weakly correlates with United States Medical Licensing Exam Step 1 (r = 0.10, = 0.36) and United States Medical Licensing Exam Step 2 Clinical Knowledge (r = 0.12, P=0.29). Grit weakly and negatively correlates with Medical College Admission Test score (r = - 0.18, P<0.05). Grit is not a significant moderator of the relationship between Medical College Admission score and Step exams scores. Exploratory factor analysis results provide preliminary validity evidence for Grit scale use in medical students. Grit does not significantly moderate the Medical College Admission Test-Step score relationship. Grit correlates negatively with Medical College Admission Test scores. Though the correlations in this study were not significant, the results showed that Grit may tend to positively correlate with Step 1 and Step 2 Clinical Knowledge examinations. Given that these three exams are taken at different levels of training, the stepwise progression towards positive correlation may suggest that the theory "effort counts twice" applies to medical training.

The Relationships among Self-Designated Disadvantage, Socioeconomic Disadvantage, and Academic Performance in Medical School: A Multi-Institutional Study.

As medical schools seek to address the growing disparity between the socioeconomic makeup of their students and the general population, it is important to understand the academic trajectory of disadvantaged students. We used a locally-developed multicomponent socioeconomic disadvantage (SED) measure and the self-designated disadvantaged (SDA) question ["yes" (+) or "no" (-)] from the American Medical College Application Service application to examine academic performance of students from three disadvantaged categories (high SED/SDA+, high SED/SDA-, and low SED/SDA+); with low SED/SDA-as the reference group across five California schools. Compared with reference, the DA+ subgroups scored lower on USMLE Step 1 and Step 2 Clinical Knowledge examinations and received fewer clerkship Honors. After adjustment for academic metrics and sociodemographic variables, high SED subgroups performed similarly to reference, but performance gaps for low SED/SDA+ students persisted. Medical schools must better understand the institutional and other drivers of academic success in disadvantaged students.

Investigation of answer changes on the USMLE\xae Step 2 Clinical Knowledge examination

BackgroundExaminees often believe that changing answers will lower their scores; however, empirical studies suggest that allowing examinees to change responses may improve their performance in classroom assessments. To date, no studies have been able to examine answer changes during large scale professional credentialing or licensing examinations.MethodsIn this study, we expand the research on answer changes by analyzing responses from 27,830 examinees who completed the Step 2 Clinical Knowledge (CK) examination between August of 2015 and August of 2016.ResultsThe results showed that although 68% of examinees changed at least one item, the overall average number of changes was small. Among the examinees who changed answers, approximately 45% increased their scores and approximately 28% decreased their scores. On average, examinees spent shortest time on the item changes from wrong to right and they were more likely to change their scores from wrong to right than right to wrong.ConclusionsConsistent with previous studies, these findings support the beneficial effects of answer changes in high-stakes medical examinations and suggest that examinees who are overly cautious about changing answers may put themselves at a disadvantage.

Effect of Grading and Class Rank on Performance in a Surgical Simulation Course

This study seeks to evaluate the psychological and academic effects of grading a medical school simulation course. This study also seeks to evaluate whether a student's class rank affects these results and whether the academic effects of a simulation course persist long-term. Two separate medical school classes were evaluated. The first class participated in an ungraded surgery simulation course with only formative feedback. The second class participated in a tier-graded course. Qualitative assessment of both courses included daily standardized and self-reported stress surveys, student feedback surveys at the end of the course, and student feedback surveys at the end of the surgery clerkship. Quantitative assessment included pre and postcourse examinations, NBME Surgery Subject Assessment percentile ranks, and COMLEX 2 surgery subscores. Qualitative and quantitative assessments were compared between classes as a whole and between students in the upper and lower half of each class. Results were analyzed using a Student two-tailed t test. Rocky Vista University College of Osteopathic Medicine Institute of Simulation, Parker, Colorado. One hundred forty-eight students from the Rocky Vista University College of Osteopathic Medicine Class of 2017 and one hundred fifty-one students from the Rocky Vista University College of Osteopathic Medicine Class of 2018. Daily standardized stress surveys indicated some increase in stress for students in the graded course, especially for students in the upper half of the graded class. All students adapted to stress significantly by the end of each course, but upper-half students adapted more significantly. Students' self-reported stress levels demonstrated a significant decrease during the ungraded course, but a slight increase during the graded course. Students' self-reported confidence levels rose significantly during the ungraded course, but less significantly during the graded course. Student feedback surveys demonstrated a positive student perception of both courses, but, when stratified, some responses were significantly different between students in the upper and lower half of each class, with upper-half students generally giving a higher rating to the graded course. Clinical knowledge examination scores and improvement in scores within the course were significantly higher for the graded course, with no significant differences between students in the upper and lower half of each class. Students' mean NBME Surgery Subject Assessment percentile rank did not differ significantly between the two classes, but were significantly higher than for previous classes who had not participated in a simulation course. Mean NBME Surgery Subject Assessment percentile rank and COMLEX 2 mean surgery subscore were significantly higher for students in the upper half than for students in the lower half of each class. COMLEX 2 mean surgery subscore did not differ significantly between classes who had participated in a simulation course and previous classes who had not. Students in the upper half of each class in this study performed significantly better than students in the lower half of each class on the mean COMLEX 2 surgery subscore. This study reaches the following conclusions: (1) objective stress level measurements did not correlate with students' self-reported stress levels; (2) grading a simulation course raises objective stress levels in higher-performing students more than in lower-performing students, but higher-performing students demonstrate the ability to effectively adapt to the increased levels of stress and achieve significant increases in confidence; (3) higher performing students generally demonstrate a greater sense of satisfaction with the simulation course experience when the course is graded; (4) higher-performing and lower-performing students within a class perform in a similar fashion on examinations within the intensely structured environment of a simulation course, whether the course is graded or not; (5) grading a course stimulates all students to perform better on examinations within the course; (6) this higher level of performance and similarity of performance between higher and lower-performing students does not continue when students are not in a structured environment, and students stratify into disparate performance groups in accordance with their class rank; and (7) students exposed to a surgical simulation course perform better on NBME surgery subject examinations than students who have not participated in a simulation course, but this increased level of performance does not extend long-term to national board exams.

Does Clerkship Rotation Sequence Affect Performance on National Board of Medical Examiners (NBME) Clinical Subject Examinations and United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) Examination?

To investigate whether starting the clerkship year in family medicine (FM), internal medicine (IM), pediatrics, or surgery influences NBME shelf and USMLE Step 2 CK examination performance. USMLE Step 1, Step 2 CK, and shelf examination scores for FM, IM, pediatrics, and surgery were collected. Sequences were selected on the following assignment criteria: rotation 1 (either FM or IM), rotation 5 (pediatrics), rotation 8 (surgery), rotation 1 (either pediatrics or surgery), and rotation 8 (IM). Multivariate analysis of covariance, with Step 1 as the covariate, was used to investigate rotation sequence on examination performance. Wilks's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in FM or IM) on the pediatrics, surgery, and Step 2 CK examinations (Λ = .95, F[3,51] = .93, p ≤ .432). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .488, F[3,51] = 17.827, p ≤ .001), indicating the two groups differ on Step 1 performance. Wilk's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in pediatrics or surgery) on the IM and Step 2 CK results (Λ = .925, F[2,75] = 3.036, p ≤ .054). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .309, F[2,75] = 83.915, p ≤ .001) indicating that the two groups differ on Step 1 performance. Starting the clerkship year in FM, IM, pediatrics, or surgery does not influence subsequent performance on shelf examinations or on Step 2 CK.

Medical Student Use of Electronic and Paper Health Records During Inpatient Clinical Clerkships: Results of a National Longitudinal Study.

An important goal of medical education is to teach students to use an electronic health record (EHR) safely and effectively. The purpose of this study is to examine medical student accounts of EHR use during their core inpatient clinical clerkships using a national sample. Paper health records (PHRs) are similarly examined. An online survey about health record use within the inpatient component of six core clerkships was administered to medical students after they completed Step 2 Clinical Knowledge of the United States Medical Licensing Examination. The sample included 17,202 U.S. medical students graduating between 2012 and 2016. Mean percentages of clerkships in which students engaged in various health record activities were computed, and analysis of variance was used to examine differences. The mean percentages of clerkships in which a student accessed or entered information into an EHR increased from 78% to 93% and 59% to 72%, respectively. For students who used an EHR, the mean percentage of clerkships in which they entered information remained constant at 76%. Students entered notes during the majority of their clerkships, with increases over time. However, students entered orders in less than a quarter of their clerkships, with decreases over time. The percentage of clerkships in which students used PHRs was lower and declining. Although students used an EHR in the majority of their inpatient core clerkships, they received limited educational experiences related to order and note writing, which could translate into a lack of preparedness for future training and practice.

25 - Low Fidelity, High Validity OSCKE Assessment in the High Fidelity Era

Background: Our Pediatric Residency has been conducts an annual Objective Structured Clinical Knowledge Examination (OSCKE). While anecdotally it appears that OSCKE scores provide a predictive index for board pass rates and as a formative tool to fill knowledge gaps, this theory had not been formally analyzed.

Academic outcomes of a community-based longitudinal integrated clerkships program

Background: Longitudinal integrated clerkships (LICs) receive recognition internationally as effective, innovative alternatives to traditional block rotations (TBRs) in undergraduate medical education. No studies of LICs in the USA have assessed how students perform on all the standardized exams.Aim: To compare performance on standardized tests of students in the first four years of LICs at the University of North Carolina School of Medicine–Asheville (UNC SOM–Asheville) with students from UNC SOM’s Chapel Hill main campus in TBRs.Methods: LIC and TBR students’ previous academic performance was considered using Medical College Admissions Test (MCAT) and United States Medical Licensing Examination (USMLE) Step 1 scores. Step 1 exam tests students’ pre-clinical, basic science knowledge. Outcome measures included all eight standardized National Board of Medical Examiners (NBME) Subject Shelf Examinations and USMLE Step 2 Clinical Knowledge (CK) examinations, which are used widely in the US to assess students’ progress and as prerequisites to eventual licensure. TBR students were selected using propensity scores to match LIC students. Groups were also compared on the required core clinical conditions documented, and on residency specialty choice.Results: Asheville LIC students earned higher scores on the Step 2 CK examination and the six shelf examinations linked to longitudinal clerkships than the matched TBR students (Step 2 CK exam, Family Medicine and Ambulatory Medicine shelf exams reached statistical significance). LIC students logged greater percentages of core conditions than TBR students and more often chose primary care residencies.Conclusions: UNC School of Medicine medical students participating in a longitudinal integrated curriculum in a community setting outperformed fellow students who completed a more TBR curriculum within the school’s academic medical center. Differences were found in performance on standard tests of clinical knowledge (six NBME exams and Step 2 CK exam), documented breadth of clinical experiences, and likelihood of choosing primary care residency programs.

The effect of rescheduling National Board of Medical Examiners subject examinations on United States Medical Licensing Examination Step 2 Clinical Knowledge performance

Historically, the University of Mississippi School of Medicine administered all National Board of Medical Examiners subject examinations at the end of the third year of the medical curriculum, just before the comprehensive United States Medical Licensing Examination Step 2 Clinical Knowledge examination. Due in part to grading and accreditation concerns, subject examinations were moved in a phased fashion from the end of the year to the end of the clerkship associated with each subject examination; this research aims to determine the effect of this rescheduling. Multiple hierarchical regression was performed on Step 2 Clinical Knowledge examination scores by whether the subject examination was given at end-of-clerkship or end-of-year, while controlling for covariates known to affect examination performance. This analysis found that performance on the Step 2 Clinical Knowledge examination is not significantly affected by the scheduling of subject examinations. The benefits gained from moving the subject examinations from end-of-year to end-of-clerkship, such as timely feedback and compliance with accreditation standards, do not appear to come at the cost of decreased performance on the Step 2 Clinical Knowledge examination.

The Utility of Pre-Residency Standardized Tests for Anesthesiology Resident Selection

The resident selection process could be improved if United States Medical Licensing Examination (USMLE) scores obtained during residency application were found to predict success on the American Board of Anesthesiology (ABA) written examination (part 1). In this study, we compared USMLE performance during medical school to anesthesiology residency standardized examination performance. Sixty-nine anesthesiology residents' USMLE, ABA/American Society of Anesthesiologists (ASA) In-Training Examination, and ABA written board examination (part 1) scores were compared. Linear regression, adjusted Pearson partial correlation, multiple regression, and analysis of variance were used to cross-correlate pre-residency and intra-residency scores. Residents' school of medicine location and year of graduation were noted. Both USMLE step 1 and step 2 Clinical Knowledge examinations correlated significantly with all intra-residency standardized tests. Averaged step 1 and step 2 USMLE score correlated to ABA written examination (part 1) score with a slope of 0.72 and r of 0.48 (P = 0.001). The USMLE is a significant predictor of residency ABA/ASA In-Training Examination and ABA written examination performance in anesthesiology. Our program has significantly increased its average written board examination performance while increasing the relative importance of USMLE in resident selection.

Louisiana State University Health Sciences Center School of Medicine at New Orleans

Louisiana State University Health Sciences Center School of Medicine at New Orleans

Composite Score Validity in Clerkship Grading

Composite score validity depends on its reliability, content, and comparison with other outcomes measures. This study examined all three aspects of a composite score used in a third-year surgery clerkship. Composite score reliability was calculated using stratified alpha; several other reasonable composite combinations were tested. Correlation coefficients between clerkship composite scores and grades were calculated with other achievement measures. Two consecutive years were studied (N = 162, N = 159). The clerkship composite score reliability was .76 (Year 1) and .81 (Year 2). Positive correlations (P <or= .05) were found between composite scores and other clerkships' test scores and the United States Medical Licensing Examination Step 2 Clinical Knowledge examination. Positive correlations (P <or= .05) were found between surgery clerkship grades and other clerkship grades. This study documents the reliability of a composite score and provides evidence for its validity in a surgery clerkship.

Medical Education in the Caribbean: Variability in Medical School Programs and Performance of Students

The purpose of this study was to examine the structure of medical training programs in the Caribbean and the performance of the students. There are 56 medical schools in 16 countries currently recognized and open in the Caribbean. Almost 30,000 students from this region applied for exams leading to Educational Commission for Foreign Medical Graduates (ECFMG) certification between 1993 and 2007. The authors summarized school characteristics and pass rates on United States Medical Licensing Examinations, by country. The structure of medical education varies across the region, including existence of accrediting bodies and review processes. First-attempt pass rates by country ranged from 19.4% to 84.4% for Step 1, from 26.3% to 79.7% for the Step 2 Clinical Knowledge Examination, and from 60.6% to 97.2% for the Step 2 Clinical Skills Examination. There is significant variability in undergraduate medical education and the performance of students of medical schools in Caribbean nations.

Relationships among Subcomponents of the USMLE Step 2 Clinical Skills Examination, The Step 1, and the Step 2 Clinical Knowledge Examinations

This research examined relationships between and among scores from the United States Medical Licensing Examination (USMLE) Step 1, Step 2 Clinical Knowledge (CK), and subcomponents of the Step 2 Clinical Skills (CS) examination. Correlations and failure rates were produced for first-time takers who tested during the first year of Step 2 CS Examination administration (June 2004 to July 2005). True-score correlations were high between patient note (PN) and data gathering (DG), moderate between communication and interpersonal skills and DG, and low between the remaining score pairs. There was little overlap between examinees failing Step 2 CK and the different components of Step 2 CS. Results suggest that combining DG and PN scores into a single composite score is reasonable and that relatively little redundancy exists between Step 2 CK and CS scores.

Assessing the validity of the USMLE step 2 clinical knowledge examination through an evaluation of its clinical relevance.

To assess the validity of the USMLE Step 2 Clinical Knowledge (CK) examination by addressing the degree to which experts view item content as clinically relevant and appropriate for Step 2 CK. Twenty-seven experts were asked to complete three survey questions related to the clinical relevance and appropriateness of 150 Step 2 CK multiple-choice questions. Percentages, reliability estimates, and correlation coefficients were calculated and ordinary least squares regression was used. Results showed that 92% of expert judgments indicated the item content was clinically relevant, 90% indicated the content was appropriate for Step 2 CK, and 85% indicated the content was used in clinical practice. The regression indicated that more difficult items and more frequently used items are considered more appropriate for Step 2 CK. Results suggest that the majority of item content is clinically relevant and appropriate, thus providing validation support for Step 2 CK.