Why should the first be last? “seasonal” variations in the National Board of Medical Examiners (NBME) Subject Examination Program for medical students in surgery

Abstract

We recently noticed that many of the students whom past experience had proved good performers did not do as well on the early (July) NBME Surgery examination as expected. Moreover, students with weaker records did better than expected in the later (April) tests. Because the raw scores of the NBME examinations are important for students’ course grades and residency placement, we decided to quantitatively analyze the observed trend to determine if some correction of “seasonal” scores might be necessary. If the raw surgical subject scores vary seasonally, the ability to accurately compare students taking the examination at different rotations is undercut. Medical students are educated and evaluated in a wide variety of styles and settings. Recognizing this, medical schools have increasingly relied on more objective, standardized criteria to aid in the grading and ranking of medical students upon the completion of surgical clerkships. To this end, the NBME has prepared and offers a subject examination (sometimes called a “shelf” examination) for use by medical schools to aid in the evaluation of medical students. Some schools require threshold scores on the NBME subject examination in order for students to pass their given clerkships. Subject examination scores are variably used as a major component in the determination of final student grade in the surgery clerkship. In many schools, the raw scores on the NBME subject examination serve as a prerequisite for achieving clerkship honors. Clerkship honors, in turn, are required by many of the “better” residency programs for consideration for an interview and possible appointment. Thus, the NBME subject examination can play a substantial role in evaluation of surgical clinical clerks and potential residents. Although no comprehensive study has been done on this topic, Ripkey et al have mentioned that medical students sitting for subject examinations in Medicine and Surgery later in the examination season tended to have higher scores relative to their peers who were tested earlier in the examination season. The NBME already acknowledges “seasonal” variations for the subject examination in Medicine. The NBME states in its Spring 2001 Subject Examination Newsletter that, “it is common knowledge that scores in certain clinical exams are progressively higher for students of equivalent ability who take the relevant rotation later in the academic year. This is particularly true for Medicine, where the nature of the subject enables substantive contributions to the score from knowledge accumulated over a variety of rotations. Beginning July 2001, national norms will be provided each quarter as well as the entire year.” The NBME, however, does not account for “seasonal” variations in any of its other subject examinations, such as Surgery, where we thought the greatest degree of “seasonal” variation existed. Our study considers the need for a reexamination of the method by which the raw scores of the Surgery subject examination are interpreted by the NBME and program directors.