Purpose: Per the medical education research literature, there remains no gold standard for determining medical education exam cut scores (passing scores). 1 Most U.S. medical schools use an absolute or relative standard method; each has advantages and limitations. Given the diverse goals schools strive to accomplish when administering exams, neither approach is sufficient. For example, in Phase 1 of our basic science curriculum, we endeavor to develop exams that are valid, reliable, appropriately rigorous, and criterion-referenced; prepare students for the high-stakes testing situations they will encounter in their medical education and graduate medical education programs and State Board exams; and provide a valid means to adjust cut scores due to cohort performance and item difficulty. A combination cut score method allows schools to accomplish such goals in a manner that is also easy to explain and defend to students and faculty. This presentation will discuss the advantages, process, and outcomes of using a combination cut score method for basic science assessment compared with absolute and relative standard setting methods. Approach: Per Cohen-Schotanus and van der Vleuten’s study, 2 we employed a combination cut score method, which uses the best performing students as point of reference. Our class size is 60 students per cohort, and our Phase 1 basic science curriculum is delivered via 14 modules. Faculty co-leaders develop module exams via NBME’s Customized Assessment Services. Specifically, for each exam, the cut score is 65% of the 95th percentile student’s score. To date, we have completed 5 module exams with promising psychometric outcomes. Results: Thus far, we have found that our outcomes are comparable to previous years’ while being more manageable and slightly more consistent. Given that we pivoted to administering NBME exams virtually during the pandemic, managing exam/retake creation and online proctoring consumes considerable time and resources. These more predictable and consistent outcomes of the combination method have aided in our ability to manage our pandemic contingency plans effectively. Specifically, our exam outcomes include more consistent exam cut scores and number of student failures. Across the 14 exams last year, the cut scores ranged from 56 to 65. Across the 5 exams administered thus far this year, the range of cut scores is 57 to 65. More significantly, last year’s number of failures ranged from 0 to 5, and this year the number of failures range from 1 to 3. As predicted when the combination cut score method was proposed, the lower ranges of cut scores and increased consistency in the number of student failures are positive outcomes that were expected. Discussion: Thus far, we have learned that the combination cut score method is producing outcomes as expected with fewer surprises. Students report appreciation for the straightforward nature of the cut score method and our transparency in reporting it at the beginning of the course. The use of consistent cut scores has allayed student and faculty concerns regarding exam validity and reliability, and the more consistent number of failures has allowed our assessment team to support the exam retake process more efficiently and with fewer human resources. Significance: As medical schools continue to grapple with exam cut score decisions, a combination cut score method offers several advantages. It supports criterion-referenced exams that measure student attainment of objectives while also considering cohort abilities and test item difficulty. It is simple and inexpensive to employ, transparent and easy to understand, yields consistent and manageable results, and is based upon sound research practices.
Read full abstract