Oral examinations in undergraduate medical education – What is the ‘value added’ to evaluation?

Abstract

BackgroundGiven the long tradition of oral examinations in surgical training, surgical clerkships continue to use oral examinations to evaluate medical students even though the value of oral examination at the post-graduate level has been questioned. The key issue in the context of undergraduate surgical training then is to understand value of the oral examination in assessment. The goal of this study is to clarify what oral examinations do, or appear to, test and how this complements other methods of assessment. MethodsThe study is a retrospective, qualitative study of comments provided by examiners on the oral examination score sheets evaluating performance of students completing their core surgery clerkship at an academic medical center. Through immersion in and initial familiarization with the data we develop a scheme of codes for labeling the data for subsequent synthesis. Using these inductive codes, all comments were reviewed and analyzed to determine what qualities examiners detect, or naturally comment on, when administering and scoring the oral examinations. ResultsThirteen substantive codes (Communication, Critical Thinking, Decisiveness, Demeanor, Differential Diagnosis, Focus, Knowledge, Management, Organization, Pace, Prompting, Thoroughness, and Work Up) and three valence codes (Negative, Neutral, and Positive) were developed and used to code the data. The most universal code was ‘Knowledge’, used by 43 (100%) of examiners; the most frequently used code was ‘Work Up’, applied to the comments 437 (21.1%) times. Overall, positive valence was attached to 1146 (55.2%) of codes and negative valence to 879 (42.3%) codes. The most discriminating codes in grading were ‘Demeanor’, ‘Focus’, and ‘Organization’. ConclusionsOral examinations provide rich opportunity for testing qualities readily tested on other examinations but also many intangible qualities that are otherwise less well or not well tested. As such, the ‘value-added’ by oral examinations likely justifies their continued use in the evaluation of surgical trainees. The identification of testable qualities should aid in the development of a standardized scoring rubric, the use of which may aid in minimizing subjectivity and bias in what otherwise is a rich assessment tool.