Construct The study gathers validity evidence for the use of the Oldenburg Burnout Inventory – Medical Student (OLBI-MS), a 16-item scale used to measure medical student burnout. The 16 items on the OLBI-MS are split to form two subscales, disengagement and exhaustion. Background Medical student burnout has been empirically linked to several detrimental professional and personal consequences. In recognition of the high prevalence of medical student burnout, one recommendation has been to regularly measure burnout using standardized measures that have strong validity evidence for their intended use. The OLBI-MS, a frequently used measure of medical student burnout, was adapted from the Oldenburg Burnout Inventory (OLBI). The OLBI has been studied in many occupational settings and been found to have a two-factor solution in majority of these populations, but there is limited validity evidence available that supports the use of the OLBI-MS subscales in a medical student population. Approach Two years of Association of American Medical College Year 2 Questionnaire data (n = 24,008) were used in the study for a series of exploratory and confirmatory factor analyses. The data from the first year (n = 11,586) was randomly split into a confirmatory and exploratory sample, with the data from the second year (n = 12,422) used as a secondary confirmatory sample. Because the questionnaire is administered to medical students during their second year of undergraduate medical education, we consider this a study as providing validity evidence specifically for the measure’s use with that population. Findings The two-factor structure of the OLBI-MS was not empirically supported in the second year medical-student population. Several of the items had low inter-item correlations and/or moderate correlations with unexpected items. Three modified versions of the OLBI-MS were tested using subsets of the original items. Two of the modified versions were adequate statistical explanations of the relationships in the data. However, it is unclear if these revised scales appropriately measure all aspects of the construct of burnout and additional validity evidence is needed prior to their use. Conclusions The use of the OLBI-MS is not recommended for measuring second-year medical student burnout. It is unclear if the OLBI-MS is appropriate for medical students at all, or if different measures are necessary at different stages in a medical student’s professional development. Additional research is needed to either improve the OLBI-MS or use it as a foundation for a new measure. Supplemental data for this article is available online at at www.tandfonline.com/htlm .