Effect of Remote Proctoring of the Orthopaedic In-training Examination on Scores.

Abstract

Introduction:The pandemic has created an opportunity to explore different testing settings. However, transitioning to remote testing raises concerns about cheating. The possibility of cheating raises concern about the integrity of test psychometrics. In 2020, the orthopaedic in-training examination (OITE) was offered using two administration models, with programs having the option to provide physically distanced, in-person, proctored testing or remote testing and online proctoring. We aimed to determine whether scores were higher when the OITE was administered and proctored remotely.Methods:Residents of allopathic programs from the 2020 administration of the OITE were included. The American Academy of Orthopaedic Surgeons database of examinees and deidentified aggregate scores were compared. The primary outcome variable was the number of items answered correctly. Of 275 questions and after psychometric analysis, eight questions were excluded, leaving 267 questions as the denominator for percent correct. The mean number of items answered correctly was compared between the group with in-person proctoring and the group with virtual, digital proctoring using the two-sample Student t-test.Results:A total of 4405 examinees consisted of two cohorts: 1834 residents (42%) took the OITE with in-person proctoring and 2571 residents (58%) completed the test in the remote testing and proctoring models. No difference in mean scores was observed between in-person proctored and remotely proctored examinees (in-person: 162.98 ± 21.11, remote: 162.22 ± 22.04, mean difference: −0.75 [95% CI, −2.04 to 0.53]; P = 0.25).Discussion:Remote testing with virtual proctoring has become more widely used. There was no difference in scores and no evidence of enough cheating to change the “curve” of the OITE. Knowing that there was no evidence of sufficient cheating to change overall test psychometrics, programs and residents can be reassured that the OITE remains a valid educational instrument even when administered remotely.