Increased accessibility of computer-based testing for residency application to a hospital in Brazil with item characteristics comparable to paper-based testing: a psychometric study.

Abstract

With the COVID-19 pandemic, online high-stakes exams have become a viable alternative. This study evaluated the feasibility of computer-based testing (CBT) for medical residency applications in Brazil and its impacts on item quality and applicants' access compared to paper-based testing. In 2020, an online CBT was conducted in a Ribeirao Preto Clinical Hospital in Brazil. In total, 120 multiple-choice question items were constructed. Two years later, the exam was performed as paper-based testing. Item construction processes were similar for both exams. Difficulty and discrimination indexes, point-biserial coefficient, difficulty, discrimination, guessing parameters, and Cronbach's alpha coefficient were measured based on the item response and classical test theories. Internet stability for applicants was monitored. In 2020, 4,846 individuals (57.1% female, mean age of 26.64 ± 3.37 years) applied to the residency program, versus 2,196 individuals (55.2% female, mean age of 26.47 ± 3.20 years) in 2022. For CBT, there was an increase of 2,650 (120.7%) applicants, albeit with significant differences in demographic characteristics. There was a significant increase in applicants from more distant and lower-income Brazilian regions, such as the North (5.6% vs. 2.7%) and Northeast (16.9% vs. 9.0%). No significant differences were found in difficulty and discrimination indexes, point-biserial coefficients, and Cronbach's alpha coefficients between the 2 exams. Online CBT with multiple-choice questions was a viable format for a residency application exam, improving accessibility without compromising exam integrity and quality.