Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ?

Abstract

The onset of the COVID-19 pandemic brought with it the closure of many bricks-and-mortar testing centres and a concomitant move to the remote (online) proctoring of many credentialing exams that were scheduled to take place throughout 2020. In effect, this meant changing from a standardized to a non-standardized administration of tests given that candidates could take tests at home using personal desktop or laptop devices. This in turn begs the question: do outcomes across tests taken in testing centres and via live remote proctoring differ? Due to the relative novelty of remote proctoring, research addressing this question is currently thin on the ground, thus justifying the research described in this paper. The study uses data from eleven professional licensure examinations taken by 14,097 candidates across four US States to compare outcomes for tests proctored either in test centres or remotely in real-time using live remote proctoring (LRP) software. Candidate outcomes were compared using average percent correct and passing rates. Test psychometric properties were compared using KR-20 reliability, decision consistency (Subkoviak's c), item difficulty, item discrimination and time taken to complete tests. At the individual test level, while some statistically significant differences were observed in outcomes across the two proctoring modes, most effect sizes were small. Overall, with the exception of some observable differences in metrics pertaining to item discrimination and timing, no detectable pattern was observed in favour of either mode. The study is significant in that it provides early evidence supporting the use of LRP in high stakes contexts.