A retrospective analysis of the impact of disabling item review on item performance on computerized fixed-item tests in a doctor of pharmacy program

Abstract

ObjectiveThe decision to disable item review, or ‘backward navigation,’ during computerized-fixed item tests proved controversial among faculty at our institution. We sought to determine the effect of disabling backward navigation on performance of individual exam items and overall exam performance across multiple courses within a doctor of pharmacy program. MethodsExam items that were administered unchanged and without error or adjustment of scoring between 2016 and 2017 were eligible for inclusion. Included items were evaluated for change in difficulty index, point biserial, and discrimination index for the year when backward navigation was enabled to the year after the function was disabled. Performance on matching exam pairs in each time frame was compared for any changes. ResultsWe screened 2033 items and identified 576 which met study inclusion criteria. There were no significant differences in overall item difficulty index, point biserial, discrimination index or performance of the 27% lowest-scoring students. There was a decrease of 0.95% for the highest-scoring students (z = −2.93, p = 0.003). We identified 15 pairs of exams that contained at least 30% identical items from 2016 to 2017. No difference was found in the percent score minimum, maximum, mean, median, or standard deviation. ConclusionsAlthough there was a statistically significant decrease in item performance for students with the highest scores on the exam, we were unable to demonstrate that disabling backward navigation had a significant impact on overall item performance or exam results.