On the Cover: Is my Test Speeded?

Abstract

“Is my test speeded?” is a question that test developers should ask when developing a power test, a test that measures ability rather than speed. A test is not speeded if most test takers have sufficient time to answer all test questions in the time allotted. One way to check the speededness of a test is to compute the completion rate – that is, the proportion of test takers who answered the test questions at the end of the test. When the time spent on test questions is available, the second way to evaluate the speededness is to compare the time spent on the questions at the end to the time spent on the questions in the beginning. The graphic shows the average response times for items according to their position in the test sequence and their difficulty (i.e., proportion of correct responses). The visual here answers the following research questions: 1) “Is there a general downward trend in mean item response times toward the end of the test, controlling for item difficulty?” and 2) “Do items become more difficult at the end of the test?” The plot shows data from multiple 60-item modules on a high-stakes certification exam. This graph was generated using the R package ggplot2 (Wickham, 2009), and code is available upon request. The solid black line is the grand mean response time by item position across all modules. First, as indicated by the solid black line, it does not appear that item response times are generally decreasing toward the end of the test, indicating that examinees are not “speeding up” as they reach the time limit. Second, the colored lines, indicating the trends in average response time by position as a function of item difficulty, show that more difficult items take more time to complete than easier items, but this relationship does not depend on item position. That is, items do not become more difficult if placed at the end of the test. Overall, the graphic suggests this exam is not speeded. Interested readers may contact Bozhidar M. Bashkov ([email protected]) for questions on this graph or the code to create it. I would recommend readers try varying the number of categories of item difficulty when creating the graph. The comparison may look sharper with a small number of categories – for example, three.