An exchange algorithm for optimal calibration of items in computerized achievement tests

Frank Miller,Mahmood Ul Hassan

doi:10.1016/j.csda.2021.107177

Frank Miller, Mahmood Ul Hassan

Open Access

https://doi.org/10.1016/j.csda.2021.107177

Copy DOI

Abstract

The importance of large scale achievement tests, like national tests in school, eligibility tests for university, or international assessments for evaluation of students, is increasing. Pretesting of questions for the above mentioned tests is done to determine characteristic properties of the questions by adding them to an ordinary achievement test. If computerized tests are used, it has been shown using optimal experimental design methods that it is efficient to assign pretest questions to examinees based on their abilities. The specific distribution of abilities of the available examinees are considered and restricted optimal designs are applied. A new algorithm is developed which builds on an equivalence theorem. It discretizes the design space with the possibility to change the grid adaptively during the run, makes use of an exchange idea and filters computed designs. It is illustrated how the algorithm works through some examples as well as how convergence can be checked. The new algorithm is flexible and can be used even if different models are assumed for different questions.

Highlights

Computer-based achievement tests are growing rapidly with the increase of computer usage and popularity
If the parameter values used for optimal design calculation are misspecified, it may impact the quality of the derived design
We investigate the relative efficiency of the derived design versus the random design if the true parameter values are different from the parameter values used for design calculation

Summary

Introduction

Computer-based achievement tests are growing rapidly with the increase of computer usage and popularity. The most popular large scale achievement tests e.g. the Graduate Record Examinations (GRE) test, Scholastic Assessment Test (SAT), Test of English as a Foreign Language (TOEFL), Graduate Management Admission Test (GMAT), Medical College Admission Test (MCAT), or Pharmacy College Admission Test (PCAT), are all computerized tests These tests require a sufficiently large item bank (stored questions) with known characteristics (e.g. difficulty) of items. A feasible way is to add a small calibration part to an ordinary achievement test regardless of whether it is a paper–pencil or computerized test This principle is applied to the Swedish Scholastic Assessment Test (currently a paper-and-pencil test) and the Swedish driving licence test (computerized test). The calibration part consists of five questions out of 70, see Trafikteori Online (2020)

Objectives

Methods

Results

Conclusion