Data-driven item selection for the Shirts and Shoes Test

Abstract

PurposeThe first stage of test development is the generation of a pool of potential items to be used in the assessment. Ideally, these items are then tested in the field on members of the population for which the assessment is intended. This initial analysis assures that tests build in validity at the level of the individual items. MethodSeventy-four potential test items developed for the Shirts and Shoes Test (Plante & Vance, 2012) were tested on 513 children between the ages of 3 and 6 years. An Item Response Theory (IRT) approach was used to identify potential items in the item pool which are unsuitable due to one or more types of deviation from the IRT model. Furthermore, an IRT approach provides data that permits the ordering of items by difficulty. ResultsA total of 19 items were discarded due to noteworthy deviations in one or more of the measures related to item response characteristics. The remaining 55 items reflected a range of skill difficulty and were deemed suitable for an age range of 3- to 6-years. ConclusionsItem Response Theory can be used productively to identify items that fail to add to the reliability and validity of a test during the earliest stages of test development. The number of items culled for this test demonstrates that the content of the items alone does not assure that the items perform in ways that promote accurate skill measurement.