Computerized Adaptive Testing Simulation Research Articles

Development of a parent-report cognitive function item bank using item response theory and exploration of its clinical utility in computerized adaptive testing.

The purpose of this study is to report the reliability, validity, and clinical utility of a parent-report perceived cognitive function (pedsPCF) item bank. From the U.S. general population, 1,409 parents of children aged 7-17 years completed 45 pedsPCF items. Their psychometric properties were evaluated using Item Response Theory (IRT) approaches. Receiver operating characteristic (ROC) curves and discriminant function analysis were used to predict clinical problems on child behavior checklist (CBCL) scales. A computerized adaptive testing (CAT) simulation was used to evaluate clinical utility. The final 43-item pedsPCF item bank demonstrates no item bias, has acceptable IRT parameters, and provides good prediction of related clinical problems. CAT simulation resulted in correlations of 0.98 between CAT and the full-length pedsPCF. The pedsPCF has sound psychometric properties, U.S. general population norms, and a brief-yet-precise CAT version is available. Future work will evaluate pedsPCF in other clinical populations in which cognitive function is important.

SIMPOLYCAT: An SAS program for conducting CAT simulation based on polytomous IRT models

A real-data simulation of computerized adaptive testing (CAT) is an important step in real-life CAT applications. Such a simulation allows CAT developers to evaluate important features of the CAT system, such as item selection and stopping rules, before live testing. SIMPOLYCAT, an SAS macro program, was created by the authors to conduct real-data CAT simulations based on polytomous item response theory (IRT) models. In SIMPOLYCAT, item responses can be input from an external file or generated internally on the basis of item parameters provided by users. The program allows users to choose among methods of setting initial , approaches to item selection, trait estimators, CAT stopping criteria, polytomous IRT models, and other CAT parameters. In addition, CAT simulation results can be saved easily and used for further study. The purpose of this article is to introduce SIMPOLYCAT, briefly describe the program algorithm and parameters, and provide examples of CAT simulations, using generated and real data. Visual comparisons of the results obtained from the CAT simulations are presented.

Test Design Optimization in CAT Early Stage with the Nominal Response Model

The early stage of computerized adaptive testing (CAT) refers to the phase of the trait estimation during the administration of only a few items. This phase can be characterized by bias and instability of estimation. In this study, an item selection criterion is introduced in an attempt to lessen this instability: the D-optimality criterion. A polytomous unconstrained CAT simulation is carried out to evaluate this criterion's performance under different test premises. The simulation shows that the extent of early stage instability depends primarily on the quality of the item pool information and its size and secondarily on the item selection criteria. The efficiency of the D-optimality criterion is similar to the efficiency of other known item selection criteria. Yet, it often yields estimates that, at the beginning of CAT, display a more robust performance against instability.

(886): How many questions does it take to obtain valid and precise estimates of the impact of pain on a patient’s life?

The assessment of pain and its impact requires reliable and valid patient-based assessment tools. Computerized adaptive testing (CAT) is a novel approach to pain assessment. This study examined the precision and validity of pain-impact scores and response burden from simulated CAT administration of a bank of 45 items. Adults (n=1,500) randomly sampled from the general US population via telephone or internet completed a questionnaire consisting of pain items from widely-used instruments (SF-36, Nottingham Health Profile, Sickness Impact Profile, McGill Pain, Oswestry Low Back Pain, Brief Pain Index, Aberdeen Back Pain, EuroQOL, Health Insurance Experiment, Fibromyalgia Impact Questionnaire, Short MusculoSkeletal Function Assessment). A 45-item bank meeting the assumptions of an item response theory (IRT) model was calibrated on a common metric and a series of CAT administrations with varying fixed lengths (2, 5 and 7 items) or varying precision levels were simulated. Each simulated CAT score was compared to a score (theta) based on the entire item bank. High correlations were observed between the CAT estimates and the theta score (.90-.99). The theta score showed the best measurement precision (95%CI:1.6-15.6) at all score levels; the CAT-2 showed the least precision (95%CI:5.8-14.8); the CAT-5 (95%CI:3.3-12.6) and CAT-7 (95%CI:2.8-12.7) provided acceptable precision levels, while dramatically decreasing the length of the measure. All assessments were most accurate in the middle range of the scale and showed substantial increases in measurement error towards the end points. Taking into account the variations in precision observed across CAT simulations, on average 5-6 items were needed to reach acceptable levels of precision, representing an 89% reduction in respondent burden. All CATs significantly discriminated (p<.001) between people with and without chronic pain in the hypothesized way. The results of this study suggest that it takes 5-6 tailored items to produce a valid and precise patient-reported score of pain impact. The assessment of pain and its impact requires reliable and valid patient-based assessment tools. Computerized adaptive testing (CAT) is a novel approach to pain assessment. This study examined the precision and validity of pain-impact scores and response burden from simulated CAT administration of a bank of 45 items. Adults (n=1,500) randomly sampled from the general US population via telephone or internet completed a questionnaire consisting of pain items from widely-used instruments (SF-36, Nottingham Health Profile, Sickness Impact Profile, McGill Pain, Oswestry Low Back Pain, Brief Pain Index, Aberdeen Back Pain, EuroQOL, Health Insurance Experiment, Fibromyalgia Impact Questionnaire, Short MusculoSkeletal Function Assessment). A 45-item bank meeting the assumptions of an item response theory (IRT) model was calibrated on a common metric and a series of CAT administrations with varying fixed lengths (2, 5 and 7 items) or varying precision levels were simulated. Each simulated CAT score was compared to a score (theta) based on the entire item bank. High correlations were observed between the CAT estimates and the theta score (.90-.99). The theta score showed the best measurement precision (95%CI:1.6-15.6) at all score levels; the CAT-2 showed the least precision (95%CI:5.8-14.8); the CAT-5 (95%CI:3.3-12.6) and CAT-7 (95%CI:2.8-12.7) provided acceptable precision levels, while dramatically decreasing the length of the measure. All assessments were most accurate in the middle range of the scale and showed substantial increases in measurement error towards the end points. Taking into account the variations in precision observed across CAT simulations, on average 5-6 items were needed to reach acceptable levels of precision, representing an 89% reduction in respondent burden. All CATs significantly discriminated (p<.001) between people with and without chronic pain in the hypothesized way. The results of this study suggest that it takes 5-6 tailored items to produce a valid and precise patient-reported score of pain impact.