Computerized Adaptive Testing Research Articles

Evolution of Curriculum Standards: A Comprehensive Review of Historical Development

This article offers a thorough examination of curriculum standards in higher education, utilizing a theoretical taxonomy to assess significant empirical studies from the last three decades. It rigorously analyzes the progression of curricular standards, delineating their impact on curriculum development, academic standards, quality assurance, and the formulation and regulation of curriculum. The article delineates the historical influences that have shaped curriculum standards, emphasizing the effects of economic, political, and social considerations. It delineates deficiencies and shortcomings in the current literature, especially about the historical evolution of curriculum assessment, and provides a comprehensive evaluation of prior studies. The prevalence of economic and political ideologies, which can overshadow the educational objectives of curriculum standards and lead to a test-centric methodology that undermines broader educational principles, is a critical concern. The article also examines the future of curriculum standards, emphasizing problems including the influence of technology on assessments, the evolving role of educators, and the necessity for innovative curriculum design. Assessment processes view technology as a transformative force that encompasses automated scoring, computer-adaptive testing, and alignment with evolving content standards. The article then advocates for more discussion among researchers, policymakers, and educators to develop effective, evidence-based standards that reconcile local control with the necessity for educational reform. Thus, it emphasizes the necessity of addressing unresolved tensions and existing research deficiencies in the discourse surrounding curriculum standards.

Development of self-report measures of physical, mental, and emotional fatigability: the michigan fatigability index (MIFI).

To develop item banks and static short-forms for a patient reported outcome measure of perceived physical, mental, and emotional fatigability-the Michigan Fatigability Index (MIFI). Building on earlier qualitative work, this study utilized cognitive interviews in samples representing the general adult population across the lifespan, people with multiple sclerosis (MS), or people with fibromyalgia (FM) to develop initial item pools. A nationwide cross-sectional survey study in MS and FM samples was used to field test items to develop item banks. Exploratory and confirmatory factors analyses indicated single underlying dimensions for each of the MIFI subdomains, and a graded response model (GRM) supported item fit of 42, 28, and 23 items to the physical, mental, and emotional fatigability items banks, respectively. Differential item functioning was not detected for age, condition (FM vs. MS), education level, ethnicity, race, or sex. Expert review and GRM calibration data was used to inform the selection of three 6-item short forms that assess physical, mental, and emotional fatigability and to program associated computer adaptive tests. The MIFI shows initial evidence of good psychometric properties. Users can administer the MIFI as a static short form and efforts are underway to provide access to the measure as a computer adaptive test. T-Score scale conversion allows comparison of individual scores to a normative clinical sample with a wide range of fatigability.

Psychometric evaluation and computerized adaptive testing simulations of age-related macular degeneration quality of life item banks.

Psychometric evaluation and computerized adaptive testing simulations of age-related macular degeneration quality of life item banks.

Which type of PROMs to use in MS routine clinical care: The validity of PROMIS CAT questionnaires.

Which type of PROMs to use in MS routine clinical care: The validity of PROMIS CAT questionnaires.

Fast, smart, and adaptive: using machine learning to optimize mental health assessment and monitor change over time

In mental health, accurate symptom assessment and precise measurement of patient conditions are crucial for clinical decision-making and effective treatment planning. Traditional assessment methods can be burdensome, especially for vulnerable populations, leading to decreased motivation and potentially unreliable results. Computerized Adaptive Testing (CAT) has emerged as a solution, offering efficient and personalized assessments. In particular, Machine Learning-based CAT (MT-based CATs) enables adaptive, rapid, and accurate evaluations that are more easily implementable than traditional methods. This approach bypasses typical item selection processes and the associated computational costs while avoiding the rigid assumptions of traditional CAT approaches. This study investigates the effectiveness of Machine Learning-Model Tree-based CAT (ML-MT-based CAT) in detecting changes in mental health measures collected at four time points (6-month intervals between February 2018 and December 2019). Three CATs measuring generalized anxiety, depression, and social anxiety were developed and tested on a dataset with responses from 564 participants. A cross-validation approach based on real data simulations was used. Results showed that ML-MT-based CATs produced estimates of trait levels comparable to full-length tests while reducing the number of items administered by 50% or more. In addition, ML-MT-based CATs captured changes in trait levels consistent with full-length tests, outperforming short static measures.

Relationships Between PROMIS and Legacy Patient-Reported Outcome Measure (PROM) Scores in the MARS Cohort at 10-Year Follow-up.

Patient-reported outcome measures (PROMs) are used to evaluate the impact of musculoskeletal conditions and their treatment on patients' quality of life, but they have limitations, such as high responder burden and floor and ceiling effects. The Patient-Reported Outcomes Measurement Information System (PROMIS) was developed to address these issues but needs to be further evaluated in comparison with legacy PROMs. The goals of this study were to evaluate the floor and ceiling effects of, the correlations between, and the predictive ability of PROMIS scores compared with traditional legacy measures at 10-year follow-up in a cohort who underwent revision anterior cruciate ligament (ACL) reconstruction. A total of 203 patients (88.7% White; 51.7% female) who underwent revision ACL reconstruction completed the PROMIS via computer adaptive tests as well as legacy PROMs at the cross-sectional, 10-year follow-up of the longitudinal MARS cohort study (MARS cohort n = 1,234). Floor and ceiling effects and Spearman rho correlations between PROMIS and legacy measures are reported. Linear regression with quadratic terms were used to develop and evaluate conversion equations to predict legacy scores from the PROMIS. No floor or ceiling effects were reported for the PROMIS Physical Function (PF) domain, whereas a floor effect was found for 37.9% of the participants for the PROMIS Pain Interference (PI) domain, and a ceiling effect was found for 34.0% of the participants for the PROMIS Physical Mobility (PM) domain. PROMIS domains correlated moderately with the International Knee Documentation Committee total subjective score (absolute value of rho [|ρ|] = 0.68 to 0.74), fairly to moderately with the Knee injury and Osteoarthritis Outcome Score and Western Ontario and McMaster Universities Osteoarthritis Index scores (|ρ| = 0.52 to 0.67), and fairly with the Marx Activity Rating Scale (|ρ| = 0.35 to 0.44). None of the legacy-measure scores were accurately predicted by the PROMIS scores. The PROMIS PF domain has value in assessing patients 10 years after revision ACL reconstruction. Because of floor and ceiling effects, using the PI and PM domains may not allow for precision when measuring long-term changes in pain and mobility. Although the PROMIS measures correlated with the legacy measures, with effect sizes ranging from fair to moderate, the legacy scores were not accurately predicted by the PROMIS. The results suggest that knee-specific legacy measures should not be eliminated from long-term follow-up when the goal is to capture the specific knee-related information that they provide. Therapeutic Level II. See Instructions for Authors for a complete description of levels of evidence.

Comparative performance of PROMIS Sleep Disturbance computerized adaptive testing algorithms and static short form in postmenopausal women

BackgroundThe Patient-Reported Outcomes Measurement Information System (PROMIS) Sleep Disturbance v1.0 item bank (27 items) measures sleep disturbances. Rather than the full item bank, an 8-item short form (PROMIS SD SF 8b) or computerized adaptive testing (CAT) can be used. This study compares the performance of the PROMIS SD SF 8b with two CAT algorithms in postmenopausal women.MethodsThis is a secondary analysis of data collected for the original psychometric testing of the PROMIS Sleep Disturbance item bank, in a sub-sample of women aged ≥55. A graded response model (GRM) was fitted for the item bank, then simulations evaluated the performance of CAT algorithms and the short form, in terms of root mean square error (RMSE) versus the latent trait estimate derived from the full bank. Two CAT algorithms were tested: CAT1 (stop once standard error <0.3 or 12 items administered) and CAT2 (stop once 8 items administered). Convergent and divergent hypotheses for validity were tested through correlations with the Pittsburgh Sleep Quality Index (PSQI) and Epworth Sleepiness Scale (ESS). Known-groups comparisons were made between those with and without self-reported sleep disorder.ResultsA sample of 337 women was analyzed. Unidimensionality and item-level fit to the GRM was supported; however, the local independence assumption was violated. The CAT1 algorithm showed 4.18 items on average, with a minor decrease in performance (higher RMSE value) compared to CAT2 or the PROMIS SD SF 8b. Administering 8 items adaptively (CAT2) compared to fixed (PROMIS SD SF 8b) performed similarly (RMSE difference = 0.001). Reliability exceeded 0.90 across most of the latent trait for all approaches. Correlations with the PSQI and ESS were largely as hypothesized, with minor differences in coefficient values between the approaches (all within 0.05). Women reporting a sleep disorder had greater sleep disturbance than those who did not (p < 0.001 for all).ConclusionsThe results of this study support using the PROMIS Sleep Disturbance item bank in postmenopausal women. The choice of PROMIS SD SF 8b versus CAT can largely be driven by practical reasons (respondent burden and operational complexity) rather than concerns of differential reliability and validity.

Development of computer adaptive tests to assess the psychological status of individuals with an eating disorder or type 2 diabetes

BackgroundIndividuals with type 2 diabetes and eating disorders must change their eating behaviors, which are often influenced by psychological factors like depression and anxiety. To efficiently assess daily psychological status, the present study aimed to develop computerized adaptive tests (CAT) based on item response theory (IRT).MethodsIndividuals with depression, anxiety disorders, eating disorders, type 2 diabetes, and healthy persons participated in the study. Participants completed six questionnaires, including momentary and most recent one-week depression, anxiety, and positive affect. We selected items meeting the IRT assumptions, applied a graded response model, and conducted CAT simulations.ResultsAcross all six questionnaires, the CAT simulations used a smaller number of items and exhibited substantial Pearson’s correlation coefficients exceeding 0.95 between simulated and full item-set mood status estimates. These estimated mood scores demonstrated satisfactory concurrent validity with the Hospital Anxiety and Depression Scale and sufficient discriminant validity between the clinical group and healthy controls.ConclusionThese findings suggest that these scales offer efficient measurement of the mood status of individuals with an eating disorder or type 2 diabetes.

Computer adaptive testing strategies for the Edinburgh Postnatal Depression Scale (EPDS).

Perinatal mood and anxiety disorders (PMADs) include depressive and anxiety disorders during pregnancy or postpartum and can have significant consequences for the parent, child, and family. When severe, these conditions can lead to suicide. Despite numerous policy efforts to improve screening, education, and referral structures, disparities in PMAD diagnosis and treatment still exists, particularly among racial and ethnic minorities. Computer Adaptive Testing (CAT) has been shown to improve the efficiency of screening by significantly reducing test length. This study evaluates whether applying CAT to the Edinburgh Postnatal Depression Scale (EPDS) maintains diagnostic accuracy while ensuring these methods do not exacerbate racial disparities in PMAD screening outcomes. Using real data simulation, we assessed three CAT-based short-form versions of the EPDS, derived from one-, two-, and three-factor item response theory models. We evaluated their diagnostic precision and examined potential racial disparities in false negative rates compared to the full-length EPDS. We demonstrate that estimated scores from three short versions of the EPDS administered through CAT-assuming one, two, and three-factor item response theory models-are more highly correlated with the full-length EPDS measure traditionally used to make clinical decisions (r's between 0.96 and 0.97) than the major depressive disorder subtest (CAT-MDD) from CAT-Mental Health (CAT-MH®) (r =.82), as previously reported. Importantly, the false negative rates of the CAT-implied diagnoses did not significantly vary between racial groups, indicating no evidence of racial bias in diagnostic accuracy. The CAT-based versions of the EPDS offers a promising solution for improving the efficiency of PMAD screening without sacrificing diagnostic precision or exacerbating racial groups. By reducing evaluation time, these tools could facilitate more widespread and equitable screening, enabling earlier diagnosis and treatment of PMADs across diverse populations.

Online Calibration for Multidimensional CAT With Polytomously Scored Items: A Neural Network–Based Approach

Online calibration is a key technology for calibrating new items in computerized adaptive testing (CAT). As multidimensional polytomous data become popular, online calibration methods applicable to multidimensional CAT with polytomously scored items (P-MCAT) have been proposed. However, the existing methods are mainly based on marginal MLE with an expectation-maximization algorithm (MMLE/EM), making it difficult to accurately estimate parameters in high-dimensional scenarios without sufficient calibration sample size or suitable initial values. To conquer these challenges, a neural network (NN)-based online calibration framework was put forward. The new NN-based methods differ profoundly from the traditional ones in that the parameter estimates of new items are obtained by learning the patterns between input and output data instead of finding solutions to the log-marginal likelihood. Moreover, an alternative solution was proposed for traditional methods to obtain appropriate initial values. Simulation studies were conducted to compare the NN- and MMLE/EM-based methods under various conditions, and further explore the properties of the NN-based methods. Results showed that both the NN-based methods and the alternative solution found their strengths in recovering the item parameters of new items, while the MMLE/EM-based methods struggled to converge when more than three dimensions were involved in the test.

Estimating minimal important change of the National Institutes of health research task force impact score using computer adaptive measures: a secondary analysis of two randomized clinical trials in a military population with chronic pain

BackgroundThe National Institutes of Health (NIH) Research Task Force (RTF) on Research Standards for Chronic Low Back Pain impact score is a composite measure of Patient Reported Outcomes Measurement Information System (PROMIS) pain intensity, pain interference and physical function. PROMIS surveys are available in short-form and computer adaptive testing (CAT) formats. Minimal important change (MIC) can be estimated to determine if between-group differences are large enough to be important. To date, three anchor-based estimates of impact score MIC ranging from 3 to 7.5 have been published, and all were based on data collected using PROMIS short-form surveys. None used CAT versions of PROMIS surveys.MethodsSecondary analysis of data collected during the conduct of two randomized clinical trials of 6-week courses of nonpharmacological pain therapies. Research subjects were US active-duty service members referred to an interdisciplinary pain management center. Impact score was assessed at the beginning and end of treatment. The Patient Global Impression of Change (PGIC) questionnaire was administered at the end of treatment and asked respondents to report their status compared to the start of treatment using a 7-item categorical scale ranging from very much improved to very much worse. A PGIC response of “much” or “very much” improved defined important improvement. Receiver operating characteristic (ROC) curve analysis and predictive logistic regression models were used to estimate MIC for the full combined sample and stratified by study sample and baseline impact score. Measures of individual statistical change were also computed.ResultsOverall, a decrease of 3 points in impact score was the estimated MIC (2.5 for ROC analysis and 3.4 for predictive modeling approach). Larger decreases in impact score were needed for participants with moderate and severe baseline pain impact to report important improvement. Thresholds for individual statistically significant change ranged from 6 to 14.ConclusionsUsing data collected with CAT surveys, we calculated an MIC of 3 points for the NIH RTF impact score, and estimates ranged from 1.3 to 7.2 depending on the baseline impact score and statistical approach used. These findings are consistent with previous MIC estimates that were based on non-adaptive short form surveys and have implications for improving the accuracy of pain treatment response assessment.Registry informationTrial registration. ClinicalTrials.gov. Registry numbers: NCT03297905 (registered 9/29/17) and NCT04656340 (registered 11/30/20). Link to full applications: https://classic.clinicaltrials.gov/ct2/show/NCT03297905?titles=Determinants+of+Optimal+Dosage%26cntry=US%26draw=2%26rank=1; https://classic.clinicaltrials.gov/ct2/show/results/NCT04656340?titles=Complementary+and+Integrative+pain+therapies+and+functional+restoration+%28IMPPPORT%29%26draw=2%26rank=1. Patient enrollment dates: SMART: 17 March 2021, prospectively registered; IMPPPORT: 9 December 2015, retrospectively registered.

Hopelessness as a Mediator of the Association Between Parenting Factors and Adolescent Suicidality and Substance Use Among Juvenile Justice‐Referred Youth

ABSTRACTYouth involved in the juvenile justice system are more likely to have a substance use disorder and/or suicidality (e.g., suicidal thoughts and behavior) compared to other youth. Although parental support and monitoring may play an important role in youth substance use and suicidality outcomes, the potential mechanisms have not been elucidated. Our purpose was to evaluate the extent to which parental support and monitoring were associated with latent, continuous construct scores of suicidality and substance use and to determine whether youths’ hopelessness may indirectly affect these relationships among a sample of youth referred to the juvenile justice system. The sample included juvenile justice‐referred youth aged 14–17 (N = 77; 69% White, 58% male, 74% non‐Hispanic). The primary predictors of interest were parental support and monitoring, measured by the Parent Support Scale and Parental Monitoring Scale. The primary potential mediator of interest was hopelessness. Linear regression was used to model continuous suicidality and substance use severity scores, measured via a computer adaptive test, on parental support and monitoring. We tested hopelessness as a potential mediator. All analyses controlled for age, sex assigned at birth, race, ethnicity, and family income. After adjustment, parental support was associated with decreased suicidality severity (β = −0.30, p = 0.002). Parent support and monitoring were associated with youth‐reported hopelessness. The indirect mediation effects of hopelessness in the relationship between parental support (β = −0.18 [SE, 1.73]), as well as parental monitoring (β = −0.17 [SE, 0.20]), and suicidality severity were statistically significant. Parental support and youth hopelessness may be important intervention targets for improving and addressing disparities in substance use and suicidality among juvenile justice‐referred youth. Hope‐based interventions may be effectively integrated into existing juvenile justice programs, and their potential to improve both mental health and behavioral outcomes among justice‐involved youth should be examined.

Development of item banks to assess financial hardship in cancer survivors using item response theory

BackgroundFinancial hardship is common after a cancer diagnosis. Current financial hardship measures have advanced the field, but assessing the dimensions of financial hardship remains challenging. We created item banks to assess four financial hardship dimensions using item response theory (IRT). IRT-based item banks can be tailored to each context and used in computerized adaptive testing (CAT) to reduce participant burden.MethodsCancer survivors (n = 459) were recruited from a survivorship program and online survey panel to complete an online or paper survey. Four item banks were developed based on previous studies, expert feedback, and patient interviews: financial coping: 41 items, financial consequences: 21 items, financial depression: 15 items, and financial worry: 21 items. We used the two-parameter logistic and graded response models for analysis.ResultsThe IRT model fit well for all four item banks: financial coping root mean square error of approximation (RMSEA) = 0.06, financial consequences RMSEA = 0.03; financial depression RMSEA = 0.05; and financial worry RMSEA = 0.03. The accuracy parameters ranged from 1.01 to 6.53, indicating good reliability for each item. The severity parameters showed each item bank assessed financial hardship across two to three standard deviations, supporting content validity. Short forms were developed for financial consequences, depression, and worry.ConclusionThe item banks can be used to create brief screening measures and, using CAT, efficiently screen for each dimension of financial hardship while minimizing burden. Future research is required to assess the clinical utility of using the item banks to screen for financial hardship.

Development and Initial Validation of the MCL-PRO-CAT: A computerized adaptive test designed to measure multifocal contact lens performance from the patient's perspective.

Development and Initial Validation of the MCL-PRO-CAT: A computerized adaptive test designed to measure multifocal contact lens performance from the patient's perspective.

A qualitative study exploring the feasibility and acceptability of computerised adaptive testing to assess and monitor children and young people’s mental health in primary care settings in the UK

BackgroundThe increasing prevalence of mental health disorders among adolescents highlights the importance of early identification and intervention. Artemis-A is a web-based application of computerised adaptive testing (CAT), originally developed for...

Psychometric properties and post-hoc CAT analysis of the pediatric PROMIS® item banks anxiety and depressive symptoms in a combined Swedish Child and Adolescent Psychiatry and School sample.

The objective of this study is to assess the psychometric properties and reliability of the Swedish Patient-Reported Outcomes Measurement Information System (PROMIS) item banksfor anxiety and depressive symptoms with item response theory analysis and post-hoc computerized adaptive testing in a combined Swedish Child and Adolescent Psychiatry (CAP) and school sample. Participants (n = 928, age 12-20) were recruited from junior and high schools and Child and Adolescent Psychiatry Clinics in the region of Västerbotten. Unidimensionality, local independence, and monotonicity was tested. We fitted a graded response model to the data and tested differential item functioning (DIF) for sex, age group, sample type, and language (Swedish vs. U.S.). Moreover, a post-hoc computer adaptive testing (CAT) simulation was performed. All analysis were made in R. Unidimensionality, local independence, and monotonicity were acceptable. The graded response model yielded acceptable item fit, discriminative, and threshold values for all items in both item banks. DIF for language (Swedish vs. U.S.) was found for two items from the anxiety and one item from the depressive symptoms item banks. A Stocking-lord transformation was used for the items displaying language DIF, and post-hoc CAT simulations were performed. The post-hoc CAT simulation showed reliability around 0.9 for both Swedish and official U.S. item parameters T-scores calibration from within normal limits to severe anxiety and depressive symptoms. The Swedish pediatric PROMIS item banks of anxiety and depressive symptoms are appropriate to assess mild to severe symptoms of anxiety and depressive symptoms in Swedish school- and CAP samples.

Associations of Pretransplant Patient-Reported Outcomes Measurement Information System Physical Function Score With Kidney Transplant Outcomes.

Simple and validated physical function measures are needed for kidney transplant candidates because pretransplant low physical function is a common and potentially modifiable risk factor. This single-center retrospective study investigated the associations between pretransplant physical function assessed by the Patient-Reported Outcomes Measurement Information System® Physical Function (PROMIS-PF) computer adaptive testing and early posttransplant outcomes. We analyzed 154 adult kidney-alone transplant recipients. The median pretransplant PROMIS-PF score was 43 (interquartile range, 39-47). Patient characteristics were not significantly different across the score category (normal, score ≥45; mild, score of 40-45; and moderate/severe, score <40). The PROMIS-PF score was not associated with length of transplant hospital stay, delayed graft function, 6-month and 12-month graft function, or 12-month patient and graft survival. However, a lower PROMIS-PF score was significantly associated with a higher risk of emergency room visits [adjusted odds ratios compared to normal: mild, 1.68 (95% confidence interval, 0.76-3.83); moderate/severe, 3.23 (1.34-7.79)] and rehospitalization [adjusted odds ratios: mild, 2.61 (1.16-5.90); moderate/severe, 2.53 (1.07-6.00)] within 1month posttransplant. Results suggest that PROMIS-PF is a practical tool for assessing physical function in kidney transplant candidates. Larger studies are needed to confirm the utility of PROMIS-PF to identify transplant candidates who would benefit from pretransplant prehabilitation.

The impact of computerized adaptive test termination rules on accuracy across different ability estimation methods

The study aimed to compare the termination rules of computerized adaptive testing (CAT), specifically the rule of termination after a fixed number of items versus the rule of termination based on the minimum standard error (SE), using the methods of maximum likelihood estimation (MLE) and maximum a posteriori. The goal was to assess the relative accuracy of each rule to determine which method provides the highest measurement accuracy. In order to address the objectives of the study, the researcher developed a mathematics item bank for the 6&lt;sup&gt;th&lt;/sup&gt; and 7&lt;sup&gt;th&lt;/sup&gt; grades consisting of 275 items. This bank was used to develop 6 achievement tests, three for 6th grade and three for 7&lt;sup&gt;th&lt;/sup&gt; grade, with each test comprising 46 items. In addition, 10 items were common to all the tests and were used as a common core. The tests were conducted with 2612 students of class six and seven. For data and information analysis and processing, BILOG-MG-3.0, SPSS, and Fast Test Web v3.80.26 applications were used. Four different applications were performed on another sample of 403 students in order to evaluate the precision of the CAT procedure through several terminating criteria–a fixed criterion of 25 administered items and a criterion based on a SE of no larger than 0.25. From the results obtained, the accuracy of examinations is independent of the method used in estimating the parameters and that the determination of the fixed period is superior to the determination relating to SE. Moreover, the results revealed that adopting Bayes’ theorem and a termination rule determined by standard deviation improves precision of the estimation, though in a case where 25 items are used, MLE is the best.

Comparing Proctored and Unproctored Cognitive Ability Testing in High‐Stakes Personnel Selection

ABSTRACTNew advances in computerized adaptive testing (CAT) have increased the feasibility of high‐stakes unproctored testing of general mental ability (GMA) in personnel selection contexts. This study presents the results from a within‐subject investigation of the convergent validity of unproctored tests. Three batteries of cognitive ability tests were administered during personnel selection in the Norwegian Armed Forces. A total of 537 candidates completed two sets of proctored fixed‐length GMA tests before and during the selection process. In addition, an at‐home unproctored CAT battery of tests was administered before the selection process began. Differences and similarities between the convergent validity of the tests were evaluated. The convergent validity coefficients did not significantly differ between proctored and unproctored batteries, both on observed GMA scores and the latent factor level. The distribution and standardized residuals of test scores comparing proctored‐proctored and proctored‐unproctored were overall quite similar and showed no evidence of score inflation or deflation in the unproctored tests. The similarities between proctored and unproctored results also extended to the moderately searchable words similarity test. Although some unlikely individual cases were observed, the overall results suggest that the unproctored tests maintained their convergent validity.

Exploring the effects of small item pools on examinee achievement estimates for computer-adaptive tests: a simulation study

We explored the practicality of relatively small item pools in the context of low-stakes Computer-Adaptive Testing (CAT), such as CAT procedures that might be used for quick diagnostic or screening exams. We used a basic CAT algorithm without content balancing and exposure control restrictions to reflect low stakes testing scenarios. We examined the effects of small item pools under various testing conditions using a series of Monte Carlo simulations. We examined the effects of these conditions on the accuracy, precision, and stability of examinee achievement estimates. Our results showed that the effects of item pool size are strongest when there is less-precise ­targeting between item and person location parameters. Our findings suggest that small item pools can effectively support satisfactory performance in CAT, particularly when examinee targeting is adequate and a variable-length test with a standard error stopping rule is implemented. We consider implications for research and practice.