Computerized Adaptive Test Administration Research Articles

Lower Quadrant Edema Patient-Reported Outcome Measure Is Reliable, Valid, and Efficient for Patients with Lymphatic and Venous Disorders.

The main aims of this study were: (1) to create a patient-reported outcome measure (PROM) item bank for measuring the impact of lower quadrant edema (LQE) on physical function using item response theory and (2) to assess reliability, validity, and administration efficiency of LQE PROM scores based on computerized adaptive test (CAT) and the reliability of a 10-item short form (SF). This retrospective study included data from patients treated in outpatient rehabilitation clinics for lower quadrant edema who responded to all 30 candidate items at intake. Item response theory model assumptions of unidimensionality, local item independence, item fit, and presence of differential item functioning were evaluated. LQE-CAT-generated scores were assessed for reliability, validity, and administration efficiency. LQE-SF-generated scores were assessed for reliability. The total cohort included 4894 patients (mean [SD] age = 65 [14] y; range = 14-89 y). A set of 20 items was selected for the item bank based on support for its unidimensionality and fit to the item response theory model, with reliability estimates greater than 0.92 for CAT and SF administration modes. No items demonstrated differential item functioning with respect to tested variables. After controlling for scores at intake, scores discriminated among multiple patient groups in clinically logical ways with better outcomes observed for patients who were younger with less chronic symptoms and fewer comorbidities. Scores were responsive to change but the effect size was small (0.4). There were negligible floor and ceiling effects. CAT administration of the item bank required an average of 6.1 items (median = 5). Scores correlated highly with full-bank scores (Pearson correlation coefficient = 0.98). Scores on the LQE PROM were reliable, valid, and efficient for assessing perceived physical function of patients with lower quadrant edema. The LQE CAT and SF are suitable for research and routine clinical care. Reasons for the small effect size for change scores should be studied. The newly developed LQE PROM was reliable and valid and offers efficient administration modes for assessing perceived physical function of patients with LQE, both for research and routine clinical care in busy outpatient rehabilitation settings. As an item response theory-based measure, the LQE PROM allows administration of condition-specific functional questions with low response burden for patients. The 10-item LQE-SF offers a feasible alternative administration mode when CAT administration is not available. This study supports a transition to PROMs that are based on modern measurement approaches to achieve the combined benefits of high accuracy and efficiency.

Development of the PREMIUM computerized adaptive testing for measuring the access and care coordination for patients with severe mental illness

Severe mental illness (SMI) patients often have complex health needs, which makes it difficult to access and coordinate their care. This study aimed to develop a computerized adaptive testing (CAT) tool, PREMIUM CAT-ACC, to measure SMI patients' experience with access and care coordination. This multicenter and cross-sectional study included 496 adult in- and out-patients with SMI (i.e., schizophrenia, bipolar disorder, or major depressive disorder). Psychometric analysis of the 13-item bank showed adequate properties, with preliminary evidence of external validity and no substantial differential item functioning for sex, age, care setting, and diagnosis, making it suitable for CAT administration. A post-hoc CAT simulation demonstrated that the tool was efficient and accurate, with an average of seven items, compared to the full item bank administration. Its use by clinicians can contribute to optimizing patient care pathways and transitioning towards more person-centered healthcare.

The Stroke Upper and Lower Extremity Physical Function Measures Were Supported for Score Reliability, Validity, and Administration Efficiency for Patients Poststroke.

The aims of this study were to (1) evaluate the suitability of newly developed items for calibration into 2 item banks for stroke upper extremity (SUE) and stroke lower extremity (SLE) physical function patient-reported outcome measures (PROMs) and (2) assess score reliability and validity and PROM administration efficiency based on computerized adaptive testing (CAT). A retrospective longitudinal study involving patients poststroke who were treated in outpatient rehabilitation clinics responded to 28 and 25 region-specific candidate items addressing tasks related to upper or lower extremity physical function, respectively. Item response theory model assumptions of unidimensionality, local independence, item fit, and presence of differential item functioning were evaluated. CAT-generated scores were assessed for reliability, validity, and administration efficiency, and 10-item short forms were assessed for reliability. Cohorts consisted of 2017 patients with stroke involving the upper extremity and 2107 patients with stroke involving the lower extremity (mean age [SD]: SUE = 62 [14] and SLE = 63 [14]; range = 14-89). Two solutions (SUE: 28-item; SLE: 24-item) supported unidimensionality and fit to the item response theory model, with reliability estimates >0.93 for all administration modes. No items demonstrated differential item functioning. Scores discriminated among multiple patient groups in clinically logical ways, with better outcomes observed for patients who were younger, were male, had less chronicity, and had fewer comorbidities. The SUE and SLE, respectively, had 1% and 0.3% floor effects and 4.3% and 1.1% ceiling effects. Change score effect sizes were 0.5 (SUE) and 0.6 (SLE). Simulated CAT scores required an average of 6 (SUE) and 5.6 (SLE) items (median = 5). The stroke upper extremity and stroke lower extremity PROM scores were reliable, valid, and efficient and had moderate change effect sizes for assessing physical function as perceived by patients poststroke with upper and lower extremity impairments. Scores had negligible floor and acceptable ceiling effects. Based on these results, the stroke PROMs are suitable for research and routine clinical practice. As item response theory-based measures, these PROMs support clinical practice guideline recommendations for the use of outcome measures in neurologic physical therapy and the administration of condition-specific functional questions with low response burden for patients. The 10-item short forms offer a feasible alternative administration mode when CAT administration is not available.

Feasibility of PROMIS using computerized adaptive testing during inpatient rehabilitation

BackgroundThere has been an increased significance on patient-reported outcomes in clinical settings. We aimed to evaluate the feasibility of administering patient-reported outcome measures by computerized adaptive testing (CAT) using a tablet computer with rehabilitation inpatients, assess workload demands on staff, and estimate the extent to which rehabilitation inpatients have elevated T-scores on six Patient Reported Outcomes Measurement Information System® (PROMIS®) measures.MethodsPatients (N = 108) with stroke, spinal cord injury, traumatic brain injury, and other neurological disorders participated in this study. PROMIS computerized adaptive tests (CAT) were administered via a web-based platform. Summary scores were calculated for six measures: Pain Interference, Sleep Disruption, Anxiety, Depression, Illness Impact Positive, and Illness Impact Negative. We calculated the percent of patients with T-scores equivalent to 2 standard deviations or greater above the mean.ResultsDuring the first phase, we collected data from 19 of 49 patients; of the remainder, 61% were not available or had cognitive or expressive language impairments. In the second phase of the study, 40 of 59 patients participated to complete the assessment. The mean PROMIS T-scores were in the low 50 s, indicating an average symptom level, but 19–31% of patients had elevated T-scores where the patients needed clinical action.ConclusionsThe study demonstrated that PROMIS assessment using a CAT administration during an inpatient rehabilitation setting is feasible with the presence of a research staff member to complete PROMIS assessment.

Computerized-adaptive testing versus short forms for pediatric inflammatory bowel disease patient-reported outcome assessment.

Computerized-adaptive testing (CAT) may increase reliability or reduce respondent burden for assessing patient-reported outcomes compared with static short forms (SFs). We compared CAT versus SF administration of the Patient-Reported Outcomes Measurement Information System® (PROMIS®) Pediatric measures in pediatric inflammatory bowel disease (IBD). Participants completed 4-item CAT, 5- or 6-item CAT, and 4-item SF versions of the PROMIS Pediatric measures. We compared average T-scores, intra-class correlations (ICCs), floor and ceiling effects, and standard error of measurement (SEM) across forms, along with mean effect sizes between active versus quiescent IBD disease activity groups. Average PROMIS T-scores across forms were <3 points (minimally important difference) of each other. All forms correlated highly with each other (ICCs ≥0.90) and had similar ceiling effects, but the CAT-5/6 had lower floor effects. The CAT-5/6 had lower SEM than the CAT-4 and SF-4, and the CAT-4 had a lower SEM than the SF-4. Mean effect sizes were similar across forms when contrasting disease activity groups. The CAT and SF forms produced similar score results, but the CAT had better precision and lower floor effects. Researchers should consider PROMIS pediatric CAT if they anticipate that their sample will skew toward symptom extremes.

The Effect of Computerized Adaptive Testing on Motivation and Anxiety: A Systematic Review and Meta-Analysis.

Although many studies have been carried out on the psychometric aspects of computerized adaptive testing (CAT), its psychological aspects are less researched. Early studies claimed that CAT can be more motivating and induce less anxiety than traditional fixed-item tests (FIT). The purpose of this systematic review and meta-analysis was to gain a comprehensive understanding of the effects of CAT on motivation and anxiety in comparison to traditional fixed-item testing. Seven databases were examined. Articles were eligible if they employed an empirical study containing a direct comparison between CAT and FIT. Meta-analytical results showed no overall effect of test type on anxiety and motivation when comparing CAT with FIT (k = 11, g+ = 0.06, p = .28). However, easier CAT had positive effect compared with FIT (k = 2, g+ = .22, p < .001). Certain modifications in CAT administration can provide positive psychological effects for test-takers.

Upper Quadrant Edema Patient-Reported Outcome Measure Is Reliable, Valid, and Efficient for Patients With Lymphatic and Venous Disorders.

The main aims of this study were to (1) create a patient-reported outcome measure (PROM) item bank for measuring the impact of upper quadrant edema (UQE) on physical function by calibrating responses to newly developed items; and (2) assess reliability, validity, and administration efficiency of scores based on computerized adaptive test (CAT) and 10-item short-form (SF) administration modes. This was a retrospective study including data from patients treated in outpatient rehabilitation clinics for UQE that responded to all 27 candidate items at intake. Item response theory model assumptions of unidimensionality, local item independence, item fit, and presence of differential item functioning were evaluated. UQE-CAT- and UQE-SF-generated scores were assessed for reliability, validity, and administration efficiency. The total cohort included 3486 patients (mean [SD] age = 61 [13] years; range = 14-89 years). After removing 2 items, a 25-item solution was supported for its unidimensionality and fit to the item response theory model with reliability estimates of more than 0.93 for scores based on both CAT and SF administration modes. No items demonstrated differential item functioning. Scores discriminated among multiple patient groups in clinically logical ways and were moderately responsive to change with negligible floor and acceptable ceiling effects. CAT scores were generated using an average of 5.6 items (median = 5). Scores on the UQE PROM were reliable, valid, and efficient for assessing perceived physical function of patients with upper quadrant edema; thus, the measure is suitable for research and routine clinical administration. The newly developed UQE PROM is reliable and valid and offers efficient administration modes for assessing perceived physical function of patients with UQE caused by lymphatic and venous disorders, both for research and routine clinical care in busy outpatient rehabilitation settings. As an item response theory-based measure, the UQE PROM allows administration of condition-specific functional questions with low response burden for patients. This study supports a transition to PROMs that are based on modern measurement approaches to achieve high accuracy and efficiency.

Hybrid Threshold-Based Sequential Procedures for Detecting Compromised Items in a Computerized Adaptive Testing Licensure Exam.

Using classical test theory and item response theory, this study applied sequential procedures to a real operational item pool in a variable-length computerized adaptive testing (CAT) to detect items whose security may be compromised. Moreover, this study proposed a hybrid threshold approach to improve the detection power of the sequential procedure while controlling the Type I error rate. The hybrid threshold approach uses a local threshold for each item in an early stage of the CAT administration, and then it uses the global threshold in the decision-making stage. Applying various simulation factors, a series of simulation studies examined which factors contribute significantly to the power rate and lag time of the procedure. In addition to the simulation study, a case study investigated whether the procedures are applicable to the real item pool administered in CAT and can identify potentially compromised items in the pool. This research found that the increment of probability of a correct answer (p-increment) was the simulation factor most important to the sequential procedures' ability to detect compromised items. This study also found that the local threshold approach improved power rates and shortened lag times when the p-increment was small. The findings of this study could help practitioners implement the sequential procedures using the hybrid threshold approach in real-time CAT administration.

Computerized Quality of Life Assessment: A Randomized Experiment to Determine the Impact of Individualized Feedback on Assessment Experience.

BackgroundQuality of life (QoL) assessments, or patient-reported outcome measures (PROMs), are becoming increasingly important in health care and have been associated with improved decision making, higher satisfaction, and better outcomes of care. Some physicians and patients may find questionnaires too burdensome; however, this issue could be addressed by making use of computerized adaptive testing (CAT). In addition, making the questionnaire more interesting, for example by providing graphical and contextualized feedback, may further improve the experience of the users. However, little is known about how shorter assessments and feedback impact user experience.ObjectiveWe conducted a controlled experiment to assess the impact of tailored multimodal feedback and CAT on user experience in QoL assessment using validated PROMs.MethodsWe recruited a representative sample from the general population in the United Kingdom using the Oxford Prolific academic Web panel. Participants completed either a CAT version of the World Health Organization Quality of Life assessment (WHOQOL-CAT) or the fixed-length WHOQOL-BREF, an abbreviated version of the WHOQOL-100. We randomly assigned participants to conditions in which they would receive no feedback, graphical feedback only, or graphical and adaptive text-based feedback. Participants rated the assessment in terms of perceived acceptability, engagement, clarity, and accuracy.ResultsWe included 1386 participants in our analysis. Assessment experience was improved when graphical and tailored text-based feedback was provided along with PROMs (Δ=0.22, P<.001). Providing graphical feedback alone was weakly associated with improvement in overall experience (Δ=0.10, P=.006). Graphical and text-based feedback made the questionnaire more interesting, and users were more likely to report they would share the results with a physician or family member (Δ=0.17, P<.001, and Δ=0.17, P<.001, respectively). No difference was found in perceived accuracy of the graphical feedback scores of the WHOQOL-CAT and WHOQOL-BREF (Δ=0.06, P=.05). CAT (stopping rule [SE<0.45]) resulted in the administration of 25% fewer items than the fixed-length assessment, but it did not result in an improved user experience (P=.21).ConclusionsUsing tailored text-based feedback to contextualize numeric scores maximized the acceptability of electronic QoL assessment. Improving user experience may increase response rates and reduce attrition in research and clinical use of PROMs. In this study, CAT administration was associated with a modest decrease in assessment length but did not improve user experience. Patient-perceived accuracy of feedback was equivalent when comparing CAT with fixed-length assessment. Fixed-length forms are already generally acceptable to respondents; however, CAT might have an advantage over longer questionnaires that would be considered burdensome. Further research is warranted to explore the relationship between assessment length, feedback, and response burden in diverse populations.

Calibrating a new item pool to adaptively assess the Big Five.

Even though the Five Factor Model (FFM) has been the dominant paradigm in personality research for the past two decades, very few studies have measured the FFM adaptively. Thus, the purpose of this research was the building of a new item pool to develop a computerized adaptive test (CAT) for personality assessment. A pool of 480 items that measured the FFM facets was developed and applied to 826 participants. Facets were calibrated separately and item selection was performed being mindful of the preservation of unidimensionality of each facet. Then, a post-hoc simulation study was carried out to test the performance of separate CATs to measure the facets. The final item pool was composed of 360 items with good psychometric properties. Findings reveal that a CAT administration of four items per facet (total length of 120 items) provides accurate facets scores, while maintaining the factor structure of the FFM. An item pool with good psychometric properties was obtained and a CAT simulation study demonstrated that the FFM facets could be measured with precision using a third of the items in the pool.

Can Computerized Adaptive Testing Work in Students’ Admission to Higher Education Programs in Turkey?

Admission into university in Turkey is very competitive and features a number of practical problems regarding not only the test administration process itself, but also concerning the psychometric properties of test scores. Computerized adaptive testing (CAT) is seen as a possible alternative approach to solve these problems. In the first phase of the study, a series of CAT simulations based on real students’ responses to science items were conducted in order to determine which test termination rule produced more comparable results with scores made on the paper and pencil version of the test. An average of 17 items was used to terminate the CAT administration for a reasonable reliability level as opposed to the normal 45 items. Moreover, CAT based science scores not only produced similar correlations when using mathematics subtest scores as an external criterion, but also ranked the students similarly to the paper and pencil test version. In the second phase, a live CAT administration was implemented using an item bank composed of 242 items with a group of students who had previously taken the exam the paper and pencil version of the test. A correlation of .76 was found between the CAT and paper and pencil scores for this group. The results seem to support the CAT version of the subtests as a feasible alternative approach in Turkey’s university admission system.

Computerized adaptive testing of population psychological distress: simulation-based evaluation of GHQ-30.

PurposeGoldberg’s General Health Questionnaire (GHQ) items are frequently used to assess psychological distress but no study to date has investigated the GHQ-30’s potential for adaptive administration. In computerized adaptive testing (CAT) items are matched optimally to the targeted distress level of respondents instead of relying on fixed-length versions of instruments. We therefore calibrate GHQ-30 items and report a simulation study exploring the potential of this instrument for adaptive administration in a longitudinal setting.MethodsGHQ-30 responses of 3445 participants with 2 completed assessments (baseline, 7-year follow-up) in the UK Health and Lifestyle Survey were calibrated using item response theory. Our simulation study evaluated the efficiency of CAT administration of the items, cross-sectionally and longitudinally, with different estimators, item selection methods, and measurement precision criteria.ResultsTo yield accurate distress measurements (marginal reliability at least 0.90) nearly all GHQ-30 items need to be administered to most survey respondents in general population samples. When lower accuracy is permissible (marginal reliability of 0.80), adaptive administration saves approximately 2/3 of the items. For longitudinal applications, change scores based on the complete set of GHQ-30 items correlate highly with change scores from adaptive administrations.ConclusionsThe rationale for CAT-GHQ-30 is only supported when the required marginal reliability is lower than 0.9, which is most likely to be the case in cross-sectional and longitudinal studies assessing mean changes in populations. Precise measurement of psychological distress at the individual level can be achieved, but requires the deployment of all 30 items.

Content-balancing strategy in bifactor computerized adaptive patient-reported outcome measurement

Most multidimensional patient-reported outcomes (PRO) measures are lengthy to complete. Computerized adaptive testing (CAT) that selects the most informative items can potentially reduce respondent burden without sacrificing measurement accuracy. The commonly used maximum Fisher information item selection method has been reported to lead to highly unbalanced item bank usage and potentially imprecise trait estimation. This study employs the content-balancing strategy in a bifactor-modeled CAT item selection and examines its impact on measurement accuracy and item bank usage. Item responses from a population-based SF-36 survey were first calibrated using the bifactor graded response model. Four post hoc CATs using items and responses from the SF-36 data set were then created. The content-balancing strategy was adopted in the item selection procedure of the bifactor-modeled CAT. The measurement accuracy and usage of items of the CAT were compared between the tests with and without the content-balancing strategy. The results indicate that the CAT implemented with the content-balancing strategy offers a better overall measurement accuracy of both the general health status and the two health domains (physical and mental) of the SF-36. The content-balancing strategy helps the CAT-PRO to balance the selection of items and achieve improved measurement accuracy. Its implementation in real-time CAT administration to measure multidimensional PRO traits merits further studies.

Migrating from a legacy fixed-format measure to CAT administration: calibrating the PHQ-9 to the PROMIS depression measures.

We provide detailed instructions for analyzing patient-reported outcome (PRO) data collected with an existing (legacy) instrument so that scores can be calibrated to the PRO Measurement Information System (PROMIS) metric. This calibration facilitates migration to computerized adaptive test (CAT) PROMIS data collection, while facilitating research using historical legacy data alongside new PROMIS data. A cross-sectional convenience sample (n=2,178) from the Universities of Washington and Alabama at Birmingham HIV clinics completed the PROMIS short form and Patient Health Questionnaire (PHQ-9) depression symptom measures between August 2008 and December 2009. We calibrated the tests using item response theory. We compared measurement precision of the PHQ-9, the PROMIS short form, and simulated PROMIS CAT. Dimensionality analyses confirmed the PHQ-9 could be calibrated to the PROMIS metric. We provide code used to score the PHQ-9 on the PROMIS metric. The mean standard errors of measurement were 0.49 for the PHQ-9, 0.35 for the PROMIS short form, and 0.37, 0.28, and 0.27 for 3-, 8-, and 9-item-simulated CATs. The strategy described here facilitated migration from a fixed-format legacy scale to PROMIS CAT administration and may be useful in other settings.

Computerized Adaptive Test for Patients With Lumbar Spine Impairments Produced Valid and Responsive Measures of Function

Outcomes instrument validation study. We evaluated administrative efficiency and psychometric adequacy of a computerized adaptive test (CAT) for patients with lumbar spine impairments seeking rehabilitation in outpatient therapy clinics. CATs promise efficient outcomes data collection in clinical applications with little loss of measurement precision compared to paper and pencil surveys. The lumbar CAT has been developed and simulated and is currently used routinely in therapy clinics. The CAT has not been assessed for administrative efficiency, and the outcomes measures estimated using the CAT have not been assessed using prospective data collection for validity, sensitivity to change, or responsiveness. Data from 17,439 patients with lumbar spine impairments receiving outpatient rehabilitation in 377 clinics in 30 states (United States) were analyzed. We evaluated efficiency of routine CAT administration and assessed construct validity, sensitivity to change, and responsiveness of CAT measures of lumbar functional status (FS). On average, patients took less than 2 minutes (standard deviation <1 minute) to answer 7 CAT items (standard deviation, 3), which produced precise estimates of FS that adequately covered the content range with negligible floor and ceiling effects. Patients who were older had more chronic symptoms, had more surgeries, had more comorbidities, and did not exercise before receiving rehabilitation reported worse discharge FS. A total of 66% of patients obtained statistically significant change (95% confidence interval minimal detectable change) at discharge. Change of 5 FS units (scale, 0-100) represented minimal clinically important improvement, which 70% of patients obtained. Minimal detectable change and minimal clinically important improvement were associated with intake FS. We concluded the lumbar CAT administration was efficient, and CAT FS measures were precise, valid, sensitive, and responsive, supporting lumbar CAT use in clinical and research applications.

Clinical Interpretation of Computerized Adaptive Test–Generated Outcome Measures in Patients With Knee Impairments

Wang Y-C, Hart DL, Stratford PW, Mioduski JE. Clinical interpretation of computerized adaptive test–generated outcome measures in patients with knee impairments. Objective To describe meaningful interpretations of functional status (FS) outcomes measures estimated using a body-part specific computerized adaptive test (CAT). Design A prospective observational cohort study. Setting Outpatient physical therapy clinics (291 clinics) in 30 U.S. states. Participants Sample of 21,896 patients with knee impairments receiving outpatient physical therapy. Interventions Not applicable. Main Outcome Measure FS estimated using CAT administration. Results We investigated 4 approaches to clinically meaningful interpretations of outcomes data: (1) 95% confidence interval for each score estimate, (2) percentile rank of FS scores, (3) responsiveness, and (4) functional staging. Overall, precision of a single score was estimated by FS score ±5. Based on score distribution, percentile ranks at 25th, 50th, and 75th percentiles corresponded to intake FS scores of 33, 42, and 51 and discharge FS scores of 51, 61, and 74, respectively. Results showed that 9 or higher FS change units represented statistically and clinically important improvement. Patients were classified into 6 hierarchical levels of FS using functional staging. Conclusions Results suggest how CAT-generated outcomes measures can be interpreted to assist clinicians and patients during rehabilitation.

Computerized adaptive test for patients with foot or ankle impairments produced valid and responsive measures of function

We tested the item response theory (IRT) model assumptions of the original item bank, and evaluated the practical and psychometric adequacy, of a computerized adaptive test (CAT) for patients with foot or ankle impairments seeking rehabilitation in outpatient therapy clinics. Data from 10,287 patients with foot or ankle impairments receiving outpatient physical therapy were analyzed. We first examined the unidimensionality, fit, and invariance IRT assumptions of the CAT item bank. Then we evaluated the efficiency of the CAT administration and construct validity and sensitivity of change of the foot/ankle CAT measure of lower-extremity functional status (FS). Results supported unidimensionality, model fit, and invariance of item parameters and patient ability estimates. On average, the CAT used seven items to produce precise estimates of FS that adequately covered the content range with negligible floor and ceiling effects. Patients who were older, had more chronic symptoms, had more surgeries, had more comorbidities, and did not exercise prior to receiving rehabilitation reported worse discharge FS. Seventy-one percent of patients obtained statistically significant change at follow-up. Change of 8 FS units (scale 0-100) represented minimal clinically important improvement. We concluded that the foot/ankle item bank met IRT assumptions and that the CAT FS measure was precise, valid, and responsive, supporting its use in routine clinical application.

(886): How many questions does it take to obtain valid and precise estimates of the impact of pain on a patient’s life?

The assessment of pain and its impact requires reliable and valid patient-based assessment tools. Computerized adaptive testing (CAT) is a novel approach to pain assessment. This study examined the precision and validity of pain-impact scores and response burden from simulated CAT administration of a bank of 45 items. Adults (n=1,500) randomly sampled from the general US population via telephone or internet completed a questionnaire consisting of pain items from widely-used instruments (SF-36, Nottingham Health Profile, Sickness Impact Profile, McGill Pain, Oswestry Low Back Pain, Brief Pain Index, Aberdeen Back Pain, EuroQOL, Health Insurance Experiment, Fibromyalgia Impact Questionnaire, Short MusculoSkeletal Function Assessment). A 45-item bank meeting the assumptions of an item response theory (IRT) model was calibrated on a common metric and a series of CAT administrations with varying fixed lengths (2, 5 and 7 items) or varying precision levels were simulated. Each simulated CAT score was compared to a score (theta) based on the entire item bank. High correlations were observed between the CAT estimates and the theta score (.90-.99). The theta score showed the best measurement precision (95%CI:1.6-15.6) at all score levels; the CAT-2 showed the least precision (95%CI:5.8-14.8); the CAT-5 (95%CI:3.3-12.6) and CAT-7 (95%CI:2.8-12.7) provided acceptable precision levels, while dramatically decreasing the length of the measure. All assessments were most accurate in the middle range of the scale and showed substantial increases in measurement error towards the end points. Taking into account the variations in precision observed across CAT simulations, on average 5-6 items were needed to reach acceptable levels of precision, representing an 89% reduction in respondent burden. All CATs significantly discriminated (p<.001) between people with and without chronic pain in the hypothesized way. The results of this study suggest that it takes 5-6 tailored items to produce a valid and precise patient-reported score of pain impact. The assessment of pain and its impact requires reliable and valid patient-based assessment tools. Computerized adaptive testing (CAT) is a novel approach to pain assessment. This study examined the precision and validity of pain-impact scores and response burden from simulated CAT administration of a bank of 45 items. Adults (n=1,500) randomly sampled from the general US population via telephone or internet completed a questionnaire consisting of pain items from widely-used instruments (SF-36, Nottingham Health Profile, Sickness Impact Profile, McGill Pain, Oswestry Low Back Pain, Brief Pain Index, Aberdeen Back Pain, EuroQOL, Health Insurance Experiment, Fibromyalgia Impact Questionnaire, Short MusculoSkeletal Function Assessment). A 45-item bank meeting the assumptions of an item response theory (IRT) model was calibrated on a common metric and a series of CAT administrations with varying fixed lengths (2, 5 and 7 items) or varying precision levels were simulated. Each simulated CAT score was compared to a score (theta) based on the entire item bank. High correlations were observed between the CAT estimates and the theta score (.90-.99). The theta score showed the best measurement precision (95%CI:1.6-15.6) at all score levels; the CAT-2 showed the least precision (95%CI:5.8-14.8); the CAT-5 (95%CI:3.3-12.6) and CAT-7 (95%CI:2.8-12.7) provided acceptable precision levels, while dramatically decreasing the length of the measure. All assessments were most accurate in the middle range of the scale and showed substantial increases in measurement error towards the end points. Taking into account the variations in precision observed across CAT simulations, on average 5-6 items were needed to reach acceptable levels of precision, representing an 89% reduction in respondent burden. All CATs significantly discriminated (p<.001) between people with and without chronic pain in the hypothesized way. The results of this study suggest that it takes 5-6 tailored items to produce a valid and precise patient-reported score of pain impact.

Design and Evaluation of an XML-Based Platform-Independent Computerized Adaptive Testing System

This paper is an attempt to design and to evaluate the platform-independent computerized adaptive testing (CAT) system, which can expand the diversity of CAT-administering platforms. By using extensible markup language (XML) to describe the item bank, one might find the implementation of CAT on a different platform, such as a personal computer (PC), a personal digital assistant (PDA), and other handheld devices, more convenient. An experiment was conducted to examine the effects of a CAT administration platform on precision and efficiency. Fifty senior high school students were selected to take an English vocabulary CAT both on PC and PDA, which enabled them to compare the relevant advantages with the disadvantages of the two different administration platforms firsthand. Both tests used the same-size, well-calibrated item bank, ability estimation algorithm, and item selection strategy. The results indicate that the platforms on which examinees take CAT do not affect the performance of CAT. The responses of the questionnaire on the testing environment also show that most examinees prefer to take the test on PDA. It is concluded that using a PDA to administer CAT is both as precise and effective as a PC and more enjoyable and convenient.

Computerization and Adaptive Administration of the NEO PI-R

This study asks, how well does an item response theory (IRT) based computerized adaptive NEO PI-R work? To explore this question, real-data simulations (N = 1,059) were used to evaluate a maximum information item selection computerized adaptive test (CAT) algorithm. Findings indicated satisfactory recovery of full-scale facet scores with the administration of around four items per facet scale. Thus, the NEO PI-R could be reduced in half with little loss in precision by CAT administration. However, results also indicated that the CAT algorithm was not necessary. We found that for many scales, administering the "best" four items per facet scale would have produced similar results. In the conclusion, we discuss the future of computerized personality assessment and describe the role IRT methods might play in such assessments.