Simulated Computer Adaptive Tests Research Articles

Measuring Self-Reported Physical Function in Individuals With TBI: Development of the TBI-QOL Mobility and Upper Extremity Item Banks and Short Forms.

To describe the development and field testing of the patient-reported outcome measures of Mobility and Upper Extremity function from the Traumatic Brain Injury Quality of Life (TBI-QOL) measurement system, and to evaluate the use of computer adaptive testing. Five rehabilitation facilities funded as part of the TBI Model Systems network. Individuals with complicated mild, moderate, or severe traumatic brain injury (n = 590). Not available. TBI-QOL Mobility and Upper Extremity item banks. Item response theory and factor analyses supported the unidimensionality of the Mobility and Upper Extremity banks. Descriptive statistics showed a ceiling effect for both measures. Simulated computer adaptive tests (CATs) showed that measurement precision was maintained across administration formats for both measures. The Upper Extremity CAT showed a loss of precision for individuals without impairment and that a higher number of items were required to achieve sufficiently precise measurement, compared to the Mobility CAT. The TBI-QOL Upper Extremity and Mobility item banks achieved good breadth of coverage, particularly among those individuals who have experienced some degree of functional limitation. The use of CAT administration minimizes respondent burden, while allowing for the comprehensive assessment of adults with TBI. The combined use of the TBI-QOL with performance-based measures could guide the development of targeted rehabilitation treatments.

Assessing mental health service user and carer involvement in physical health care planning: The development and validation of a new patient-reported experience measure.

BackgroundPeople living with serious mental health conditions experience increased morbidity due to physical health issues driven by medication side-effects and lifestyle factors. Coordinated mental and physical healthcare delivered in accordance with a care plan could help to reduce morbidity and mortality in this population. Efforts to develop new models of care are hampered by a lack of validated instruments to accurately assess the extent to which mental health services users and carers are involved in care planning for physical health.ObjectiveTo develop a brief and accurate patient-reported experience measure (PREM) capable of assessing involvement in physical health care planning for mental health service users and their carers.MethodsWe employed psychometric and statistical techniques to refine a bank of candidate questionnaire items, derived from qualitative interviews, into a valid and reliable measure involvement in physical health care planning. We assessed the psychometric performance of the item bank using modern psychometric analyses. We assessed unidimensionality, scalability, fit to the partial credit Rasch model, category threshold ordering, local dependency, differential item functioning, and test-retest reliability. Once purified of poorly performing and erroneous items, we simulated computerized adaptive testing (CAT) with 15, 10 and 5 items using the calibrated item bank.ResultsIssues with category threshold ordering, local dependency and differential item functioning were evident for a number of items in the nascent item bank and were resolved by removing problematic items. The final 19 item PREM had excellent fit to the Rasch model fit (x2 = 192.94, df = 1515, P = .02, RMSEA = .03 (95% CI = .01-.04). The 19-item bank had excellent reliability (marginal r = 0.87). The correlation between questionnaire scores at baseline and 2-week follow-up was high (r = .70, P < .01) and 94.9% of assessment pairs were within the Bland Altman limits of agreement. Simulated CAT demonstrated that assessments could be made using as few as 10 items (mean SE = .43).DiscussionWe developed a flexible patient reported outcome measure to quantify service user and carer involvement in physical health care planning. We demonstrate the potential to substantially reduce assessment length whilst maintaining reliability by utilizing CAT.

Adjusting for cross-cultural differences in computer-adaptive tests of quality of life

PurposePrevious studies using the WHOQOL measures have demonstrated that the relationship between individual items and the underlying quality of life (QoL) construct may differ between cultures. If unaccounted for, these differing relationships can lead to measurement bias which, in turn, can undermine the reliability of results.MethodsWe used item response theory (IRT) to assess differential item functioning (DIF) in WHOQOL data from diverse language versions collected in UK, Zimbabwe, Russia, and India (total N = 1332). Data were fitted to the partial credit ‘Rasch’ model. We used four item banks previously derived from the WHOQOL-100 measure, which provided excellent measurement for physical, psychological, social, and environmental quality of life domains (40 items overall). Cross-cultural differential item functioning was assessed using analysis of variance for item residuals and post hoc Tukey tests. Simulated computer-adaptive tests (CATs) were conducted to assess the efficiency and precision of the four items banks.ResultsSplitting item parameters by DIF results in four linked item banks without DIF or other breaches of IRT model assumptions. Simulated CATs were more precise and efficient than longer paper-based alternatives.DiscussionAssessing differential item functioning using item response theory can identify measurement invariance between cultures which, if uncontrolled, may undermine accurate comparisons in computer-adaptive testing assessments of QoL. We demonstrate how compensating for DIF using item anchoring allowed data from all four countries to be compared on a common metric, thus facilitating assessments which were both sensitive to cultural nuance and comparable between countries.

Language‐related differential item functioning between English and German PROMIS Depression items is negligible

To investigate differential item functioning (DIF) of PROMIS Depression items between US and German samples we compared data from the US PROMIS calibration sample (n=780), a German general population survey (n=2,500) and a German clinical sample (n=621). DIF was assessed in an ordinal logistic regression framework, with 0.02 as criterion for R2 -change and 0.096 for Raju's non-compensatory DIF. Item parameters were initially fixed to the PROMIS Depression metric; we used plausible values to account for uncertainty in depression estimates. Only four items showed DIF. Accounting for DIF led to negligible effects for the full item bank as well as a post hoc simulated computer-adaptive test (< 0.1 point on the PROMIS metric [mean=50, standard deviation =10]), while the effect on the short forms was small (< 1 point). The mean depression severity (43.6) in the German general population sample was considerably lower compared to the US reference value of 50. Overall, we found little evidence for language DIF between US and German samples, which could be addressed by either replacing the DIF items by items not showing DIF or by scoring the short form in German samples with the corrected item parameters reported.

Electronic Quality of Life Assessment Using Computer-Adaptive Testing.

BackgroundQuality of life (QoL) questionnaires are desirable for clinical practice but can be time-consuming to administer and interpret, making their widespread adoption difficult.ObjectiveOur aim was to assess the performance of the World Health Organization Quality of Life (WHOQOL)-100 questionnaire as four item banks to facilitate adaptive testing using simulated computer adaptive tests (CATs) for physical, psychological, social, and environmental QoL.MethodsWe used data from the UK WHOQOL-100 questionnaire (N=320) to calibrate item banks using item response theory, which included psychometric assessments of differential item functioning, local dependency, unidimensionality, and reliability. We simulated CATs to assess the number of items administered before prespecified levels of reliability was met.ResultsThe item banks (40 items) all displayed good model fit (P>.01) and were unidimensional (fewer than 5% of t tests significant), reliable (Person Separation Index>.70), and free from differential item functioning (no significant analysis of variance interaction) or local dependency (residual correlations < +.20). When matched for reliability, the item banks were between 45% and 75% shorter than paper-based WHOQOL measures. Across the four domains, a high standard of reliability (alpha>.90) could be gained with a median of 9 items.ConclusionsUsing CAT, simulated assessments were as reliable as paper-based forms of the WHOQOL with a fraction of the number of items. These properties suggest that these item banks are suitable for computerized adaptive assessment. These item banks have the potential for international development using existing alternative language versions of the WHOQOL items.

Development of item bank to measure deliberate self-harm behaviours: Facilitating tailored scales and computer adaptive testing for specific research and clinical purposes

The purpose of this study was to investigate the application of item banking to questionnaire items intended to measure Deliberate Self-Harm (DSH) behaviours. The Rasch measurement model was used to evaluate behavioural items extracted from seven published DSH scales administered to 568 Australians aged 18–30 years (62% university students, 21% mental health patients, and 17% community members). Ninety four items were calibrated in the item bank (including 12 items with differential item functioning for gender and age). Tailored scale construction was demonstrated by extracting scales covering different combinations of DSH methods but with the same raw score for each person location on the latent DSH construct. A simulated computer adaptive test (starting with common self-harm methods to minimise presentation of extreme behaviours) demonstrated that 11 items (on average) were needed to achieve a standard error of measurement of 0.387 (corresponding to a Cronbach׳s Alpha of 0.85). This study lays the groundwork for advancing DSH measurement to an item bank approach with the flexibility to measure a specific definitional orientation (e.g., non-suicidal self-injury) or a broad continuum of self-harmful acts, as appropriate to a particular research/clinical purpose.

Developing an item bank and short forms that assess the impact of asthma on quality of life

The present work describes the process of developing an item bank and short forms that measure the impact of asthma on quality of life (QoL) that avoids confounding QoL with asthma symptomatology and functional impairment. Using a diverse national sample of adults with asthma (N=2032) we conducted exploratory and confirmatory factor analyses, and item response theory and differential item functioning analyses to develop a 65-item unidimensional item bank and separate short form assessments. A psychometric evaluation of the RAND Impact of Asthma on QoL item bank (RAND-IAQL) suggests that though the concept of asthma impact on QoL is multi-faceted, it may be measured as a single underlying construct. The performance of the bank was then evaluated with a real-data simulated computer adaptive test. From the RAND-IAQL item bank we then developed two short forms consisting of 4 and 12 items (reliability=0.86 and 0.93, respectively). A real-data simulated computer adaptive test suggests that as few as 4-5 items from the bank are needed to obtain highly precise scores. Preliminary validity results indicate that the RAND-IAQL measures distinguish between levels of asthma control. To measure the impact of asthma on QoL, users of these items may choose from two highly reliable short forms, computer adaptive test administration, or content-specific subsets of items from the bank tailored to their specific needs.

Adaptive screening for depression — Recalibration of an item bank for the assessment of depression in persons with mental and somatic diseases and evaluation in a simulated computer-adaptive test environment

ObjectiveThis study conducted a simulation study for computer-adaptive testing based on the Aachen Depression Item Bank (ADIB), which was developed for the assessment of depression in persons with somatic diseases. Prior to computer-adaptive test simulation, the ADIB was newly calibrated. MethodsRecalibration was performed in a sample of 161 patients treated for a depressive syndrome, 103 patients from cardiology, and 103 patients from otorhinolaryngology (mean age 44.1, SD=14.0; 44.7% female) and was cross-validated in a sample of 117 patients undergoing rehabilitation for cardiac diseases (mean age 58.4, SD=10.5; 24.8% women). Unidimensionality of the itembank was checked and a Rasch analysis was performed that evaluated local dependency (LD), differential item functioning (DIF), item fit and reliability. CAT-simulation was conducted with the total sample and additional simulated data. ResultsRecalibration resulted in a strictly unidimensional item bank with 36 items, showing good Rasch model fit (item fit residuals<|2.5|) and no DIF or LD. CAT simulation revealed that 13 items on average were necessary to estimate depression in the range of −2 and +2 logits when terminating at SE≤0.32 and 4 items if using SE≤0.50. Receiver Operating Characteristics analysis showed that θ estimates based on the CAT algorithm have good criterion validity with regard to depression diagnoses (Area Under the Curve≥.78 for all cut-off criteria). ConclusionThe recalibration of the ADIB succeeded and the simulation studies conducted suggest that it has good screening performance in the samples investigated and that it may reasonably add to the improvement of depression assessment.

Precision and content range of a parent‐reported item bank assessing lower extremity and mobility skills in children with cerebral palsy

The aim of this study was to determine the psychometric properties, content range, and measurement precision of a lower extremity physical functioning and mobility skills item bank (LE85) in children with cerebral palsy (CP). Lower extremity functioning and mobility skill items were administered to 308 parents of children (169 males, 139 females; mean age 10y 8mo, SD 4y) with spastic CP (145 diplegia, 73 hemiplegia, 89 quadriplegia; [for one person type of CP was unknown]) classified using the Gross Motor Function Classification System (75 level I, 91 level II, 79 level III, 37 level IV, 26 level V). Additional legacy measures were administered to assess concurrent validity. Psychometric characteristics, differential item functioning, content range, and score precision were examined. The LE85 had acceptable psychometric properties. Content range matched the ability range of the sample population and exceeded legacy measures with minimal differential item functioning. The LE85 had good correlation with the Paediatric Outcomes Data Collection Instrument, Functional Independence Measure for Children, Gillette Functional Assessment Questionnaire, and Paediatric Quality of Life Inventory-CP module (range r=0.63-0.86). Precision of the LE85 and 10-item simulated computer adaptive test scores outperformed legacy measures. The LE85 appears to be suitable to administer as a computer adaptive test to measure lower extremity physical functioning and mobility in children with CP.

An item response analysis of the pediatric PROMIS anxiety and depressive symptoms scales

The Patient-Reported Outcomes Measurement Information System (PROMIS) aims to develop self-reported item banks for clinical research. The PROMIS pediatrics (aged 8-17) project focuses on the development of item banks across several health domains (physical function, pain, fatigue, emotional distress, social role relationships, and asthma symptoms). The psychometric properties of the anxiety and depressive symptom item banks are described. Participants (n = 1,529) were recruited in public school settings, hospital-based outpatient and subspecialty pediatrics clinics. The anxiety (k = 18) and depressive symptoms (k = 21) items were split between two test administration forms. Hierarchical confirmatory factor-analytic models (CFA) were conducted to evaluate scale dimensionality and local dependence. IRT analyses were then used to finalize item banks and short forms. CFA results confirmed that anxiety and depressive symptoms are separate constructs and indicative of negative affect. Items with local dependence and DIF were removed resulting in 15 anxiety and 14 depressive symptoms items. The psychometric differences between short forms and simulated computer adaptive tests are presented. PROMIS pediatric item banks were developed to provide efficient assessment of health-related quality of life domains. This sample provides initial calibrations of anxiety and depressive symptoms item banks and creates PROMIS pediatric instruments, version 1.0.

Development of a parent‐report computer‐adaptive test to assess physical functioning in children with cerebral palsy II: upper‐extremity skills

The specific aims of this study were to (1) examine the psychometric properties (unidimensionality, differential item functioning, scale coverage) of an item bank of upper-extremity skills for children and adolescents with cerebral palsy (CP); (2) evaluate a simulated computer-adaptive test (CAT) using this item bank; (3) examine the concurrent validity of the CAT with the Pediatric Outcomes Data Collection Instrument (PODCI) upper-extremity core scale; and (4) determine the discriminant validity of the simulated CAT with Manual Ability Classification System (MACS) levels and CP type (i.e. diplegia, hemiplegia, or quadriplegia). Parents (n=180) of children and adolescents with CP (spastic diplegia 49%, hemiplegia 22%, or quadriplegia 28%) consisting of 102 males and 78 females with a mean age of 10 years 6 months (SD 4y 1mo, range 2-21y), and MACS levels I through V participated in calibration of an item pool and completed the PODCI. Confirmatory factor analyses supported a unidimensional model using 49 of the 53 upper-extremity items. Simulated CATs of 5, 10, and 15 items demonstrated excellent accuracy (intraclass correlation coefficient [ICCs] >0.93) with the full item bank, had high correlations with the PODCI upper-extremity core scale score (ICC 0.79), and discriminated among MACS levels. The simulated CATs demonstrated excellent overall content coverage over a wide age span and severity of upper-extremity involvement. The future development and refinement of CATs for parent report of physical function in children and adolescents with CP is supported by our work.