Item Local Independence Research Articles

An Extension of IRT-Based Equating to the Dichotomous Testlet Response Theory Model

ABSTRACTCurrent procedures for equating number-correct scores using traditional item response theory (IRT) methods assume local independence. However, when tests are constructed using testlets, one concern is the violation of the local item independence assumption. The testlet response theory (TRT) model is one way to accommodate local item dependence. This study proposes methods to extend IRT true score and observed score equating methods to the dichotomous TRT model. We also examine the impact of local item dependence on equating number-correct scores when a traditional IRT model is applied. Results of the study indicate that when local item dependence is at a low level, using the three-parameter logistic model does not substantially affect number-correct equating. However, when local item dependence is at a moderate or high level, using the three-parameter logistic model generates larger equating bias and standard errors of equating compared to the TRT model. However, observed score equating is more robust to the violation of the local item independence assumption than is true score equating.

A Comparison of Different Psychometric Approaches to Modeling Testlet Structures: An Example with C‐Tests

C‐tests are a specific variant of cloze tests that are considered time‐efficient, valid indicators of general language proficiency. They are commonly analyzed with models of item response theory assuming local item independence. In this article we estimated local interdependencies for 12 C‐tests and compared the changes in item difficulties, reliability estimates, and person parameter estimates for different modeling approaches: (a) Rasch, (b) testlet, (c) partial credit, and (d) copula models. The results are complemented with findings of a simulation study in which sample size, number of testlets, and strength of residual correlations between items were systematically manipulated. Results are discussed with regard to the pivotal question whether residual dependencies between items are an artifact or part of the construct.

Effect of Item Response Theory (IRT) Model Selection on Testlet‐Based Test Equating

The local item independence assumption underlying traditional item response theory (IRT) models is often not met for tests composed of testlets. There are 3 major approaches to addressing this issue: (a) ignore the violation and use a dichotomous IRT model (e.g., the 2‐parameter logistic [2PL] model), (b) combine the interdependent items to form a polytomous item and apply a polytomous IRT model (e.g., the graded response model [GRM]), and (c) apply a model that explicitly takes into account the dependence at the item level (e.g., the testlet response theory [TRT] model). In this study, a simulation was conducted to compare the performance of these 3 approaches on number‐correct score equating when degrees of testlet effect were manipulated. The traditional equipercentile method was used as an evaluation baseline. The results show that the 2PL and the TRT approaches produce comparable results that more closely agree with the results of the equipercentile method than the GRM does. And the number‐correct equating using the 2PL is robust to the violation of local item independence.

Polytomous multilevel testlet models for testlet‐based assessments with complex sampling designs

Applications of standard item response theory models assume local independence of items and persons. This paper presents polytomous multilevel testlet models for dual dependence due to item and person clustering in testlet-based assessments with clustered samples. Simulation and survey data were analysed with a multilevel partial credit testlet model. This model was compared with three alternative models - a testlet partial credit model (PCM), multilevel PCM, and PCM - in terms of model parameter estimation. The results indicated that the deviance information criterion was the fit index that always correctly identified the true multilevel testlet model based on the quantified evidence in model selection, while the Akaike and Bayesian information criteria could not identify the true model. In general, the estimation model and the magnitude of item and person clustering impacted the estimation accuracy of ability parameters, while only the estimation model and the magnitude of item clustering affected the item parameter estimation accuracy. Furthermore, ignoring item clustering effects produced higher total errors in item parameter estimates but did not have much impact on the accuracy of ability parameter estimates, while ignoring person clustering effects yielded higher total errors in ability parameter estimates but did not have much effect on the accuracy of item parameter estimates. When both clustering effects were ignored in the PCM, item and ability parameter estimation accuracy was reduced.

Development of a Multimorbidity Illness Perceptions Scale (MULTIPleS)

BackgroundIllness perceptions are beliefs about the cause, nature and management of illness, which enable patients to make sense of their conditions. These perceptions can predict adjustment and quality of life in patients with single conditions. However, multimorbidity (i.e. patients with multiple long-term conditions) is increasingly prevalent and a key challenge for future health care delivery. The objective of this research was to develop a valid and reliable measure of illness perceptions for multimorbid patients.MethodsCandidate items were derived from previous qualitative research with multimorbid patients. Questionnaires were posted to 1500 patients with two or more exemplar long-term conditions (depression, diabetes, osteoarthritis, coronary heart disease and chronic obstructive pulmonary disease). Data were analysed using factor analysis and Rasch analysis. Rasch analysis is a modern psychometric technique for deriving unidimensional and intervally-scaled questionnaires.ResultsQuestionnaires from 490 eligible patients (32.6% response) were returned. Exploratory factor analysis revealed five potential subscales ‘Emotional representations’, ‘Treatment burden’, ‘Prioritising conditions’, ‘Causal links’ and ‘Activity limitations’. Rasch analysis led to further item reduction and the generation of a summary scale comprising of items from all scales. All scales were unidimensional and free from differential item functioning or local independence of items. All scales were reliable, but for each subscale there were a number of patients who scored at the floor of the scale.ConclusionsThe MULTIPleS measure consists of five individual subscales and a 22-item summary scale that measures the perceived impact of multimorbidity. All scales showed good fit to the Rasch model and preliminary evidence of reliability and validity. A number of patients scored at floor of each subscale, which may reflect variation in the perception of multimorbidity. The MULTIPleS measure will facilitate research into the impact of illness perceptions on adjustment, clinical outcomes, quality of life, and costs in patients with multimorbidity.

Rasch analysis of anxiety scales in Parkinson's disease

ObjectiveAnxiety is a common non-motor symptom in Parkinson's disease (PD). This study analyzed the measurement properties of three frequently used anxiety scales in PD: the Beck Anxiety Inventory (BAI), the Hamilton Anxiety Rating Scale (HARS), and the Hospital Anxiety and Depression Scale—Anxiety subscale (HADS-A). MethodThe Rasch model was applied to a multicenter international cohort of 342 patients and assessed the following aspects: fit to the Rasch model, unidimensionality, reliability, response category ordering, item local independence, differential item functioning, and scale targeting. ResultsIn their original form, the BAI, HARS, and HADS-A, did not fit the Rasch model. A good fit to the Rasch model was only found after significant modifications, including rescoring some items and deleting those failing to fit the model. For the BAI and HADS-A, these adjustments resulted in unidimensionality. The HARS was not unidimensional and separate analyses were performed for its psychic and somatic subscales. Whereas the somatic anxiety subscale fit the Rasch model, this was achieved for the psychic anxiety subscale after modifications. ConclusionNone of the currently used anxiety scales display satisfactory measurement properties for assessing anxiety in PD. The results suggest the need to develop a new disease-specific scale for measuring anxiety in PD.

A Multilevel Testlet Model for Dual Local Dependence

The applications of item response theory (IRT) models assume local item independence and that examinees are independent of each other. When a representative sample for psychometric analysis is selected using a cluster sampling method in a testlet‐based assessment, both local item dependence and local person dependence are likely to be induced. This study proposed a four‐level IRT model to simultaneously account for dual local dependence due to item clustering and person clustering. Model parameter estimation was explored using the Markov Chain Monte Carlo method. Model parameter recovery was evaluated in a simulation study in comparison with three other related models: the Rasch model, the Rasch testlet model, and the three‐level Rasch model for person clustering. In general, the proposed model recovered the item difficulty and person ability parameters with the least total error. The bias in both item and person parameter estimation was not affected but the standard error (SE) was affected. In some simulation conditions, the difference in classification accuracy between models could go up to 11%. The illustration using the real data generally supported model performance observed in the simulation study.

The consultation and relational empathy measure: an investigation of its scaling structure

Purpose: The Consultation and Relational Empathy (CARE) measure is recommended to evaluate the quality of care. However, there is no evidence that it is valid in rehabilitation. Aims were to examine the internal construct (factorial) validity of the CARE in the assessment of the patient-therapist relationship. Method: CARE data were part of an experimental study of acupuncture and different currently used acupuncture placebo controls, including 213 patients (age 66.8, SD 8.3, 58% female) with chronic stable hip or knee pain of mechanical origin, waiting for a joint replacement. CARE was completed two weeks into the study and on completion, two weeks later. Data analysis: Cronbach alpha, factor analysis and Rasch analysis. Results: Internal construct validity was supported (82% of variance explained by the first factor; fit to the Rasch model χ2 = 18.2, P = 0.57). CARE was unidimensional, had local independence of items, good item fit, absence of Differential Item Functioning and invariance over time. Three percent of people did not complete items 9 & 10. Conclusions: CARE satisfied strict criteria for internal construct validity. An interval scale transformation is available that can be used in clinical practice and research. Further work is required to investigate item non-response and how this may be dealt with in clinical settings.Implications for RehabilitationThe Consultation and Relational Empathy (CARE) measure CARE satisfies strict criteria for internal construct validity.A transformation table is available, which can be used to convert the raw ordinal data into interval data.The CARE is suitable to evaluate the empathic nature of the consultation process between therapists and patients.

Rasch analysis informed modifications to the Work Instability Scale for Rheumatoid Arthritis for use in work-related upper limb disorders

Objective The Work Instability Scale for Rheumatoid Arthritis (RA-WIS) is a promising prognostic tool for future work disability outcomes. Rasch analysis was conducted to examine the psychometric performance of the RA-WIS in work-related upper limb disorders. Study Design and Setting Eligible injured workers ( n = 396) attending a Shoulder and Elbow Specialty clinic participated in a 1-year study with surveys fielded at four time points. Fit of RA-WIS data (range, 0–23 with 23 = highest work instability) to the Guttman structure was evaluated by item-fit, person-fit, item-trait interaction statistics, and the person separation index (PSI). Differential item functioning (DIF) was evaluated by two-way analyses of variance of the residuals across age, sex, location of injury, perceived exertion at work, and repeated testing over time. Unidimensionality was evaluated by principal component analysis of residuals and tests of local independence. Results RA-WIS data showed significant deviations from the Guttman structure (item-trait interaction χ 2 = 181.6, P < 0.0001, PSI = 0.86). A sequential removal of the six most misfitting items was performed, resulting in a 17-item scale that met all Rasch model expectations ( χ 2 = 57.5, P = 0.007, PSI = 0.83), including unidimensionality, local independence of items, and the absence of DIF across all tested factors. Conclusion A new 17-item Upper Limb Work Instability Scale that satisfied assumptions for interval-level scaling was derived.

Formulation of a DIMTEST Effect Size Measure (DESM) and Evaluation of the DESM Estimator Bias

DIMTEST is a widely used and studied method for testing the hypothesis of test unidimensionality as represented by local item independence. However, DIMTEST does not report the amount of multidimensionality that exists in data when rejecting its null. To provide more information regarding the degree to which data depart from unidimensionality, a DIMTEST‐based Effect Size Measure (DESM) was formulated. In addition to detailing the development of the DESM estimate, the current study describes the theoretical formulation of a DESM parameter. To evaluate the efficacy of the DESM estimator according to test length, sample size, and correlations between dimensions, Monte Carlo simulations were conducted. The results of the simulation study indicated that the DESM estimator converged to its parameter as test length increased, and, as desired, its expected value did not increase with sample size (unlike the DIMTEST statistic in the case of multidimensionality). Also as desired, the standard error of DESM decreased as sample size increased.

The Effect on Latent Trait Estimation in case of the Assumption of Local Item Independence Violated

The Effect on Latent Trait Estimation in case of the Assumption of Local Item Independence Violated

Evaluation of diagnostic criteria for night eating syndrome using item response theory analysis

Uniform diagnostic criteria for the night eating syndrome (NES), a disorder characterized by a delay in the circadian pattern of eating, have not been established. Proposed criteria for NES were evaluated using item response theory (IRT) analysis. Six studies yielded 1,481 Night Eating Questionnaires which were coded to reflect the presence/absence of five night eating symptoms. Symptoms were evaluated based on the clinical usefulness of their diagnostic information and on the assumptions of IRT analysis (unidimensionality, monotonicity, local item independence, correct model specification), using a two parameter logistic (2PL) IRT model. Reports of (1) nocturnal eating and/or evening hyperphagia, (2) initial insomnia, and (3) night awakenings showed high precision in discriminating those with night eating problems, while morning anorexia and delayed morning meal provided little additional information. IRT is a useful tool for evaluating the diagnostic criteria of psychiatric disorders and can be used to evaluate potential diagnostic criteria of NES empirically. Behavioral factors were identified as useful discriminators of NES. Future work should also examine psychological factors in conjunction with those identified here.

The Presence and Impact of Local Item Dependence on Objective Structured Clinical Examinations Scores and the Potential Use of the Polytomous, Many-Facet Rasch Model

Objective The purpose of this research project was to extend the research on the robustness of the dichotomous Rasch model to violations of the local independence assumption to the polytomous many-facet Rasch model (MFRM). Candidate scores from oral examinations and objective structured clinical examinations (OSCEs) have been shown to contain variance due to rater error/bias. If the MFRM is robust to local item dependence (LID), then the MFRM could theoretically be applied to medical OSCEs. Methods Five OSCEs were used in the study: 3 chiropractic licensure OSCEs and 2 nursing licensure OSCEs. Items were assigned to split-halves based on common stimulus. Split-half correlations were compared with Spearman-Brown estimates of reliability based on Cronbach α with all items contributing. Two- and 3-facet MFRM analyses were performed, first with individual items contributing and second with station totals contributing. Correlations were estimated between the 2 MFRM estimates. Results Cronbach α estimates with all items contributing were all very high (>.87). Spearman-Brown estimates were all considerably higher than split-half correlations. Correlations between MFRM by items and by stations were all very high (>.993). Conclusions The research project provided evidence that OSCEs violate the local item independence assumption. The project also showed that the MFRM is quite robust to such violations. The authors recommend that the MFRM be applied to OSCEs by station totals for estimates of candidate ability, and by items for item performance measures and quality control programs.

Simulated computerized adaptive tests for measuring functional status were efficient with good discriminant validity in patients with hip, knee, or foot/ankle impairments

Background and Objective To develop computerized adaptive tests (CATs) designed to assess lower extremity functional status (FS) in people with lower extremity impairments using items from the Lower Extremity Functional Scale and compare discriminant validity of FS measures generated using all items analyzed with a rating scale Item Response Theory model (θ IRT) and measures generated using the simulated CATs (θ CAT). Methods Secondary analysis of retrospective intake rehabilitation data. Results Unidimensionality of items was strong, and local independence of items was adequate. Differential item functioning (DIF) affected item calibration related to body part, that is, hip, knee, or foot/ankle, but DIF did not affect item calibration for symptom acuity, gender, age, or surgical history. Therefore, patients were separated into three body part specific groups. The rating scale model fit all three data sets well. Three body part specific CATs were developed: each was 70% more efficient than using all LEFS items to estimate FS measures. θ IRT and θ CAT measures discriminated patients by symptom acuity, age, and surgical history in similar ways. θ CAT measures were as precise as θ IRT measures. Conclusion Body part-specific simulated CATs were efficient and produced precise measures of FS with good discriminant validity.

Local Item Dependence for Items Across Tests Connected by Common Stimuli

A parallel design, in which items across different scales within an instrument share common stimuli and subjects respond to the common stimulus for each scale, is sometimes used in questionnaires or inventories. Because the items across scales share the same stimuli, the assumption of local item independence may not hold, thereby violating the assumption of local item independence under standard psychometric models. In this study, the authors describe a multidimensional item response model to detect specific forms of local item dependence. Three real data sets were analyzed to illustrate implications and applications of the proposed method.

Scaling Performance Assessments: Strategies for Managing Local Item Dependence

Performance assessments appear on a priori grounds to be likely to produce far more local item dependence (LID) than that produced in the use of traditional multiple‐choice tests. This article (a) defines local item independence, (b) presents a compendium of causes of LID, (c) discusses some of LID's practical measurement implications, (d) details some empirical results for both performance assessments and multiple‐choice tests, and (e) suggests some strategies for managing LID in order to avoid negative measurement consequences.