Item Response Theory Item Parameter Research Articles

IRT 능력 분포 추정을 위한 정규적 방법과 비모수적 방법의 기능 비교

Many applications of item response theory (IRT) require that the population distribution for a latent ability variable should be specified or estimated. Given IRT item parameters and item response vectors of a sample of examinees from a population, the ability distribution may be estimated using the normal or non-parametric method. The purpose of this study was to use computer simulations to examine the characteristics and relative performance of the Gauss-Hermite quadrature and fixed-points quadrature based normal methods and the non-parametric method for estimating ability distributions. Test type, distribution shape, sample size, and number of ability points were considered as simulation factors. Main results were as follows. First, the two normal methods estimated properly the mean and standard deviation (SD) of the population distribution when the distribution was normal but they estimated the distribution parameters with some bias when the distribution was skewed. In contrast, the non-parametric method estimated the mean and SD without bias regardless of the distribution shape. Second, the standard errors of the mean and SD estimates decreased approximately inversely proportional to the square root of the sample size as the sample size increased. Third, overall, the fixed-points quadrature method estimated the distribution parameters with less bias and more stably than the Gauss-Hermite quadrature method. Fourth, when the number of ability points was 20 or 30, the distribution estimated by the non-parametric method tended to have a choppy shape.

5 PediaTracTM: A Novel Measure for Screening Early Motor Development in Infants Born Preterm and Other At-Risk Populations

AbstractObjective:Infant motor development is a robust predictor of long-term developmental outcomes, especially in infants at high risk for neurodevelopmental impairments, such as those born preterm (PT, gestational age [GA] <37 weeks). Although direct assessments of motor development are available, they are infrequently applied by pediatricians in routine screening of the broader population of infants born preterm. Parent ratings, such as the Ages and Stages Questionnaire, 3rd Ed., can be used to screen for motor delays. However, this and other existing screening measures focus on whether children have reached milestones based on pre-established cutoffs, rather than on assessing development along a continuum of ability. The present study examined the validity of the Motor domain of the recently developed caregiver report screening tool, PediaTracTM, in distinguishing infants born PT from infants born full term (FT, GA ^37 weeks) across the first 6 months of life. The reliability and factor structure of this motor scale were also evaluated.Participants and Methods:PediaTracTM is a web-based caregiver report assessing infant development across multiple domains, including motor functioning. This study reports on results from the PediaTracTM Motor domain for the study sample of 571 caregiver-infant dyads (240 PT, 331 FT). Caregivers rated their infants on age-targeted motor skills during the newborn period (NB, defined as term equivalent for the PT group) and at 2-, 4-, and 6-months after the NB period. Item Response Theory (IRT) methods were applied to assess the reliability (i.e., information) of caregiver-reported motor skills at each age. Using the IRT item parameters of discrimination and difficulty, items were selected for inclusion and to estimate theta, an index of the latent trait, motor ability, for each infant. Analyses conducted at each age assessed the effects of group, sex, and group x sex on the motor trait. Scale reliabilities and factor structure were also examined.Results:The PT group had significantly higher scores than the FT group on the motor trait at the NB period but significantly lower scores by 4 and 6 months, suggesting slower development of motor skills in the PT group. Means (SD) theta scores (similar to z scores) for the PT and FT groups, respectively, were .14 (.88) and -.05 (.91) for the NB period, -.01 (.90) and .01 (.91) for 2 months, .20 (.90) and .36 (88) for 4 months, and .46 (78) and .66 (.89) for 6 months. Effects for sex and group x sex interactions were not significant. Reliabilities, estimated at a point close to mean theta, were .94, .93, .96, and .98 at the NB, 2-, 4-, and 6-month periods, respectively. Exploratory factor analyses revealed evidence for a single primary motor factor and multiple second-order factors at each age.Conclusions:Findings provide strong support for applications of the caregiver reported PediaTracTM motor scales in screening infants born PT and other at-risk populations for early delays or abnormalities in motor development. Advantages of this method include its ease of administration, sensitivity to developmental change, and promise in assessing subdomains of motor skill.

Investigating the insomnia severity spectrum using the Pittsburgh Insomnia Rating Scale (PIRS)

To understand how a population natively conceptualizes insomnia along with related mental health and related disabilities, understanding how an insomnia severity spectrum operates in any given population will be a goal for sleep epidemiology. To facilitate this international goal, the comparatively theory-tolerant 65-item Pittsburgh Insomnia Rating Scale (PIRS65) had been constructed with a structure designed to mimic the classification structure of the WHO's International Classification of Functioning, Disability, and Health(ICF), while measuring insomnia symptoms comprehensively. The PIRS65 underwent validation tests using both Classical Test Theory (CTT) and Item Response Theory (IRT) analyses in a research clinic-based case-control study in a North American sample. These analyses indicated that the insomnia was mostly unidimensional, but with correlations to sleep-item-free measures of fatigue, depression, and anxiety. The PIRS65’s sleep clock-time items, posed as metrics of psychological disability, performed poorly as such, despite good external validation against prospective sleep logs. PIRS65 item stems with symptom attribution to a time pattern of “not sleeping well” had the highest IRT discrimination parameters, pointing to how these respondents understood their disability from insomnia. Accordingly, a 2-item PIRS2 is proposed as particularly applicable to epidemiological studies for which an insomnia metric needs to be concise. IRT item parameters for all PIRS65 items are also provided as benchmark data for possible later international comparisons of insomnia severity spectra.

Semisupervised Learning Method to Adjust Biased Item Difficulty Estimates Caused by Nonignorable Missingness in a Virtual Learning Environment.

In data collected from virtual learning environments (VLEs), item response theory (IRT) models can be used to guide the ongoing measurement of student ability. However, such applications of IRT rely on unbiased item parameter estimates associated with test items in the VLE. Without formal piloting of the items, one can expect a large amount of nonignorable missing data in the VLE log file data, and this is expected to negatively affect IRT item parameter estimation accuracy, which then negatively affects any future ability estimates utilized in the VLE. In the psychometric literature, methods for handling missing data have been studied mostly around conditions in which the data and the amount of missing data are not as large as those that come from VLEs. In this article, we introduce a semisupervised learning method to deal with a large proportion of missingness contained in VLE data from which one needs to obtain unbiased item parameter estimates. First, we explored the factors relating to the missing data. Then we implemented a semisupervised learning method under the two-parameter logistic IRT model to estimate the latent abilities of students. Last, we applied two adjustment methods designed to reduce bias in item parameter estimates. The proposed framework showed its potential for obtaining unbiased item parameter estimates that can then be fixed in the VLE in order to obtain ongoing ability estimates for operational purposes.

Evaluating Item Fit Statistic Thresholds in PISA: Analysis of Cross‐Country Comparability of Cognitive Items

AbstractIn Programme for International Student Assessment (PISA), item response theory (IRT) scaling is used to examine the psychometric properties of items and scales and to provide comparable test scores across participating countries and over time. To balance the comparability of IRT item parameter estimations across countries with the best possible model fit, a partial invariance approach is used in PISA. In this approach, international or common item parameters are estimated for the majority of items, while unique or country‐specific item parameters are allowed for item‐country combinations where a misfit to the common parameters can be identified. The goal of the current study is to establish item fit statistic thresholds for identifying such misfits. We investigated the impact of various thresholds on scale and score estimation. To evaluate the impact of various item fit thresholds, we systematically examined the number of unique item parameters and country performance distributions and compared the overall model fit statistics using data from PISA 2015 and 2018. Results showed that RMSD = .10 provides the best fitting model while still establishing stable parameter estimations and sufficient comparability across groups. The applications and implications of the results are discussed.

Standard Errors of IRT Parameter Scale Transformation Coefficients: Comparison of Bootstrap Method, Delta Method, and Multiple Imputation Method

AbstractThe present study evaluated the multiple imputation method, a procedure that is similar to the one suggested by Li and Lissitz (2004), and compared the performance of this method with that of the bootstrap method and the delta method in obtaining the standard errors for the estimates of the parameter scale transformation coefficients in item response theory (IRT) equating in the context of the common‐item nonequivalent groups design. Two different estimation procedures for the variance‐covariance matrix of the IRT item parameter estimates, which were used in both the delta method and the multiple imputation method, were considered: empirical cross‐product (XPD) and supplemented expectation maximization (SEM). The results of the analyses with simulated and real data indicate that the multiple imputation method generally produced very similar results to the bootstrap method and the delta method in most of the conditions. The differences between the estimated standard errors obtained by the methods using the XPD matrices and the SEM matrices were very small when the sample size was reasonably large. When the sample size was small, the methods using the XPD matrices appeared to yield slight upward bias for the standard errors of the IRT parameter scale transformation coefficients.

IRT Item Parameter Recovery With Marginal Maximum Likelihood Estimation Using Loglinear Smoothing Models

Loglinear smoothing (LLS) estimates the latent trait distribution while making fewer assumptions about its form and maintaining parsimony, thus leading to more precise item response theory (IRT) item parameter estimates than standard marginal maximum likelihood (MML). This article provides the expectation-maximization algorithm for MML estimation with LLS embedded and compares LLS to other latent trait distribution specifications, a fixed normal distribution, and the empirical histogram solution, in terms of IRT item parameter recovery. Simulation study results using a 3-parameter logistic model reveal that LLS models matching four or five moments are optimal in most cases. Examples with empirical data compare LLS to these approaches as well as Ramsay-curve IRT.

How Does Calibration Timing and Seasonality Affect Item Parameter Estimates?

Continuously administered examination programs, particularly credentialing programs that require graduation from educational programs, often experience seasonality where distributions of examine ability may differ over time. Such seasonality may affect the quality of important statistical processes, such as item response theory (IRT) item calibration and equating. The lead time required for producing pre-equated test forms in the continuous testing framework further complicates issues. This study examines the effect of seasonality in test data on Rasch IRT item parameter estimates. Data came from four credentialing examination programs that represented both programs with and without seasonality, as well as medium and low examinee volume. Results showed that calibrating items during certain times can lead to quite poor item parameter estimates. While certain programs could conduct IRT calibrations without waiting for the full examination cycle to be completed, other types of programs should wait as long as possible before calibrating items.

Testing item response theory invariance of the standardized Quality-of-life Disease Impact Scale (QDIS(®)) in acute coronary syndrome patients: differential functioning of items and test.

The Quality-of-life (QOL) Disease Impact Scale (QDIS(®)) standardizes the content and scoring of QOL impact attributed to different diseases using item response theory (IRT). This study examined the IRT invariance of the QDIS-standardized IRT parameters in an independent sample. The differential functioning of items and test (DFIT) of a static short-form (QDIS-7) was examined across two independent sources: patients hospitalized for acute coronary syndrome (ACS) in the TRACE-CORE study (N = 1,544) and chronically ill US adults in the QDIS standardization sample. "ACS-specific" IRT item parameters were calibrated and linearly transformed to compare to "standardized" IRT item parameters. Differences in IRT model-expected item, scale and theta scores were examined. The DFIT results were also compared in a standard logistic regression differential item functioning analysis. Item parameters estimated in the ACS sample showed lower discrimination parameters than the standardized discrimination parameters, but only small differences were found for thresholds parameters. In DFIT, results on the non-compensatory differential item functioning index (range 0.005-0.074) were all below the threshold of 0.096. Item differences were further canceled out at the scale level. IRT-based theta scores for ACS patients using standardized and ACS-specific item parameters were highly correlated (r = 0.995, root-mean-square difference = 0.09). Using standardized item parameters, ACS patients scored one-half standard deviation higher (indicating greater QOL impact) compared to chronically ill adults in the standardization sample. The study showed sufficient IRT invariance to warrant the use of standardized IRT scoring of QDIS-7 for studies comparing the QOL impact attributed to acute coronary disease and other chronic conditions.

The Sequential Probability Ratio Test and Binary Item Response Models

The sequential probability ratio test (SPRT) is a common method for terminating item response theory (IRT)-based adaptive classification tests. To decide whether a classification test should stop, the SPRT compares a simple log-likelihood ratio, based on the classification bound separating two categories, to prespecified critical values. As has been previously noted (Spray & Reckase, 1994), the SPRT test statistic is not necessarily monotonic with respect to the classification bound when item response functions have nonzero lower asymptotes. Because of nonmonotonicity, several researchers (including Spray & Reckase, 1994) have recommended selecting items at the classification bound rather than the current ability estimate when terminating SPRT-based classification tests. Unfortunately, this well-worn advice is a bit too simplistic. Items yielding optimal evidence for classification depend on the IRT model, item parameters, and location of an examinee with respect to the classification bound. The current study illustrates, in depth, the relationship between the SPRT test statistic and classification evidence in binary IRT models. Unlike earlier studies, we examine the form of the SPRT-based log-likelihood ratio while altering the classification bound and item difficulty. These investigations motivate a novel item selection algorithm based on optimizing the expected SPRT criterion given the current ability estimate. The new expected log-likelihood ratio algorithm results in test lengths noticeably shorter than current, commonly used algorithms, and with no loss in classification accuracy.

The Communicative Participation Item Bank (CPIB): item bank calibration and development of a disorder-generic short form.

The purpose of this study was to calibrate the items for the Communicative Participation Item Bank (CPIB; Baylor, Yorkston, Eadie, Miller, & Amtmann, 2009; Yorkston et al., 2008) using item response theory (IRT). One overriding objective was to examine whether the IRT item parameters would be consistent across different diagnostic groups, thereby allowing creation of a disorder-generic instrument. The intended outcomes were the final item bank and a short form ready for clinical and research applications. Self-report data were collected from 701 individuals representing 4 diagnoses: multiple sclerosis, Parkinson's disease, amyotrophic lateral sclerosis, and head and neck cancer. Participants completed the CPIB and additional self-report questionnaires. CPIB data were analyzed using the IRT graded response model. The initial set of 94 candidate CPIB items were reduced to an item bank of 46 items demonstrating unidimensionality, local independence, good item fit, and good measurement precision. Differential item functioning analyses detected no meaningful differences across diagnostic groups. A 10-item, disorder-generic short form was generated. The CPIB provides speech-language pathologists with a unidimensional, self-report outcomes measurement instrument dedicated to the construct of communicative participation. This instrument may be useful to clinicians and researchers wanting to implement measures of communicative participation in their work.

Unidimensional Interpretations for Multidimensional Test Items

This article considers potential problems that can arise in estimating a unidimensional item response theory (IRT) model when some test items are multidimensional (i.e., show a complex factorial structure). More specifically, this study examines (1) the consequences of model misfit on IRT item parameter estimates due to unintended minor item‐level multidimensionality, and (2) whether a Projection IRT model can provide a useful remedy. A real‐data example is used to illustrate the problem and also is used as a base model for a simulation study. The results suggest that ignoring item‐level multidimensionality might lead to inflated item discrimination parameter estimates when the proportion of multidimensional test items to unidimensional test items is as low as 1:5. The Projection IRT model appears to be a useful tool for updating unidimensional item parameter estimates of multidimensional test items for a purified unidimensional interpretation.

Bifactor Models and Rotations: Exploring the Extent to Which Multidimensional Data Yield Univocal Scale Scores

The application of psychological measures often results in item response data that arguably are consistent with both unidimensional (a single common factor) and multidimensional latent structures (typically caused by parcels of items that tap similar content domains). As such, structural ambiguity leads to seemingly endless “confirmatory” factor analytic studies in which the research question is whether scale scores can be interpreted as reflecting variation on a single trait. An alternative to the more commonly observed unidimensional, correlated traits, or second-order representations of a measure's latent structure is a bifactor model. Bifactor structures, however, are not well understood in the personality assessment community and thus rarely are applied. To address this, herein we (a) describe issues that arise in conceptualizing and modeling multidimensionality, (b) describe exploratory (including Schmid–Leiman [Schmid & Leiman, 1957] and target bifactor rotations) and confirmatory bifactor modeling, (c) differentiate between bifactor and second-order models, and (d) suggest contexts where bifactor analysis is particularly valuable (e.g., for evaluating the plausibility of subscales, determining the extent to which scores reflect a single variable even when the data are multidimensional, and evaluating the feasibility of applying a unidimensional item response theory (IRT) measurement model). We emphasize that the determination of dimensionality is a related but distinct question from either determining the extent to which scores reflect a single individual difference variable or determining the effect of multidimensionality on IRT item parameter estimates. Indeed, we suggest that in many contexts, multidimensional data can yield interpretable scale scores and be appropriately fitted to unidimensional IRT models.

A review of the effects on IRT item parameter estimates with a focus on misbehaving common items in test equating

Many studies have investigated the topic of change or drift in item parameter estimates in the context of item response theory (IRT). Content effects, such as instructional variation and curricular emphasis, as well as context effects, such as the wording, position, or exposure of an item have been found to impact item parameter estimates. The issue becomes more critical when items with estimates exhibiting differential behavior across test administrations are used as common for deriving equating transformations. This paper reviews the types of effects on IRT item parameter estimates and focuses on the impact of misbehaving or aberrant common items on equating transformations. Implications relating to test validity and the judgmental nature of the decision to keep or discard aberrant common items are discussed, with recommendations for future research into more informed and formal ways of dealing with misbehaving common items.

A Systematic Comparison Between Classical Optimal Scaling and the Two-Parameter IRT Model

In this article, the relationship between two alternative methods for the analysis of multivariate categorical data is systematically explored. It is shown that the person score of the first dimension of classical optimal scaling correlates strongly with the latent variable for the two-parameter item response theory (IRT) model. Next, under the assumption that the latent variable has a standard normal distribution, it is derived how the IRT item parameters are related to the optimal category weights.

Comparing Latent Structures of the Grade of Membership, Rasch, and Latent Class Models

This paper focuses on model interpretation issues and employs a geometric approach to compare the potential value of using the Grade of Membership (GoM) model in representing population heterogeneity. We consider population heterogeneity manifolds generated by letting subject specific parameters vary over their natural range, while keeping other population parameters fixed, in the marginal space (based on marginal probabilities) and in the full parameter space (based on cell probabilities). The case of a 2 × 2 contingency table is discussed in detail, and a generalization to 2J tables with J ≥ 3 is sketched. Our approach highlights the main distinction between the GoM model and the probabilistic mixture of classes by demonstrating geometrically the difference between the concepts of partial and probabilistic memberships. By using the geometric approach we show that, in special cases, the GoM model can be thought of as being similar to an item response theory (IRT) model in representing population heterogeneity. Finally, we show that the GoM item parameters can provide quantities analogous to more general logistic IRT item parameters. As a latent structure model, the GoM model might be considered a useful alternative for a data analysis when both classes of extreme responses, and additional heterogeneity that cannot be captured by those latent classes, are expected in the population.

IRT Item Parameters and the Reliability and Validity of Pretest, Posttest, and Gain Scores

The effect of different combinations of item response theory (IRT) item parameters (item difficulty, item discrimination, and the guessing probability) on the reliability and construct validity (correlation with the latent trait being measured) of pretest, posttest, and gain scores was analytically examined using the 3-parameter logistic (3PL) model (Birnbaum, 1968). The analyses were performed at the single-item level using items with specified combinations of item parameters. In general, the pretest item validities and reliabilities were much higher than those of the gain score. It is also the case that the reliability of the gain score is at a minimum when that of the pretest score is at a maximum, namely, near the middle of the distribution of item difficulty (i.e., for moderately difficult tests). Furthermore, using IRT we show that the regression of the gain score on the latent trait being measured is nonmonotonic, meaning a person at a given low ability level has the same expected gain score as one of the corresponding high ability. These results not only are of theoretical interest but also may be of interest to applied practitioners and test developers.

Chapter 4: Item Response Theory Scale Linking in NAEP

In educational assessments, it is often necessary to compare the performance of groups of individuals who have been administered different forms of a test. If these groups are to be validly compared, all results need to be expressed on a common scale. When assessment results are to be reported using an item response theory (IRT) proficiency metric, as is done for the National Assessment of Educational Progress (NAEP), establishing a common metric becomes synonymous with expressing IRT item parameter estimates on a common scale. Procedures that accomplish this are referred to here as scale linking procedures. This chapter discusses the need for scale linking in NAEP and illustrates the specific procedures used to carry out the linking in the context of the major analyses conducted for the 1990 NAEP mathematics assessment.

Item Response Theory Scale Linking in NAEP

In educational assessments, it is often necessary to compare the performance of groups of individuals who have been administered different forms of a test. If these groups are to be validly compared, all results need to be expressed on a common scale. When assessment results are to be reported using an item response theory (IRT) proficiency metric, as is done for the National Assessment of Educational Progress (NAEP), establishing a common metric becomes synonymous with expressing IRT item parameter estimates on a common scale. Procedures that accomplish this are referred to here as scale linking procedures. This chapter discusses the need for scale linking in NAEP and illustrates the specific procedures used to carry out the linking in the context of the major analyses conducted for the 1990 NAEP mathematics assessment.

A Comparative Study of the Effects of Recency of Instruction on the Stability of IRT and Conventional Item Parameter Estimates

A potential concern for individuals interested in using item response theory (IRT) with achievement test data is that such tests have been specifically designed to measure content areas related to course curriculum and students taking the tests at different points in their coursework may not constitute samples from the same population. In this study, data were obtained from three administrations of two forms of a Biology achievement test. Data from the newer of the two forms were collected at a spring administration, made up of high school sophomores just completing the Biology course, and at a fall administration, made up mostly of seniors who completed their instruction in the course from 6–18 months prior to the test administration. Data from the older form, already on scale, were collected at only a fall administration, where the sample was comparable to the newer form fall sample. IRT and conventional item difficulty parameter estimates for the common items across the two forms were compared for each of the two form/sample combinations. In addition, conventional and IRT score equatings were performed between the new and old forms for each o f the form sample combinations. Widely disparate results were obtained between the equatings based on the two form/sample combinations. Conclusions are drawn about the use o f both classical test theory and IRT in situations such as that studied, and implications o f the results for achievement test validity are also discussed