Item Parameters Research Articles

Testing for differences in chain equating

The comparability of the scores obtained in different forms of a test is certainly an essential requirement. This paper proposes a statistical test for the detection of noncomparable scores based on item response theory (IRT) methods. When the IRT model is fit separately for different forms of a test, the item parameter estimates are expressed on different measurement scales. The first step to obtain comparable scores is to convert the item parameters to a common metric using two constants, called equating coefficients. The equating coefficients can be estimated for two forms with common items, or derived through a chain of forms. The proposal of this paper is a statistical test to verify whether the scale conversions provided by the equating coefficients are as expected when the assumptions of the model are satisfied, hence leading to comparable scores. The method is illustrated through simulation studies and a real‐data example.

Dynamic slate recommendation with gated recurrent units and Thompson sampling

We consider the problem of recommending relevant content to users of an internet platform in the form of lists of items, called slates. We introduce a variational Bayesian Recurrent Neural Net recommender system that acts on time series of interactions between the internet platform and the user, and which scales to real world industrial situations. The recommender system is tested both online on real users, and on an offline dataset collected from a Norwegian web-based marketplace, FINN.no, that is made public for research. This is one of the first publicly available datasets which includes all the slates that are presented to users as well as which items (if any) in the slates were clicked on. Such a data set allows us to move beyond the common assumption that implicitly assumes that users are considering all possible items at each interaction. Instead we build our likelihood using the items that are actually in the slate, and evaluate the strengths and weaknesses of both approaches theoretically and in experiments. We also introduce a hierarchical prior for the item parameters based on group memberships. Both item parameters and user preferences are learned probabilistically. Furthermore, we combine our model with bandit strategies to ensure learning, and introduce ‘in-slate Thompson sampling’ which makes use of the slates to maximise explorative opportunities. We show experimentally that explorative recommender strategies perform on par or above their greedy counterparts. Even without making use of exploration to learn more effectively, click rates increase simply because of improved diversity in the recommended slates.

Testing Differential Item Functioning Without Predefined Anchor Items Using Robust Regression

Differential item functioning (DIF) occurs when the probability of endorsing an item differs across groups for individuals with the same latent trait level. The presence of DIF items may jeopardize the validity of an instrument; therefore, it is crucial to identify DIF items in routine operations of educational assessment. While DIF detection procedures based on item response theory (IRT) have been widely used, a majority of IRT-based DIF tests assume predefined anchor (i.e., DIF-free) items. Not only is this assumption strong, but violations to it may also lead to erroneous inferences, for example, an inflated Type I error rate. We propose a general framework to define the effect sizes of DIF without a priori knowledge of anchor items. In particular, we quantify DIF by item-specific residuals from a regression model fitted to the true item parameters in respective groups. Moreover, the null distribution of the proposed test statistic using robust estimator can be derived analytically or approximated numerically even when there is a mix of DIF and non-DIF items, which yields asymptotically justified statistical inference. The Type I error rate and the power performance of the proposed procedure are evaluated and compared with the conventional likelihood-ratio DIF tests in a Monte Carlo experiment. Our simulation study has shown promising results in controlling Type I error rate and power of detecting DIF items. Even when there is a mix of DIF and non-DIF items, the true and false alarm rate can be well controlled when a robust regression estimator is used.

Psychometric Properties of the COVID-19 Burnout Scale in Brazil

The present study aims to adapt and analyze the psychometric properties (evidence of factor validity and internal consistency) of the COVID-19 Burnout Scale in a sample of Brazilian health professionals. Two studies were carried out (N total = 427). In Study 1 [202 health professionals (age over 18), mainly women (82%) and psychologists (31.2%)], an exploratory factor analysis support the one-factor solution and the suitability of the ten items. Moreover, was also presented in Study 2 [225 health professionals (over 18 years old), the majority of women (79.6%) and psychologists (24.8%)] results of confirmatory factor analysis and item parameters (Item Response Theory) that corroborate the unifactorial structure and convergent validity with the Coronavirus Anxiety Scale. The results indicate that this measure may be suitable for use in research and in the burnout evaluation scenario.

Regularized Variational Estimation for Exploratory Item Factor Analysis.

Item factor analysis (IFA), also known as Multidimensional Item Response Theory (MIRT), is a general framework for specifying the functional relationship between respondents' multiple latent traits and their responses to assessment items. The key element in MIRT is the relationship between the items and the latent traits, so-called item factor loading structure. The correct specification of this loading structure is crucial for accurate calibration of item parameters and recovery of individual latent traits. This paper proposes a regularized Gaussian Variational Expectation Maximization (GVEM) algorithm to efficiently infer item factor loading structure directly from data. The main idea is to impose an adaptive -type penalty to the variational lower bound of the likelihood to shrink certain loadings to 0. This new algorithm takes advantage of the computational efficiency of GVEM algorithm and is suitable for high-dimensional MIRT applications. Simulation studies show that the proposed method accurately recovers the loading structure and is computationally efficient. The new method is also illustrated using the National Education Longitudinal Study of 1988 (NELS:88) mathematics and science assessment data.

Examining and Improving the Gender and Language DIF in the VERA 8 Tests

The purpose of this study was to examine and improve differential item functioning (DIF) across gender and language groups in the VERA 8 tests. We used multigroup concurrent calibration with full and partial invariance based on the Rasch and two-parameter logistic (2PL) models, and classified students into proficiency levels based on their test scores and previously defined cut scores. The results indicated that some items showed gender- and language-specific DIF when using the Rasch model, but we did not detect large misfit items (suspected as DIF) when using the 2PL model. When the item parameters were estimated using the 2PL model with partial invariance assumption (PI-2PL), only small or negligible misfit items were found in the overall tests for both groups. It is argued in this study that the 2PL model should be preferred because both of its approaches provided less bias. However, especially in the presence of unweighted sample sizes of German and non-German students, the non-German students had the highest misfit item proportions. Although the items with medium or small misfit did not have a significant effect on the scores and performance classifications, the items with large misfit changed the proportions of students at the highest and lowest performance levels.

Feature Data Generation for Computer Adaptive Testing

There is no gain-saying that machines can learn from data to derive patterns and insights to aid various applications, also known as artificial intelligence, which is gaining relevance today. This study implemented feature data generation (FDG) as a novel technique for psychometrics improvements using the Post-hoc Simulation approach. The descriptive design of the correlation type was adopted for this study and deployed quantitatively. The instrument for the study was a test aligned to the behavioural objectives of the Postgraduate Certificate Curriculum with the programme enrollees as the study participants. The test underwent a thorough validation procedure which yielded a reliability coefficient of 0.98. The item parameters of the test were analysed using XCalibre 4.2 to analyse the real data from 38 respondents, while the WINGEN application through the post-hoc approach was used to generate the simulated data with 500 respondents. The findings of the study revealed that the 3 Parameter Logistic Model fit the generated data determined using chi-square goodness of fit statistics, and the FDG is a viable approach with a strong and positive correlation between real data and simulated data, which enables the generalisation of findings on the basis on which conclusions were made. The developed FDG method for psychometric improvement has wide applicability, a plus for the novel technique while strengthening transdisciplinary research.

The Effect of Item Form on Estimating Person's Ability, Item Parameters, and Information Function According to Item Response Theory (IRT)

The Effect of Item Form on Estimating Person's Ability, Item Parameters, and Information Function According to Item Response Theory (IRT)

Measurement Equivalence of the Turkish Version of the Satisfaction with Life Scale across Age

The Satisfaction with Life Scale is a widely used measurement tool. In this study, it was aimed to examine the measurement invariance of the Turkish version of the life satisfaction scale across ages. Of the 483 people participating in the study, 198 were men and 285 were women. Participants were divided into two age ranges, 18-24 and 25-43. A first-order single factor solution was provided for both the entire group and all age groups. The findings showed that the configural invariance was achieved by age groups. However, metric invariance could not be provided for age groups. The absence of equivalent factor loadings between the ages of 18-24 and 25-43 means that the latent structure measured by life satisfaction according to age groups does not have the same meaning. Partial metric invariance was obtained when the constraint of the 4th item parameter was freely estimated in further analysis. Subsequent analysis showed that scalar invariance was supported. On the other hand, full strict invariance could not be obtained, but only partially when the parameter constraint of item 1 was released. In summary, the results of this study revealed that comparison of age groups is possible with invariant items. It is hoped that this research will help us to clarify and deepen our inferences about life satisfaction and lifespan.

Asymptotic Standard Errors of IRT Linking Coefficients for the Bifactor Extension of the 3PL Model

The bifactor extension of the three-parameter logistic (B3PL) model has been used in applications of multidimensional item response theory (IRT) such as test equating and vertical scaling. Developing a common multidimensional IRT scale (that is, multidimensional coordinate system) is critical in those applications. Three common-item scale linking methods, the direct least squares (DLS), mean/least squares (MLS), and item response function (IRF) methods, for the B3PL model have been found to be effective in developing a common multidimensional IRT ability scale between two test forms to be linked. In this paper, the asymptotic standard errors (SEs) of IRT linking coefficients estimated by the DLS, MLS, and IRF methods are derived assuming that the B3PL model holds and the asymptotic variance-covariance matrix of item parameter estimates from separate calibrations is available. The delta method is used for the derivations. Computer simulations which investigate the accuracy of the derivations under various conditions are given, showing that the derivations are reasonably accurate when sufficiently large samples are used and that in general the SEs of the IRF method are smaller than those of the DLS and MLS methods. The simulation results also suggest that the SEs of linking coefficient estimates are, approximately, inversely proportional to the square root of the sample size when two test forms are administered to the same number of examinees.

인지진단모형을 활용한 초등학생의 읽기 이해력 분석

Objectives The purpose of this study is to analyze the reading comprehension of elementary school students using the cognitive diagnosis model, centering on the G city regional children's center. Methods The test scores of a total of 184 students who cooperated with the study through the cooperation of 40 regional children's centers in G Metropolitan City were analyzed. As a test tool, some items from the JNE Reading Ability Diagnostic Test developed by the Jeollanam-do Office of Education were used. For the analysis, the Fusion model was used among the cognitive diagnostic models, and a total of 7 cognitive factors were confirmed through the validation of the cognitive factor Q matrix. Results We divided the group into two types, the low-achieving group in reading comprehension and the normal-achieving group in reading comprehension, and compared the difference in scores and mastery of cognitive factors according to factual comprehension, inferential comprehension, and evaluative comprehension. And as a result of performing latent hierarchical analysis using M plus version 5.21, there were clear latent hierarchies based on three levels of understanding, and about 30% of the group had difficulty in reading. Conclusions In a study conducted assuming seven reading comprehension cognitions, it was found that the percentage of mastery of all seven cognitive elements showed the highest percentage through the item parameter estimation by the Fusion model. And the results of comparing the factual comprehension, inferential comprehension, and evaluative comprehension scores of the low-achieving reading comprehension group and the reading comprehension average-achieving group through item difficulty showed significant differences. Implications and discussions are presented through this.

인지진단모형을 활용한 초등학생의 읽기 이해력 분석

Objectives The purpose of this study is to analyze the reading comprehension of elementary school students using the cognitive diagnosis model, centering on the G city regional children's center. Methods The test scores of a total of 184 students who cooperated with the study through the cooperation of 40 regional children's centers in G Metropolitan City were analyzed. As a test tool, some items from the JNE Reading Ability Diagnostic Test developed by the Jeollanam-do Office of Education were used. For the analysis, the Fusion model was used among the cognitive diagnostic models, and a total of 7 cognitive factors were confirmed through the validation of the cognitive factor Q matrix. Results We divided the group into two types, the low-achieving group in reading comprehension and the normal-achieving group in reading comprehension, and compared the difference in scores and mastery of cognitive factors according to factual comprehension, inferential comprehension, and evaluative comprehension. And as a result of performing latent hierarchical analysis using M plus version 5.21, there were clear latent hierarchies based on three levels of understanding, and about 30% of the group had difficulty in reading. Conclusions In a study conducted assuming seven reading comprehension cognitions, it was found that the percentage of mastery of all seven cognitive elements showed the highest percentage through the item parameter estimation by the Fusion model. And the results of comparing the factual comprehension, inferential comprehension, and evaluative comprehension scores of the low-achieving reading comprehension group and the reading comprehension average-achieving group through item difficulty showed significant differences. Implications and discussions are presented through this.

Developing learning progress monitoring tests using difficulty-generating item characteristics: An example for basic arithmetic operations in primary schools

This study investigates difficulty-generating item characteristics (DGICs) in the context of basic arithmetic operations for numbers up to 100 to illustrate their use in item-generating systems for learning progress monitoring (LPM). The fundament of the item-generating system is based on three theory-based DGICs: arithmetic operation, the necessity of crossing 10, and the number of second-term digits. The Rasch model (RM) and the linear logistic test model (LLTM) were used to estimate and predict the DGICs. The results indicate that under the LLTM approach all of the three hypothesized DGICs were significant predictors of item difficulty. Furthermore, the DGICs explain with 20% a solid part of the variance of the RM’s item parameters. The identification and verification of the DGICs under the LLTM approach provide important insights into how to address the challenges in the development of future LPM tests in mathematics.

Characterizing Sampling Variability for Item Response Theory Scale Scores in a Fixed-Parameter Calibrated Projection Design.

A common practice of linking uses estimated item parameters to calculate projected scores. This procedure fails to account for the carry-over sampling variability. Neglecting sampling variability could consequently lead to understated uncertainty for Item Response Theory (IRT) scale scores. To address the issue, we apply a Multiple Imputation (MI) approach to adjust the Posterior Standard Deviations of IRT scale scores. The MI procedure involves drawing multiple sets of plausible values from an approximate sampling distribution of the estimated item parameters. When two scales to be linked were previously calibrated, item parameters can be fixed at their original published scales, and the latent variable means and covariances of the two scales can then be estimated conditional on the fixed item parameters. The conditional estimation procedure is a special case of Restricted Recalibration (RR), in which the asymptotic sampling distribution of estimated parameters follows from the general theory of pseudo Maximum Likelihood (ML) estimation. We evaluate the combination of RR and MI by a simulation study to examine the impact of carry-over sampling variability under various simulation conditions. We also illustrate how to apply the proposed method to real data by revisiting Thissen et al. (2015).

Accuracy of mixture item response theory models for identifying sample heterogeneity in patient-reported outcomes: a simulation study.

Mixture item response theory (MixIRT) models can be used to uncover heterogeneity in responses to items that comprise patient-reported outcome measures (PROMs). This is accomplished by identifying relatively homogenous latent subgroups in heterogeneous populations. Misspecification of the number of latent subgroups may affect model accuracy. This study evaluated the impact of specifying too many latent subgroups on the accuracy of MixIRT models. Monte Carlo methods were used to assess MixIRT accuracy. Simulation conditions included number of items and latent classes, class size ratio, sample size, number of non-invariant items, and magnitude of between-class difference in item parameters. Bias and mean square error in item parameters and accuracy of latent class recovery were assessed. When the number of latent classes was correctly specified, the average bias and MSE in model parameters decreased as the number of items and latent classes increased, but specification of too many latent classes resulted in modest decrease (i.e., < 10%) in the accuracy of latent class recovery. The accuracy of MixIRT model is largely influenced by the overspecification of the number of latent classes. Appropriate choice of goodness-of-fit measures, study design considerations, and a priori contextual understanding of the degree of sample heterogeneity can guide model selection.

Seeking the real item difficulty: bias-corrected item difficulty and some consequences in Rasch and IRT modeling

When the response pattern in a test item deviates from the deterministic pattern, the percentage of correct answers (p) is shown to be a biased estimator for the latent item difficulty (π). This is specifically true with the items of medium item difficulty. Four elements of impurities in p are formalized in the binary settings and four new estimators of π are proposed and studied. Algebraic reasons and a simulation suggest that, except the case of deterministic item discrimination, the real item difficulty is almost always more extreme than what p indicates. This characteristic of p to be biased toward a medium-leveled item difficulty has a strict consequence to item response theory (IRT) and Rasch modeling. Because the classical estimator of item difficulty p is a biased estimator of the latent difficulty level, the item parameters A and B and the person parameter θ within IRT modeling are, consequently, biased estimators of item discrimination and item difficulty as well as ability levels of the test takers.

An analysis of item response theory using program R

The test is one of the instruments used to assess the extent of student understanding in learning. Multiple choice is a type of test commonly used in testing students. In addition to testing students' understanding, the quality of the tests used also needs to be tested. This study aims to determine the characteristics of the national mathematics test items in Baubau in the 2015/2016 academic year and the test information function with the item response theory approach. This research is an ex-post-facto study with a sample size of 574 students using a random sampling technique. Data was collected through documentation and analyzed using the LTM R package program. Findings indicated that there were four items (I1, I2, I4, and I8) for the 1-PL model, six items (I1, I2, I4, I7, I8, and I10) for the 2-PL model, and seven items (I1, I2, I3, I4, I7, I9, and I10) (3-PL) that fit the model (FM). The percentage of good (G) item parameters using R was 90% for (b) (1-PL), 90% (b) and 100% (a) (2-PL), and 90% (b), 10% (a), and 70% (c) (3-PL). The percentage of good quality items in each model for the 1-PL model was 40% or four items, the 2-PL model was 60% or six items, and the 3-PL model was 0%, or none was included in the good quality item category.

Application of Sampling Variance of Item Response Theory Parameter Estimates in Detecting Outliers in Common Item Equating.

In common item equating, the existence of item outliers may impact the accuracy of equating results and bring significant ramifications to the validity of test score interpretations. Therefore, common item equating should involve a screening process to flag outlying items and exclude them from the common item set before equating is conducted. The current simulation study demonstrated that the sampling variance associated with the item response theory (IRT) item parameter estimates can help detect outliers in the common items under the 2-PL and 3-PL IRT models. The results showed the proposed sampling variance statistic (SV) outperformed the traditional displacement method with cutoff values of 0.3 and 0.5 along a variety of evaluation criteria. Based on the favorable results, item outlier detection statistics based on estimated sampling variability warrant further consideration in both research and practice.

Dealing with Missing Responses in Cognitive Diagnostic Modeling

Missing data are a common problem in educational assessment settings. In the implementation of cognitive diagnostic models (CDMs), the presence and/or inappropriate treatment of missingness may yield biased parameter estimates and diagnostic information. Using simulated data, this study evaluates ten approaches for handling missing data in a commonly applied CDM (the deterministic inputs, noisy “and” gate (DINA) model): treating missing data as incorrect (IN), person mean (PM) imputation, item mean (IM) imputation, two-way (TW) imputation, response function (RF) imputation, logistic regression (LR), expectation-maximization (EM) imputation, full information maximum likelihood (FIML) estimation, predictive mean matching (PMM), and random imputation (RI). Specifically, the current study investigates how the estimation accuracy of item parameters and examinees’ attribute profiles from DINA are impacted by the presence of missing data and the selection of missing data methods across conditions. While no single method was found to be superior to other methods across all conditions, the results suggest the use of FIML, PMM, LR, and EM in recovering item parameters. The selected methods, except for PM, performed similarly across conditions regarding attribute classification accuracy. Recommendations for the treatment of missing responses for CDMs are provided. Limitations and future directions are discussed.

Multistage Testing in Heterogeneous Populations: Some Design and Implementation Considerations.

A central challenge in international large-scale assessments is adequately measuring dozens of highly heterogeneous populations, many of which are low performers. To that end, multistage adaptive testing offers one possibility for better assessing across the achievement continuum. This study examines the way that several multistage test design and implementation choices can impact measurement performance in this setting. To attend to gaps in the knowledge base, we extended previous research to include multiple, linked panels, more appropriate estimates of achievement, and multiple populations of varied proficiency. Including achievement distributions from varied populations and associated item parameters, we design and execute a simulation study that mimics an established international assessment. We compare several routing schemes and varied module lengths in terms of item and person parameter recovery. Our findings suggest that, particularly for low performing populations, multistage testing offers precision advantages. Further, findings indicate that equal module lengths-desirable for controlling position effects-and classical routing methods, which lower the technological burden of implementing such a design, produce good results. Finally, probabilistic misrouting offers advantages over merit routing for controlling bias in item and person parameters. Overall, multistage testing shows promise for extending the scope of international assessments. We discuss the importance of our findings for operational work in the international assessment domain.