Item Parameters Research Articles

Analysis of differential item functioning in agricultural science examination across southwestern nigeria's senior schools

Differential Item Functioning poses a threat to test fairness and validity in educational assessments. To this end, this study probes the incidence of DIF in the Senior School Certificate Agricultural Science Examination (SSCASE) across school locations in Southwestern Nigeria. It determined the magnitude of DIF using item parameters in SSCASE. The study applies one, two and three-parameter logistic models to analyze the dichotomous data responses of 818 randomly selected students from urban and rural locations. The ex-post facto design adopted the 2015 National Examination Council SSCE in Agricultural Science as its instrument (α =0.887) in this study. The study population comprised SSCE candidates from Osun, Ondo, and Oyo. The results showed the magnitude of DIF in SSCASE was large and moderate (1PLM=0.52 significant DIF, 2PLM=0.45 moderate DIF, 3PLM=0.47 moderate DIF) across school locations. The study concluded that 1, 2 and 3 PLMs can produce fair items in SSCASE.

A many-facet Rasch measurement model approach to investigating objective structured clinical examination item parameter drift.

Objective Structured Clinical Examinations (OSCEs) are widely used for assessing clinical competence, especially in high-stakes environments such as medical licensure. However, the reuse of OSCE cases across multiple administrations raises concerns about parameter stability, known as item parameter drift (IPD). AIMS &OBJECTIVES: This study aims to investigate IPD in reused OSCE cases while accounting for examiner scoring effects using a Many-facet Rasch Measurement (MFRM) model. Data from 12 OSCE cases, reused over seven administrations of the Internationally Educated Nurse Competency Assessment Program (IENCAP), were analyzed using the MFRM model. Each case was treated as an item, and examiner scoring effects were accounted for in the analysis. The results indicated that despite accounting for examiner effects, all cases exhibited some level of IPD, with an average absolute IPD of 0.21 logits. Three cases showed positive directional trends. IPD significantly affected score decisions in 1.19% of estimates, at an invariance violation of 0.58 logits. These findings suggest that while OSCE cases demonstrate sufficient stability for reuse, continuous monitoring is essential to ensure the accuracy of score interpretations and decisions. The study provides an objective threshold for detecting concerning levels of IPD and underscores the importance of addressing examiner scoring effects in OSCE assessments. The MFRM model offers a robust framework for tracking and mitigating IPD, contributing to the validity and reliability of OSCEs in evaluating clinical competence.

A Note on Standard Errors for Multidimensional Two-Parameter Logistic Models Using Gaussian Variational Estimation.

Accurate item parameters and standard errors (SEs) are crucial for many multidimensional item response theory (MIRT) applications. A recent study proposed the Gaussian Variational Expectation Maximization (GVEM) algorithm to improve computational efficiency and estimation accuracy (Cho et al., 2021). However, the SE estimation procedure has yet to be fully addressed. To tackle this issue, the present study proposed an updated supplemented expectation maximization (USEM) method and a bootstrap method for SE estimation. These two methods were compared in terms of SE recovery accuracy. The simulation results demonstrated that the GVEM algorithm with bootstrap and item priors (GVEM-BSP) outperformed the other methods, exhibiting less bias and relative bias for SE estimates under most conditions. Although the GVEM with USEM (GVEM-USEM) was the most computationally efficient method, it yielded an upward bias for SE estimates.

Delving into the Complexity of Analogical Reasoning: A Detailed Exploration with the Generalized Multicomponent Latent Trait Model for Diagnosis.

Research on analogical reasoning has facilitated the understanding of response processes such as pattern identification and creative problem solving, emerging as an intelligence predictor. While analogical tests traditionally combine various composition rules for item generation, current statistical models like the Logistic Latent Trait Model (LLTM) and Embretson's Multicomponent Latent Trait Model for Diagnosis (MLTM-D) face limitations in handling the inherent complexity of these processes, resulting in suboptimal model fit and interpretation. The primary aim of this research was to extend Embretson's MLTM-D to encompass complex multidimensional models that allow the estimation of item parameters. Concretely, we developed a three-parameter (3PL) version of the MLTM-D that provides more informative interpretations of participant response processes. We developed the Generalized Multicomponent Latent Trait Model for Diagnosis (GMLTM-D), which is a statistical model that extends Embretson's multicomponent model to explore complex analogical theories. The GMLTM-D was compared with LLTM and MLTM-D using data from a previous study of a figural analogical reasoning test composed of 27 items based on five composition rules: figure rotation, trapezoidal rotation, reflection, segment subtraction, and point movement. Additionally, we provide an R package (GMLTM) for conducting Bayesian estimation of the models mentioned. The GMLTM-D more accurately replicated the observed data compared to the Bayesian versions of LLTM and MLTM-D, demonstrating a better model fit and superior predictive accuracy. Therefore, the GMLTM-D is a reliable model for analyzing analogical reasoning data and calibrating intelligence tests. The GMLTM-D embraces the complexity of real data and enhances the understanding of examinees' response processes.

Experience of Service Questionnaire (ESQ) in children and adolescents: factor structure, reliability, validity, item parameters and interpretability of the parent version for practical use in Greece.

Health systems need tools to assess patient's experience of service, but existing tools lack reliability and validity assessment. Our aim is to investigate the factor structure, reliability, validity, item parameters and interpretability of the parent version of the Experience of Service Questionnaire (ESQ) for practical use in Greece. A total of 265 caregivers that were using mental health services in Greece participated in this study as part of the Nationwide cross-sectional survey from the Child and Adolescent Mental Health Initiative (CAMHI). Confirmatory Factor Analysis was used to test factor structure. Reliability of all models were measured with omega coefficients. Tobit regression analysis was used to test for convergent and discriminant validity with specifically designed questions. Item parameters were assessed via Item Response Theory. Interpretability was assessed by means of IRT-based scores. We found that ESQ is best represented and scored as a unidimensional construct, given potential subscales would not have enough reliability apart from a general factor. Convergent and discriminant validity was demonstrated, as caregivers who perceived that their child benefited from the received mental health care had 6.50 higher summed scores (SMD=1.14, p<0.001); while those who believed that their child needed additional help had 5.08 lower summed scores on the ESQ (SMD=-0.89, p<0.001). Average z-scores provided five meaningful categories of services, in terms of user satisfaction, compared to the national average. Our study presents evidence for the reliability and validity of the ESQ and provides recommendations for its practical use in Greece. ESQ can be used to measure experience of service and might help drive improvements in service delivery in the Greek mental health sector.

Leveraging Item Parameter Drift to Assess Transfer Effects in Vocabulary Learning

ABSTRACT Longitudinal models typically emphasize between-person predictors of change but ignore how growth varies within persons because each person contributes only one data point at each time. In contrast, modeling growth with multi-item assessments allows evaluation of how relative item performance may shift over time. While traditionally viewed as a nuisance under the label of “item parameter drift” (IPD), IPD may be of substantive interest if it reflects how learning manifests on different items or subscales at different rates. In this study, we apply the Explanatory Item Response Model to assess IPD in a causal inference context. Simulation results show that when IPD is not accounted for, both parameter estimates and standard errors can be affected. We illustrate with an empirical application to the persistence of transfer effects from a content literacy intervention , revealing how researchers can leverage IPD to achieve a more fine-grained understanding of how vocabulary learning develops over time.

Personality Measurement of Students Using Item Response Theory Models: Stability Responses from Nigerian Institutions

Item Response Theory (IRT) is utilised to detect bias in assessment tools and address issues such as faked or manipulated responses, enhancing the reliability and stability of conclusions in personality assessment. This article examines the item parameter estimates of a scale and the effectiveness of one-, two-, and three-parameter logistic models in analysing response stability in personality measurement from repeated administration. Three hundred undergraduate students at three tertiary institutions in Nigeria were sampled using a multi-stage sampling procedure. Data was collected using an adapted version of the Big Five Inventory (BFI) with a reliability coefficient of 0.85. The results showed that the item parameter estimates (mean threshold) are within the recommended benchmarks. A comparison of the three IRT models based on the Likelihood ratio (InL), Akaike Information Criterion (AIC), and Bayesian Information Criterion (BIC) values revealed that the two-parameter logistic model best fit the personality data among undergraduates from repeated administration. It is recommended that, rather than relying solely on a statistical decision-making process, IRT fit and model comparison should be applied to gain insight into the functioning of items and tests.

Impact of Violating Unidimensionality on Rasch Calibration for Mixed-Format Tests

ABSTRACT Mixed-format tests that include both multiple-choice (MC) and constructed-response (CR) items have become widely used in many large-scale assessments. When an item response theory (IRT) model is used to score a mixed-format test, the unidimensionality assumption may be violated if the CR items measure a different construct from that measured by MC items. In this simulation study, we explore the impact of different degrees of violation on the unidimensionality assumption on Rasch item parameter recovery and model-data fit statistics. Our results suggest that, when the unidimensionality assumption is violated for mixed-format tests, both estimated Rasch item parameters and model-data fit statistics are distorted, with greater impact corresponding to higher degrees of violations. The findings from this study may provide guidance for test practitioners for working with mixed-format tests within the Rasch context.

Stability of estimation item parameter in IRT dichotomy considering the number of participants

This research is related to item response theory (IRT) which is needed to measure the goodness of a test set, while item parameter estimation is needed to determine the technical properties of a test item. Stability of item parameter estimation is conducted to determine the minimum sample that can be used to obtain good item parameter estimation results. The purpose of this study is to describe the effect of the number of test takers on the stability of item parameter estimation with the Bayes method (expected a posteriori, EAP) on dichotomous data. This research is an exploratory descriptive research with a bootstrap approach using the EAP method. The EAP method is performed by modifying the likelihood and function to include prior information about the participant's 9 score. Bootstrapping on the original data is done to take bootstrap samples. with ten different sample sizes of 100, 150, 250, 300, 500, 700, 1,000, 1,500, 2,000, 2,500 were then replicated ten times and grain parameter estimation was performed. Each sample data with ten replications was calculated Root Mean Squared Difference (RMSD) value. The results showed that the 2PL model was chosen as the best model. The RMSD value obtained proves that many test participants affect the stability of item parameter estimation on dichotomous data with the 2PL model. The minimum sample to ensure the stability of item parameter estimates with the 2PL model is 1,000 test participants.

A Comparison of Limited Information Estimation Methods for the Two-Parameter Normal-Ogive Model with Locally Dependent Items

The two-parameter normal-ogive (2PNO) model is one of the most popular item response theory (IRT) models for analyzing dichotomous items. Consistent parameter estimation of the 2PNO model using marginal maximum likelihood estimation relies on the local independence assumption. However, the assumption of local independence might be violated in practice. Likelihood-based estimation of the local dependence structure is often computationally demanding. Moreover, many IRT models that model local dependence do not have a marginal interpretation of item parameters. In this article, limited information estimation methods are reviewed that allow the convenient and straightforward handling of local dependence in estimating the 2PNO model. In detail, pairwise likelihood, weighted least squares, and normal-ogive harmonic analysis robust method (NOHARM) estimation are compared with marginal maximum likelihood estimation that ignores local dependence. A simulation study revealed that item parameters can be consistently estimated with limited information methods. At the same time, marginal maximum likelihood estimation resulted in biased item parameter estimates in the presence of local dependence. From a practical perspective, there were only minor differences regarding the statistical quality of item parameter estimates of the different estimation methods. Differences between the estimation methods are also compared for two empirical datasets.

Estimation of Standard Error, Linking Error, and Total Error for Robust and Nonrobust Linking Methods in the Two-Parameter Logistic Model

The two-parameter logistic (2PL) item response theory model is a statistical model for analyzing multivariate binary data. In this article, two groups are brought onto a common metric using the 2PL model using linking methods. The linking methods of mean–mean linking, mean–geometric–mean linking, and Haebara linking are investigated in nonrobust and robust specifications in the presence of differential item functioning (DIF). M-estimation theory is applied to derive linking errors for the studied linking methods. However, estimated linking errors are prone to sampling error in estimated item parameters, thus resulting in artificially increased the linking error estimates in finite samples. For this reason, a bias-corrected linking error estimate is proposed. The usefulness of the modified linking error estimate is demonstrated in a simulation study. It is shown that a simultaneous assessment of the standard error and linking error in a total error must be conducted to obtain valid statistical inference. In the computation of the total error, using the bias-corrected linking error estimate instead of the usually employed linking error provides more accurate coverage rates.

The Rank-2PL IRT Models for Forced-Choice Questionnaires: Maximum Marginal Likelihood Estimation with an EM Algorithm

The rank two-parameter logistic (Rank-2PL) item response theory models refer to a set of models applying the 2PL model in a sequential ranking process that occurs in forced-choice questionnaires. The multi-unidimensional pairwise preference with 2PL model (MUPP-2PL) is a Rank-2PL model for items with two statements. Focusing on items with three statements, we develop a maximum marginal likelihood estimation with an expectation-maximization algorithm to estimate item parameters and their standard errors. A simulation study is conducted to check parameter recovery, and then the model is applied to a real dataset. Finally, the findings are summarized and discussed, and future research is suggested.

Investigating Directional Invariance in an Item Response Tree Model for Extreme Response Style and Trait-Based Unfolding Responses.

Item response tree (IRTree) approaches have received increasing attention in the response style literature due to their capability to partial out response style latent traits from content-related latent traits by considering separate decisions for agreement and level of agreement. Additionally, it has shown that the functioning of the intensity of agreement decision may depend upon the agreement decision with an item, so that the item parameters and person parameters may differ by direction of agreement; when the parameters across direction are the same, this is called directional invariance. Furthermore, for non-cognitive psychological constructs, it has been argued that the response process may be best described as following an unfolding process. In this study, a family of IRTree models to handle unfolding responses with the agreement decision following the hyperbolic cosine model and the intensity of agreement decision following a graded response model is investigated. This model family also allows for investigation of item- and person-level directional invariance. A simulation study is conducted to evaluate parameter recovery; model parameters are estimated with a fully Bayesian approach using JAGS (Just Another Gibbs Sampler). The proposed modeling scheme is demonstrated with two data examples with multiple model comparisons allowing for varying levels of directional invariance and unfolding versus dominance processes. An approach to visualizing the final model item response functioning is also developed. The article closes with a short discussion about the results.

Beyond group comparisons: Accounting for intersectional sources of bias in international survey measures

International surveys are increasingly being used to understand nonacademic outcomes like math and science motivation, and to inform education policy changes within countries. Such instruments assume that the measure works consistently across countries, ethnicities, and languages—that is, they assume measurement invariance. While studies have already demonstrated that some items in international survey measures are noninvariant using basic group comparisons, they do not investigate complex, intersectional sources of bias that go beyond group membership. In this study, we use an emergent method to examine the sensitivity of item parameters from the Programme for International Student Assessment (PISA) survey instruments to intersectional sources of bias. Results indicate that non-invariance exists for most of the items examined, which can change individual scores after accounting for the moderators. Although country-level ranking did not change substantively after accounting for bias, policymaking is likely to be influenced negatively if such sources of non-invariance are not addressed.

Analytical Comparison of Item Statistics Employing IRT 3PLM in WASSCE and NECO SSCE 2023 English Language Multiple Choice Tests

This assessment encompassed the analytical comparison of item statistics employing the Item Response Theory Three Parameter Logistic Model (IRT 3PLM) in WASSCE and NECO SSCE 2023 English Language multiple choice tests. Three hypothesised questions were articulated. In this study, a survey research design was utilised. This investigation’s population comprised 72,400 English Language SS 3 students in 384 public senior secondary schools in Ogun State. A multistage sampling procedure was applied to pick a sample size of 8,000 English Language SS 3 students. The assessment instruments used were WASSCE and NECO SSCE 2023 English Language multiple choice tests. The data collected were subjected to IRT 3PLM using jMetrik psychometric software to generate the item parameters. All hypothesised questions were tested for significance utilising independent samples t-test statistics. This assessment’s outcomes divulged a significant dissimilarity in the guessing parameter between WASSCE and NECO SSCE 2023 English Language multiple-choice items. However, no significant dissimilarities existed in the discrimination and difficulty parameters between the two. The pragmatic extrapolation of this analytical comparison accentuated in the findings alluded that employing IRT 3PLM to establish the item statistics before administering the tests could further enhance the quality of English Language multiple choice items developed by WAEC and NECO.

Nonparametric CD-CAT for multiple-choice items: Item selection method and Q-optimality.

Computerized adaptive testing for cognitive diagnosis (CD-CAT) achieves remarkable estimation efficiency and accuracy by adaptively selecting and then administering items tailored to each examinee. The process of item selection stands as a pivotal component of a CD-CAT algorithm, with various methods having been developed for binary responses. However, multiple-choice (MC) items, an important item type that allows for the extraction of richer diagnostic information from incorrect answers, have been underemphasized. Currently, the Jensen-Shannon divergence (JSD) index introduced by Yigit etal. (Applied Psychological Measurement, 2019, 43, 388) is the only item selection method exclusively designed for MC items. However, the JSD index requires a large sample to calibrate item parameters, which may be infeasible when there is only a small or no calibration sample. To bridge this gap, the study first proposes a nonparametric item selection method for MC items (MC-NPS) by implementing novel discrimination power that measures an item's ability to effectively distinguish among different attribute profiles. A Q-optimal procedure for MC items is also developed to improve the classification during the initial phase of a CD-CAT algorithm. The effectiveness and efficiency of the two proposed algorithms were confirmed by simulation studies.

Extending the PROMIS item bank “ability to participate in social roles and activities”: a psychometric evaluation using IRT

ObjectiveOur objective was to explore whether the extension of the PROMIS item bank Ability to Participate in Social Roles and Activities (APSRA) with new items would result in more effective targeting (i.e., selecting items that are appropriate for each individual's trait level), and more reliable measurements across all latent trait levels.MethodsA sample of 1,022 Dutch adults completed all 35 items of the original item bank plus 17 new items (in Dutch). The new items presented in this publication have been translated provisionally from Dutch into English for presentation purposes. We evaluated the basic IRT assumptions unidimensionality, local independence, and monotonicity. Furthermore, we examined the item parameters, and assessed differential item functioning (DIF) for sex, education, region, age, and ethnicity. In addition, we compared the test information functions, item parameters, and θ scores, for the original and extended item bank in order to assess whether the measurement range had improved.ResultsWe found that the extended item bank was compatible with the basic IRT assumptions and showed good reliability. Moreover, the extended item bank improved the measurement in the lower trait range, which is important for reliably assessing functioning in clinical populations (i.e., persons reporting lower levels of participation).ConclusionWe extended the PROMIS-APSRA item bank and improved its psychometric quality. Our study contributes to PROMIS measurement innovation, which allows for the addition of new items to existing item banks, without changing the interpretation of the scores and while maintaining the comparability of the scores with other PROMIS instruments.

Development and application of the Meal and Snack Assessment (MESA) quality scale for children and adolescents using item response theory

BackgroundMeals differ in terms of food items and nutritional quality. The aim of the present study was to propose a scale to measure the meals quality of schoolchildren according to food processing degree, perform a preliminary evaluation of the scale's validity and reliability and apply the scale to a representative sample of schoolchildren in a city in southern Brazil.MethodsA methodological study based on the generalized graded unfolding model (GGUM) of item response theory (IRT) with analysis of secondary data was carried out in 6,399 schoolchildren of 6-15y-old attending 2nd to 5th grades of public elementary schools in Florianópolis, Brazil, in 2013–2015 who answered the validated Food Intake and Physical Activities of Schoolchildren (WebCAAFE) questionnaire. Meal quality was the latent trait. The steps for the development of the scale included: latent trait definition; item generation; dimensionality analysis; estimation of item parameters; scale levels definition; assessment of validity and reliability; and assessment of the meal quality of a subsample of schoolchildren aged 7 to 12 years (n = 6,372).ResultsEleven out of eighteen items had adequate parameters, without differential item functioning for sex or age. Meal quality was categorized into three levels: healthy, mixed, and unhealthy. Higher scores indicate a greater prevalence of ultra-processed foods in daily meals. Most schoolchildren had mixed (40.6%) and unhealthy (41%) meal patterns.ConclusionsIRT analysis allowed the development of the scale, which measures the quality of meals and snacks based on the degree of food processing. At all snack times, there was a higher frequency of ultra-processed foods consumption, therefore foods consumed as snacks are a potential focus for nutritional interventions.

Does your job shape your experience or interpretation of workplace telepressure? Exploring measurement invariance across occupational characteristics

Workplace telepressure is a psychological experience related to work-related messages. Research on measurement invariance for telepressure is scarce, especially with respect to occupations. This paper used a moderated nonlinear factor analysis technique to examine how occupational characteristics predict telepressure experiences and differential ratings across two studies with full-time workers. Email-related behaviors and demands predicted factor means and variances of telepressure, but there were no consistent results for job control and time pressure. Invariance testing at the item-level showed that occupational characteristics did not moderate most item parameter estimates; that is, employees interpreted workplace telepressure items equivalently regardless of occupational context.

A neural network paradigm for modeling psychometric data and estimating IRT model parameters: Cross estimation network.

This paper presents a novel approach known as the cross estimation network (CEN) for fitting the datasets obtained from psychological or educational tests and estimating the parameters of item response theory (IRT) models. The CEN is comprised of two subnetworks: the person network (PN) and the item network (IN). The PN processes the response pattern of individual respondent and generates an estimate of the underlying ability, while the IN takes in the response pattern of individual item and outputs the estimates of the item parameters. Four simulation studies and an empirical study were comprehensively and rigorously conducted to investigate the performance of CEN on parameter estimation of the two-parameter logistic model under various testing scenarios. Results showed that CEN effectively fit the training data and produced accurate estimates of both person and item parameters. The trained PN and IN adhered to AI principles and acted as intelligent agents, delivering commendable evaluations for even unseen patterns of new respondents and items.