Item Response Theory Parameter Estimates Research Articles

Methods for measuring career readiness of high school students: based on multidimensional item response theory and text mining

In contemporary society, career readiness holds paramount significance for individual life, exerting a direct influence on initial employment, job satisfaction, and the sense of career identity. Framed within multidimensional item response theory and text mining, this study embarks on exploring assessment methodologies for high school students’ career readiness by revising the “Career Readiness Questionnaire – Adolescent Version” and employing text mining techniques. Study One collected 1261 valid data points through cluster sampling. With the aid of Bayesian multivariate item response theory parameter estimation procedures and R language, the career readiness measurement tool was revised, yielding a concise scale that aligns with psychometric requirements. The research findings indicated that the concept of “career readiness” is more suitable for the multidimensional graded response model than for the bifactor model. The dataset’s discrimination parameters fell within the range of [1.59, 3.84], the difficulty parameters fell between [−2.91, 2.24], and the peak values of the maximum information functions fell within [0.24, 2.35]. After six items with the lowest peaks were removed (Items 4, 5, 6, 31, 32, and 33), the remaining 30 items composed the Chinese concise version “Career Readiness Questionnaire – Adolescent Version,” with discrimination parameters ranging from [1.45, 3.38], difficulty parameters between [−3.31, 1.76], and maximum information function peaks within [0.50, 2.64]. Building upon the effective participants from Study One, Study Two matched questionnaire data with textual information, resulting in 1012 valid participants. Leveraging text mining, a machine learning model was constructed to predict high school students’ career readiness based on essay texts. The results of Study 2 prove that the revised lexicon was more accurate in feature extraction. Building upon this, the machine learning model for essay text demonstrated excellent performance in predicting career readiness, with random forest outperforming the other algorithms. This study provides a novel approach for schools and parents to comprehend the state of career readiness among high school students, offering a convenient and effective tool for educational activities related to students’ career development.

Application of Sampling Variance of Item Response Theory Parameter Estimates in Detecting Outliers in Common Item Equating.

In common item equating, the existence of item outliers may impact the accuracy of equating results and bring significant ramifications to the validity of test score interpretations. Therefore, common item equating should involve a screening process to flag outlying items and exclude them from the common item set before equating is conducted. The current simulation study demonstrated that the sampling variance associated with the item response theory (IRT) item parameter estimates can help detect outliers in the common items under the 2-PL and 3-PL IRT models. The results showed the proposed sampling variance statistic (SV) outperformed the traditional displacement method with cutoff values of 0.3 and 0.5 along a variety of evaluation criteria. Based on the favorable results, item outlier detection statistics based on estimated sampling variability warrant further consideration in both research and practice.

An Investigation of Age-Related Differential Item Functioning in the EQ-5D-5L Using Item Response Theory and Logistic Regression

ObjectivesIn economic evaluations, quality of life is measured using patient-reported outcome measures (PROMs), such as the EQ-5D-5L. A key assumption for the validity of PROMs data is measurement invariance, which requires that PROM items and response options are interpreted the same across respondents. If measurement invariance is violated, PROMs exhibit differential item functioning (DIF), whereby individuals from different groups with the same underlying health respond differently, potentially biasing scores. One important group of healthcare consumers who have been shown to have different views or priorities over health is older adults. This study investigates age-related DIF in the EQ-5D-5L using item response theory (IRT) and ordinal logistic regression approaches. MethodsMultiple-group IRT models were used to investigate DIF, by assessing whether older adults aged 65+ years and younger adults aged 18 to 64 years with the same underlying health had different IRT parameter estimates and expected item and EQ-5D-5L level sum scores. Ordinal logistic regression was also used to examine whether DIF resulted in meaningful differences in expected EQ level sum scores. Effect sizes examined whether DIF indicated meaningful score differences. ResultsThe anxiety/depression item exhibited meaningful DIF in both approaches, with older adults less likely to report problems. Pain/discomfort and mobility exhibited DIF to a lesser extent. ConclusionsWhen using the EQ-5D-5L to evaluate interventions and make resource allocation decisions, scoring bias due to DIF should be controlled for to prevent inefficient service provision, where the most cost-effective services are not provided, which could be detrimental to patients and the efficiency of health budgets.

Detecting Differential Item Functioning: Item Response Theory Methods Versus the Mantel-Haenszel Procedure

This Monte Carlo study assessed Type I error in differential item functioning analyses using Lord’s chi-square (LC), Likelihood Ratio Test (LRT), and Mantel-Haenszel (MH) procedure. Two research interests were investigated: item response theory (IRT) model specification in LC and the LRT and continuity correction in the MH procedure. This study enhances the literature by investigating LC and the LRT using correct and incorrect model-data fit and comparing those results to the MH procedure. There were three fixed factors (number of test items, IRT parameter estimation method, and item parameter equating) and four varied factors (IRT model used to generate data and fit the data, sample size, and impact). The findings suggested the MH procedure without the continuity correction is best based on Type I error rate.

Reliability of Differential Item Functioning in Alcohol Use Disorder: Bayesian Meta-Analysis of Criteria Discrimination Estimates.

Numerous studies leverage item response theory (IRT) methods to examine measurement characteristics of alcohol use disorder (AUD) diagnostic criteria. Less work has examined the consistency of AUD IRT parameter estimates, an essential step for establishing measurement invariance, making statements about symptom diagnosticity, and validating the theoretical construct. A Bayesian meta-analysis of IRT discrimination values for AUD criteria across 33 independent samples (Total N = 321,998) revealed that overall consistency of AUD criteria discriminations was low (generalized intraclass correlation range = .105-.249). However, specific study characteristics accounted for substantial variability, suggesting that the unreliability is partially systematic. We replicated evidence of differential item functioning (DIF) via established factors (e.g., age, gender), but the magnitudes were small compared with DIF associated with assessment instrument. These results offer practical recommendations regarding which instruments to use when specific AUD criteria are of interest and which criteria are most sensitive when comparing demographic groups.

The Uniform Prior for Bayesian Estimation of Ability in Item Response Theory Models

Item Response Theory (IRT) models traditionally assume a normal distribution for ability. Although normality is often a reasonable assumption for ability, it is rarely met for observed scores in educational and psychological measurement. Assumptions regarding ability distribution were previously shown to have an effect on IRT parameter estimation. In this study, the normal and uniform distribution prior assumptions for ability were compared for IRT parameter estimation when the actual distribution was either normal or uniform. A simulation study that included a short test with a small sample size and a long test with a large sample size was conducted for this purpose. The results suggested using a uniform distribution prior for ability to achieve more accurate estimates of the ability parameter in the 2PL and 3PL models when the true distribution of ability is not known. For the Rasch model, an explicit pattern that could be used to obtain more accurate item parameter estimates was not found.

Proceedings of Contributed Presentations 10th World Federation of Chiropractic/Association of Chiropractic Colleges Education Conference

Proceedings of Contributed Presentations 10th World Federation of Chiropractic/Association of Chiropractic Colleges Education Conference

Item response theory analysis of Centers for Disease Control and Prevention Health-Related Quality of Life (CDC HRQOL) items in adults with arthritis.

BackgroundExamine the feasibility of performing an item response theory (IRT) analysis on two of the Centers for Disease Control and Prevention health-related quality of life (CDC HRQOL) modules – the 4-item Healthy Days Core Module (HDCM) and the 5-item Healthy days Symptoms Module (HDSM). Previous principal components analyses confirm that the two scales both assess a mix of mental (CDC-MH) and physical health (CDC-PH). The purpose is to conduct item response theory (IRT) analysis on the CDC-MH and CDC-PH scales separately.Methods2182 patients with self-reported or physician-diagnosed arthritis completed a cross-sectional survey including HDCM and HDSM items. Besides global health, the other 8 items ask the number of days that some statement was true; we chose to recode the data into 8 categories based on observed clustering. The IRT assumptions were assessed using confirmatory factor analysis and the data could be modeled using an unidimensional IRT model. The graded response model was used for IRT analyses and CDC-MH and CDC-PH scales were analyzed separately in flexMIRT.ResultsThe IRT parameter estimates for the five-item CDC-PH all appeared reasonable. The three-item CDC-MH did not have reasonable parameter estimates.ConclusionsThe CDC-PH scale is amenable to IRT analysis but the existing The CDC-MH scale is not. We suggest either using the 4-item Healthy Days Core Module (HDCM) and the 5-item Healthy days Symptoms Module (HDSM) as they currently stand or the CDC-PH scale alone if the primary goal is to measure physical health related HRQOL.

Rasch Model Parameter Estimation in the Presence of a Nonnormal Latent Trait Using a Nonparametric Bayesian Approach.

Standard approaches for estimating item response theory (IRT) model parameters generally work under the assumption that the latent trait being measured by a set of items follows the normal distribution. Estimation of IRT parameters in the presence of nonnormal latent traits has been shown to generate biased person and item parameter estimates. A number of methods, including Ramsay curve item response theory, have been developed to reduce such bias, and have been shown to work well for relatively large samples and long assessments. An alternative approach to the nonnormal latent trait and IRT parameter estimation problem, nonparametric Bayesian estimation approach, has recently been introduced into the literature. Very early work with this method has shown that it could be an excellent option for use when fitting the Rasch model when assumptions cannot be made about the distribution of the model parameters. The current simulation study was designed to extend research in this area by expanding the simulation conditions under which it is examined and to compare the nonparametric Bayesian estimation approach to the Ramsay curve item response theory, marginal maximum likelihood, maximum a posteriori, and the Bayesian Markov chain Monte Carlo estimation method. Results of the current study support that the nonparametric Bayesian estimation approach may be a preferred option when fitting a Rasch model in the presence of nonnormal latent traits and item difficulties, as it proved to be most accurate in virtually all scenarios that were simulated in this study.

IRT Parameter Estimation With Response Times as Collateral Information

Hierarchical modeling of responses and response times on test items facilitates the use of response times as collateral information in the estimation of the response parameters. In addition to the regular information in the response data, two sources of collateral information are identified: (a) the joint information in the responses and the response times summarized in the estimates of the second-level parameters and (b) the information in the posterior distribution of the response parameters given the response times. The latter is shown to be a natural empirical prior distribution for the estimation of the response parameters. Unlike traditional hierarchical item response theory (IRT) modeling, where the gain in estimation accuracy is typically paid for by an increase in bias, use of this posterior predictive distribution improves both the accuracy and the bias of IRT parameter estimates. In an empirical study, the improvements are demonstrated for the estimation of the person and item parameters in a three-parameter response model.

Home fruit, juice, and vegetable pantry management and availability scales: A validation

Home fruit, 100% juice, and vegetables (FJV) availability is related to increased FJV consumption by children. While FJV must be purchased for use in the home, no scales have been reported on home FJV pantry management practices. A scale for home FJV pantry management practices was generated from focus group discussions with diverse 162 food shoppers. A commonly used scale of home FJV availability was also assessed. A grocery store intercept survey recruited 171 food shoppers with children in front of supermarkets and grocery stores. Survey instruments were administered twice, separated by 6 weeks. Single dimensionality was observed for each scale. Item Response Theory parameter estimates revealed easily interpreted patterns in the sequence of items by difficulty of response. These scales are available to help better understand influences on family FJV purchase decisions.

Parent outcome expectancies for purchasing fruit and vegetables: a validation

To validate four scales -- outcome expectancies for purchasing fruit and for purchasing vegetables, and comparative outcome expectancies for purchasing fresh fruit and for purchasing fresh vegetables versus other forms of fruit and vegetables (F&V). Survey instruments were administered twice, separated by 6 weeks. Recruited in front of supermarkets and grocery stores; interviews conducted by telephone. One hundred and sixty-one food shoppers with children (18 years or younger). Single dimension scales were specified for fruit and for vegetable purchasing outcome expectancies, and for comparative (fresh vs. other) fruit and vegetable purchasing outcome expectancies. Item Response Theory parameter estimates revealed easily interpreted patterns in the sequence of items by difficulty of response. Fruit and vegetable purchasing and fresh fruit comparative purchasing outcome expectancy scales were significantly correlated with home F&V availability, after controlling for social desirability of response. Comparative fresh vegetable outcome expectancy scale was significantly bivariately correlated with home vegetable availability, but not after controlling for social desirability. These scales are available to help better understand family F&V purchasing decisions.

The factor structure of test task characteristics and examinee performance

The present study focuses on the task characteristics of reading passages and key sentences in a test of second language reading. Using a new methodological approach to describe variation in test task characteristics and explore how differences in these characteristics might relate to examinee performance, it posed the two following research questions: First, how do the characteristics of the texts used in a high-stakes test of English for Academic Purposes reading vary? Second, what relationships exist between its text characteristics and examinee performance? An expanded test task characteristics instrument was constructed, following Freedle and Kostin (1993) and Bachman et al. (1996), and adding a large number of syntactic features (Celce-Murcia and Larsen-Freeman, 1999). Ratings and numerical counts were compiled for three forms of the Test of English as a Foreign Language (TOEFL) Reading Comprehension Section. Taking items as the object of measurement, the results were then used in a series of exploratory and confirmatory factor analyses, along with IRT (item response theory) parameter estimates for the items in question.

DIF Identification Using HGLM for Polytomous Items

The relationship between the hierarchical generalized linear model (HGLM) and item response theory (IRT) models has been demonstrated for dichotomous items. The current study demonstrated the use of the HGLM for polytomous items (termed PHGLM) for identification of differential item functioning (DIF). First, the algebraic equivalence between parameterizations of a constrained form of Muraki's rating scale model and PHGLM was demonstrated. Next, as a precursor to using the PHGLM for DIF identification, a simulation was conducted to compare estimation of item and person latent trait parameters using HLM 5 and PARSCALE software. The resulting PHGLM and IRT parameter estimates were comparable. Last, a simulation was conducted that compared the performance of PHGLM with that of the generalized Mantel-Haenszel statistic for DIF identification. Both approaches performed similarly. Benefits and limitations of using PHGLM for DIF identification are presented. Index terms: hierarchical linear modeling, item response theory, polytomous items, parameter estimation, differential item functioning, attitude measurement.

A Comparison of Bootstrap Standard Errors of IRT Equating Methods for the Common-Item Nonequivalent Groups Design

The primary purpose of this study was to compare bootstrap standard errors of 5 item response theory (IRT) equating methods for the common-item nonequivalent groups design. For true-score (Method 1) and observed-score (Method 2) equating, IRT parameters were estimated separately, and a linear scaling transformation method was used to rescale the IRT parameter estimates for Form X onto the Form Y scale. For IRT chained true-score equating (Method 3), IRT parameters for Form X and Form Y were estimated separately, and then IRT chained true-score equating was performed. For the last 2 methods, IRT parameters for both forms were estimated simultaneously. Using the simultaneously estimated parameter estimates, IRT true-score (Method 4) and observed-score (Method 5) equatings were performed. For each method, the standard deviation was computed over 500 bootstrap replications to obtain the standard error of IRT equating at each raw score point for the new form. The estimated bootstrap standard errors for Methods 4 and 5 were slightly less than those for Methods 1 and 2. Method 3 produced the greatest standard errors. However, the standard errors for all 5 methods were small enough to suggest that standard errors of equating less than 0.1 standard deviation units could be obtained with any method, even with sample sizes of 500.

FIELD TEST OF A COMPUTER‐BASED GRE GENERAL TEST

ETS Research Report SeriesVolume 1993, Issue 1 p. i-47 ArticleFree Access FIELD TEST OF A COMPUTER-BASED GRE GENERAL TEST Gary A. Schaeffer, Gary A. SchaefferSearch for more papers by this authorClyde M. Reese, Clyde M. ReeseSearch for more papers by this authorManfred Steffen, Manfred SteffenSearch for more papers by this authorRobert L. McKinley, Robert L. McKinleySearch for more papers by this authorCraig N. Mills, Craig N. MillsSearch for more papers by this author Gary A. Schaeffer, Gary A. SchaefferSearch for more papers by this authorClyde M. Reese, Clyde M. ReeseSearch for more papers by this authorManfred Steffen, Manfred SteffenSearch for more papers by this authorRobert L. McKinley, Robert L. McKinleySearch for more papers by this authorCraig N. Mills, Craig N. MillsSearch for more papers by this author First published: June 1993 https://doi.org/10.1002/j.2333-8504.1993.tb01518.xCitations: 23AboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Citing Literature Volume1993, Issue1June 1993Pages i-47 ReferencesRelatedInformation

A Comparative Study of the Effects of Recency of Instruction on the Stability of IRT and Conventional Item Parameter Estimates

A potential concern for individuals interested in using item response theory (IRT) with achievement test data is that such tests have been specifically designed to measure content areas related to course curriculum and students taking the tests at different points in their coursework may not constitute samples from the same population. In this study, data were obtained from three administrations of two forms of a Biology achievement test. Data from the newer of the two forms were collected at a spring administration, made up of high school sophomores just completing the Biology course, and at a fall administration, made up mostly of seniors who completed their instruction in the course from 6–18 months prior to the test administration. Data from the older form, already on scale, were collected at only a fall administration, where the sample was comparable to the newer form fall sample. IRT and conventional item difficulty parameter estimates for the common items across the two forms were compared for each of the two form/sample combinations. In addition, conventional and IRT score equatings were performed between the new and old forms for each o f the form sample combinations. Widely disparate results were obtained between the equatings based on the two form/sample combinations. Conclusions are drawn about the use o f both classical test theory and IRT in situations such as that studied, and implications o f the results for achievement test validity are also discussed

Putting Item Response Theory into Perspective

Abstract Item response theory (IRT) is a mathematically rigorous and computationally complex measurement theory which has potential utility in psychomotor measurement. The present paper complements Spray's heuristic description of IRT and one of its applications “adaptive testing“; presenting additional applications of IRT in physical education, drawbacks of IRT parameter estimation and theory, and recommendations for future IRT research in physical education.