Generalized Partial Credit Model Research Articles

Exploration of the linear and nonlinear relationships between learning strategies and mathematics achievement in South Korea using the nominal response model : PISA 2012

Learning strategies have been recognized as important predictors of mathematical achievement. In recent studies, it has been found that Asian students use combined learning strategies, primarily including metacognitive strategies, rather than rote memorization. To the best of the authors’ knowledge, there is only one prior study including South Korea in investigations of the relationship between learning strategies and mathematics achievement in PISA 2012. In that study, students were classified into groups using specific learning strategies, and their mathematics achievements were compared. There are two research gaps: (1) previous studies insufficiently explored how students use learning strategies in the South Korean education system, and (2) there is little research applying the nominal response model (NRM) to explore the association between learning strategy use and mathematics achievement in PISA 2012. Thus, the present study explores to what extent the NRM fits the data compared to the generalized partial credit model (GPCM). We created a learning strategy score from the NRM for South Korean students in PISA 2012 (N = 3,310). Then, using correlation analysis and quadratic regression analysis, we identified linear and nonlinear relations between learning strategy scores from the NRM and mathematics achievement. The findings indicated that (1) NRM was a better fit for creating learning strategy scores than GPCM, (2) the average correlation coefficient between the learning strategy score and mathematics achievement was 0.18 (p < .05), and (3) for the curvilinear relationship between the learning strategy score and mathematics achievement, the standardized quadratic coefficient was − 0.090 (p < .001). Overall, the NRM represents an appropriate model for explaining the relationship between learning strategy and mathematical achievement. Additionally, high-performing South Korean students tend to primarily use metacognitive strategies with memorization. The negative quadratic coefficient captured the limited effect of the primary use of metacognitive strategies with memorization. The implications for the South Korean education system are discussed.

Genetic associations with dementia-related proteinopathy: Application of item response theory.

Although dementia-related proteinopathy has a strong negative impact on public health, and is highly heritable, understanding of the related genetic architecture is incomplete. We applied multidimensional generalized partial credit modeling (GPCM) to test genetic associations with dementia-related proteinopathies. Data were analyzed to identify candidate single nucleotide variants for the following proteinopathies: Aβ, tau, α-synuclein, and TDP-43. Final included data comprised 966 participants with neuropathologic and WGS data. Three continuous latent outcomes were constructed, corresponding to TDP-43-, Aβ/Tau-, and α-synuclein-related neuropathology endophenotype scores. This approach helped validate known genotype/phenotype associations: for example, TMEM106B and GRN were risk alleles for TDP-43 pathology; and GBA for α-synuclein/Lewy bodies. Novel suggestive proteinopathy-linked alleles were also discovered, including several (SDHAF1, TMEM68, and ARHGEF28) with colocalization analyses and/or high degrees of biologic credibility. A novel methodology using GPCM enabled insights into gene candidates for driving misfolded proteinopathies. Latent factor scores for proteinopathies were estimated using a generalized partial credit model. The three latent continuous scores corresponded well with proteinopathy severity. Novel genes associated with proteinopathies were identified. Several genes had high degrees of biologic credibility for dementia risk factors.

Using IRTree Models to Promote Selection Validity in the Presence of Extreme Response Styles.

The measurement of psychological constructs is frequently based on self-report tests, which often have Likert-type items rated from "Strongly Disagree" to "Strongly Agree". Recently, a family of item response theory (IRT) models called IRTree models have emerged that can parse out content traits (e.g., personality traits) from noise traits (e.g., response styles). In this study, we compare the selection validity and adverse impact consequences of noise traits on selection when scores are estimated using a generalized partial credit model (GPCM) or an IRTree model. First, we present a simulation which demonstrates that when noise traits do exist, the selection decisions made based on the IRTree model estimated scores have higher accuracy rates and have less instances of adverse impact based on extreme response style group membership when compared to the GPCM. Both models performed similarly when there was no influence of noise traits on the responses. Second, we present an application using data collected from the Open-Source Psychometrics Project Fisher Temperament Inventory dataset. We found that the IRTree model had a better fit, but a high agreement rate between the model decisions resulted in virtually identical impact ratios between the models. We offer considerations for applications of the IRTree model and future directions for research.

Analyzing Polytomous Test Data: A Comparison Between an Information-Based IRT Model and the Generalized Partial Credit Model

Item response theory (IRT) models the relationship between the possible scores on a test item against a test taker’s attainment of the latent trait that the item is intended to measure. In this study, we compare two models for tests with polytomously scored items: the optimal scoring (OS) model, a nonparametric IRT model based on the principles of information theory, and the generalized partial credit (GPC) model, a widely used parametric alternative. We evaluate these models using both simulated and real test data. In the real data examples, the OS model demonstrates superior model fit compared to the GPC model across all analyzed datasets. In our simulation study, the OS model outperforms the GPC model in terms of bias, but at the cost of larger standard errors for the probabilities along the estimated item response functions. Furthermore, we illustrate how surprisal arc length, an IRT scale invariant measure of ability with metric properties, can be used to put scores from vastly different types of IRT models on a common scale. We also demonstrate how arc length can be a viable alternative to sum scores for scoring test takers.

Numeric rating scale for pain should be used in an ordinal but not interval manner. A retrospective analysis of 346,892 patient reports of the quality improvement in postoperative pain treatment registry.

To assess postoperative pain intensity in adults, the numeric rating scale (NRS) is used. This scale has shown acceptable psychometric features, although its scale properties need further examination. We aimed to evaluate scale properties of the NRS using an item response theory (IRT) approach. Data from an international postoperative pain registry (QUIPS) was analyzed retrospectively. Overall, 346,892 adult patients (age groups: 18-20 years: 1.6%, 21-30 years: 6.7%, 31-40 years: 8.3%, 41-50 years: 13.2%, 51-60 years: 17.1%, 61-70 years: 17.3%, 71-80 years: 16.4%, 81-90 years: 3.9%, >90: 0.2%) were included. Among the patients, 55.7% are female and 38% had preoperative pain. Three pain items (movement pain, worst pain, least pain) were analyzed using 4 different IRT models: partial credit model (PCM), generalized partial credit model (GPCM), rating scale model (RSM), and graded response model (GRM). Fit indices were compared to decide the best fitting model (lower fit indices indicate a better model fit). Subgroup analyses were done for sex and age groups. After collapsing the highest and the second highest response category, the GRM outperformed other models (lowest Bayesian information criterion) in all subgroups. Overlapping categories were found in category boundary curves for worst and minimum pain and particularly for higher pain ratings. Response category widths differed depending on pain intensity. For female, male, and age groups, similar results were obtained. Response categories on the NRS are ordered but have different widths. The interval scale properties of the NRS should be questioned. In dealing with missing linearity in pain intensity ratings using the NRS, IRT methods may be helpful.

Transcultural Validity of Concept of Internalized Stigma in Patients with Mental Illnesses (Psychometric Properties of Russian Version of Internalized Stigma of Mental Illness Scale)

INTRODUCTION: One of the commonest methods of evaluation of internalized stigma is Internalized Stigma of Mental Illness (ISMI) scale where the differences in the prevalence and perception of mental stigma in different cultures are specified.&#x0D; AIM: Validation of Russian version of ISMI scale and evaluation of peculiarities of perception of stigma by Russian-speaking patients with severe psychiatric disorders.&#x0D; MATERIALS AND METHODS: Four hundred thirty nine patients with severe psychiatric disorders (F2 and F3 according to the International Statistical Classification of Diseases and Related Health Problems, 10th revision) were examined. The psychometric properties of the Russian version of ISMI scale were evaluated using internal consistency analysis. Exploratory and confirmatory factor analysis (EFA and CFA, respectively) were performed. To check age- and gender-related invariance, MIMIC and MIFA models were used, and discriminative power and accuracy of scoring were identified using generalized partial credit model (GPCM). The results of EFA were compared with the data obtained in the Regional Clinical Hospital (n = 120) using independent alternative version of ISMI translation into Russian.&#x0D; RESULTS: The internal consistency of the Russian version of ISMI questionnaire was high: Chronbachs 0.90. EFA permitted to identify 4 factors. The first factor (Demoralization, F1) combined elements corresponding to subscale 1 (Alienation) and subscale 2 (Social withdrawal) of the original version of ISMI questionnaire. The other three factors identified in this study according to the content of the items, corresponded to the remaining subscales of the original ISMI scale. In all kinds of the conducted analysis, item 14 had low factor weight. CFA confirmed the four-factor structure. In the sample, MIMIC and MIFA showed the general age- and gender-related invariance with their insufficient power. GPCM showed good characteristics of discriminative power and difficulties in choice of scoring for all factors except the fourth one (Resistance to stigma). EFA performed on the data of the alternative sample, neither could reliably reproduce the five-factor structure of the original ISMI scale due to eclectic mixing of the items of the original scale within five isolated structures.&#x0D; CONCLUSION: The suitability of using the internalized stigma assessment tool for the Russian-speaking population has been confirmed. The study highlights the relevance and necessity of expanding transcultural studies of psychological reactions of psychiatric patients.

A Mixed Sequential IRT Model for Mixed-Format Items.

To provide more insight into an individual's response process and cognitive process, this study proposed three mixed sequential item response models (MS-IRMs) for mixed-format items consisting of a mixture of a multiple-choice item and an open-ended item that emphasize a sequential response process and are scored sequentially. Relative to existing polytomous models such as the graded response model (GRM), generalized partial credit model (GPCM), or traditional sequential Rasch model (SRM), the proposed models employ an appropriate processing function for each task to improve conventional polytomous models. Simulation studies were carried out to investigate the performance of the proposed models, and the results indicated that all proposed models outperformed the SRM, GRM, and GPCM in terms of parameter recovery and model fit. An application illustration of the MS-IRMs in comparison with traditional models was demonstrated by using real data from TIMSS 2007.

Comparing the score interpretation across modes in PISA: an investigation of how item facets affect difficulty

BackgroundMode effects, the variations in item and scale properties attributed to the mode of test administration (paper vs. computer), have stimulated research around test equivalence and trend estimation in PISA. The PISA assessment framework provides the backbone to the interpretation of the results of the PISA test scores. However, an identified gap in the current literature is whether mode effects have affected test score interpretation as defined by the assessment framework, and whether the interpretations of the PBA and CBA test scores are comparable.MethodsThis study uses the 2015 PISA field trial data from thirteen countries to compare test modes through a construct representation approach. It is investigated whether item facets defined by the assessment framework (e.g., different cognitive demands) affect item difficulty comparably across modes using a unidimensional two-group generalized partial credit model (GPCM).ResultsLinking the assessment framework to item difficulty using linear regression showed that for both maths and science domains, item categorisation relates to item difficulty, however for the reading domain no such conclusion was possible. In comparing PBA to CBA in representations across the three domains, maths had one facet with a significant difference in representation, reading had all three facets significantly different, and for science, four out of six facets had significant differences. Modelling items labelled “mode invariant” in PISA 2015, the results indicated that in every domain, two facets showed significant differences between the test modes. The graphical inspection of difficulty patterns confirmed that reading shows stronger differences while the patterns of the other domains were quite consistent between modes.ConclusionsThe present study shows that the mode effects on difficulty vary within the task facets proposed by the PISA assessment framework, in particular for reading. These findings shed light on whether the comparability of score interpretation between modes is compromised. Given the limitations of the link between the reading domain and item difficulty, any conclusions in this domain are limited. Importantly, the present study adds a new approach and empirical findings to the investigation of the cross-mode equivalence in PISA domains.

The effect of scoring correction and model fit on the estimation of ability parameter and person fit on polytomous item response theory

Scoring quality has been recognized as one of the important aspects that should be of concern to both test developers and users. This study aimed to investigate the effect of scoring correction and model fit on the estimation of ability parameters and person fit in the polytomous item response theory. The result of 165 students in the Statistics course (SATS4410) test at one of the universities in Indonesia was used to answer the problems in this study. The polytomous data obtained from scoring the test results were analyzed using the Item Response Theory (IRT) approach with the Partial Credit Model (PCM), Graded Response Model (GRM), and Generalized Partial Credit Model (GPCM). The effect of scoring correction and model fit on the estimation of ability and person fit was tested using multivariate analysis. Among the three models used, GRM showed the best fit based on p-value and RSMEA. The results of the analysis also showed that there was no significant effect of scoring correction and model fit on the estimation of the test taker’s ability and person fit. From the results of this study, we recommend the importance of evaluating the levels or categories used in scoring student work on a test.

Analysis of HOTS Instrument for Prospective Physics Teacher Using Generalized Partial Credit Model

This study aims to analyze item parameters' characteristics and estimate prospective physics teachers' ability on the Higher Order Thinking Skills (HOTS) Instrument using the Generalized Partial Credit Model (GPCM). The research subjects were 251 prospective physics teacher students in two universities concerned with producing prospective teacher graduates. Two-Tier Multiple Choice (TTMC) forms with a polytomous score of four categories make up the test instrument. Data analysis includes two stages: testing assumptions and the suitability of the Politomus IRT model. The results showed that the most suitable polytomous scoring IRT model was GPCM2PL. The item parameter analysis for the HOTS Instrument test shows the discriminating power parameter (a) value for all items included in the good category, namely the interval 0.00 to 2.00. The difficulty level analysis (b) also shows the percentage of 100% of items included in the medium category because the b value of all items is in the interval (-2) to (2).

AutoRasch: An R Package to Do Semi-Automated Rasch Analysis.

The R package autoRasch has been developed to perform a Rasch analysis in a (semi-)automated way. The automated part of the analysis is achieved by optimizing the so-called in-plus-out-of-questionnaire log-likelihood (IPOQ-LL) or IPOQ-LL-DIF when differential item functioning (DIF) is included. These criteria measure the quality of fit on a pre-collected survey, depending on which items are included in the final instrument. To compute these criteria, autoRasch fits the generalized partial credit model (GPCM) or the generalized partial credit model with differential item functioning (GPCM-DIF) using penalized joint maximum likelihood estimation (PJMLE). The package further allows the user to reevaluate the output of the automated method and use it as a basis for performing a manual Rasch analysis and provides standard statistics of Rasch analyses (e.g., outfit, infit, person separation reliability, and residual correlation) to support the model reevaluation.

Pamukkale eleştirel düşünme becerisi ölçeği: geçerlik ve güvenirlik çalışması

The aim of this study is to develop a valid and reliable measurement tool that measures critical thinking skills of university students. Pamukkale Critical Thinking Skills Scale was developed as two separate forms; multiple choice and open-ended. The validity and reliability studies of the multiple-choice form were constructed on two different theoretical frameworks as classical test theory and item-response theory. According to classical test theory, exploratory and confirmatory factor analyses were performed, to item-response theory, the Generalized Partial Credit Model (GPCM) for one-dimensional and multi-category scales was tested for the construct validity of the multiple-choice form of the scale. Analysis results supported the unidimensional structure of the scale. The reliability analyzes showed that the internal consistency coefficient of the scale and the item-total correlation values were high enough. The test-retest analysis results supported that the scale shows stability over time regarding the field it measures. The results of the item-response theory-based analysis also showed that the scale met the item-model fit assumptions. In the evaluation of the open-ended form of the scale, a rubric was used. Several studies were conducted on the validity and reliability of the open-ended form, and the results of the analysis provided psychometric support for the validity and reliability. As a result, Pamukkale Critical Thinking Skills Scale, which was developed in two forms, is a valid and reliable measurement tool to measure critical thinking skills of university students. The findings were discussed in the light of the literature and some suggestions were given.

Detecting Rating Scale Malfunctioning With the Partial Credit Model and Generalized Partial Credit Model.

Rating scale analysis techniques provide researchers with practical tools for examining the degree to which ordinal rating scales (e.g., Likert-type scales or performance assessment rating scales) function in psychometrically useful ways. When rating scales function as expected, researchers can interpret ratings in the intended direction (i.e., lower ratings mean "less" of a construct than higher ratings), distinguish between categories in the scale (i.e., each category reflects a unique level of the construct), and compare ratings across elements of the measurement instrument, such as individual items. Although researchers have used these techniques in a variety of contexts, studies are limited that systematically explore their sensitivity to problematic rating scale characteristics (i.e., "rating scale malfunctioning"). I used a real data analysis and a simulation study to systematically explore the sensitivity of rating scale analysis techniques based on two popular polytomous item response theory (IRT) models: the partial credit model (PCM) and the generalized partial credit model (GPCM). Overall, results indicated that both models provide valuable information about rating scale threshold ordering and precision that can help researchers understand how their rating scales are functioning and identify areas for further investigation or revision. However, there were some differences between models in their sensitivity to rating scale malfunctioning in certain conditions. Implications for research and practice are discussed.

The Big Three Perfectionism Scale-Short Form: An item response theory analysis of Italian workers.

BackgroundThe present study examined the psychometric properties of the Big Three Perfectionism Scale–Short Form (BTPS-SF) using Rasch and Mokken item response theory (IRT) analyses, which have not previously been applied to the BTPS-SF.Materials and methodsA total of 401 Italian workers (Mage = 46.78; SD = 10.1; male = 48.9%; female = 51.1%) completed the BTPS-SF questionnaire. We conducted confirmatory factor analyses of the BTPS-SF and IRT analyses using the generalized partial credit model (GPCM) and Mokken scale analysis. Discrimination and difficulty parameters were calculated. The Loevinger coefficient of scalability was computed. Item characteristic curves (ICC), test information function (TIF), and differential item functioning (DIF) for gender were calculated.ResultsA three-factor solution revealed the best fit. Thus, IRT analyses were performed for each BTPS-SF factor: rigid perfectionism (RP), self-critical perfectionism (SP), and narcissistic perfectionism (NP). All the items showed Loevinger coefficients from medium to strong and discrimination parameters from medium to very high. No DIF for gender was found.ConclusionThe Big BTPS-SF shows good psychometric properties for Italian workers. Future research is warranted to examine the findings in workers from different countries.

Development of an integrated ability assessment instrument in reaction rate material

This research aimed to develop a valid and reliable integrated ability assessment instrument (IAAI) to measure the multiple representations and chemical literacy of senior high school students related to the reaction rates. The framework developed in this research was implemented based on a 4-D model (defining, designing, developing, and disseminating). The integrated assessment consisted of five items of essay questions. The preliminary version of the IAAI was initially piloted on reviews by experts, and the paper-pencil test was given to a group of students. The obtained data were analyzed using Item Response Theory (IRT). In addition, Generalized Partial Credit Model (GPCM) model was conducted to examine the parameter quality, which estimated the test items. The content validity, construct validity, and quality of the items suggest that the integrated assessment ability instrument has a high score in validity and reliability. The analysis results of the instrument trial data indicate that the integrated abilities of students are in the medium category.

Effects of Using Double Ratings as Item Scores on IRT Proficiency Estimation

ABSTRACT This article presents the performance of item response theory (IRT) models when double ratings are used as item scores over single ratings when rater effects are present. Study 1 examined the influence of the number of ratings on the accuracy of proficiency estimation in the generalized partial credit model (GPCM). Study 2 compared the accuracy of proficiency estimation of two IRT models (GPCM versus the hierarchical rater model, HRM) for double ratings. The main findings were as follows: (a) rater effects substantially reduced the accuracy of IRT proficiency estimation; (b) double ratings relieved the negative impact of rater effects on proficiency estimation and improved the accuracy relative to single ratings; (c) IRT estimators showed different patterns in the conditional accuracy; (d) as more items and a larger number of score categories were used, the accuracy of proficiency estimation improved; and (e) the HRM consistently showed better performance than the GPCM.

Comparative Analysis of Question Item Parameters and Students' Ability between Dichotomy and Polytomic Score Versions; Research on Mathematics National Exam Test Participants

The purpose of this study was to determine the differences in the estimation of item parameters and students' abilities between the dichotomous scoring version and the polytomy scoring version. The data used in this study is the response of participants in a particular year national mathematics exam test which is presented in two versions of the scoring form, namely dichotomy and polytomy. In the dichotomous scoring version, each item is scored individually. Alternatively, each item that is scored separately on the dichotomous version, is aggregated into the same indicator, and then summed to obtain the polytomy version. Estimation of item parameters and students' abilities in the dichotomous scoring version was carried out using the 3 PL (Logistics Parameter) model using MLE estimation and the polytomy scoring version using the GPCM (Generalized Partial Credit Model) using EAP. Both were analyzed using PARSCALE software. Comparative analysis of the two scoring versions by looking at the average results of the estimated difficulty level, graph analysis, calculating correlations, and the results of the value of the information function. The results of the analysis show that the average difficulty level of the dichotomous version is 0.166, with a standard deviation of 1.137, while the average difficulty level of the polytomy version is 0.033, with a standard deviation of 0.940. The value of the dichotomous scoring version of the information function is higher than the polytomy scoring version. These results indicate that the math exam test with the dichotomous scoring version is better than the polytomy scoring version. graph analysis, calculate correlations, and result in value information functions. The results of the analysis show that the average difficulty level of the dichotomous version is 0.166, with a standard deviation of 1.137, while the average difficulty level of the polytomy version is 0.033, with a standard deviation of 0.940. The value of the dichotomous scoring version of the information function is higher than the polytomy scoring version

Performance of Polytomous IRT Models With Rating Scale Data: An Investigation Over Sample Size, Instrument Length, and Missing Data

The implementation of polytomous item response theory (IRT) models such as the graded response model (GRM) and the generalized partial credit model (GPCM) to inform instrument design and validation has been increasing across social and educational contexts where rating scales are usually used. The performance of such models has not been fully investigated and compared across conditions with common survey-specific characteristics such as short test length, small sample size, and data missingness. The purpose of the current simulation study is to inform the literature and guide the implementation of GRM and GPCM under these conditions. For item parameter estimations, results suggest a sample size of at least 300 and/or an instrument length of at least five items for both models. The performance of GPCM is stable across instrument lengths while that of GRM improves notably as the instrument length increases. For person parameters, GRM reveals more accurate estimates when the proportion of missing data is small, whereas GPCM is favored in the presence of a large amount of missingness. Further, it is not recommended to compare GRM and GPCM based on test information. Relative model fit indices (AIC, BIC, LL) might not be powerful when the sample size is less than 300 and the length is less than 5. Synthesis of the patterns of the results, as well as recommendations for the implementation of polytomous IRT models, are presented and discussed.

Polytomous Testlet Response Models for Technology-Enhanced Innovative Items: Implications on Model Fit and Trait Inference.

The development of technology-enhanced innovative items calls for practical models that can describe polytomous testlet items. In this study, we evaluate four measurement models that can characterize polytomous items administered in testlets: (a) generalized partial credit model (GPCM), (b) testlet-as-a-polytomous-item model (TPIM), (c) random-effect testlet model (RTM), and (d) fixed-effect testlet model (FTM). Using data from GPCM, FTM, and RTM, we examine performance of the scoring models in multiple aspects: relative model fit, absolute item fit, significance of testlet effects, parameter recovery, and classification accuracy. The empirical analysis suggests that relative performance of the models varies substantially depending on the testlet-effect type, effect size, and trait estimator. When testlets had no or fixed effects, GPCM and FTM led to most desirable measurement outcomes. When testlets had random interaction effects, RTM demonstrated best model fit and yet showed substantially different performance in the trait recovery depending on the estimator. In particular, the advantage of RTM as a scoring model was discernable only when there existed strong random effects and the trait levels were estimated with Bayes priors. In other settings, the simpler models (i.e., GPCM, FTM) performed better or comparably. The study also revealed that polytomous scoring of testlet items has limited prospect as a functional scoring method. Based on the outcomes of the empirical evaluation, we provide practical guidelines for choosing a measurement model for polytomous innovative items that are administered in testlets.

Calibration of Polytomous Response Mathematics Achievement Test Using Generalized Partial Credit Model

This study assessed the empirical comparability of item calibration in the developed essay (DEVessay-MAT) and (NECOessay-MAT) mathematics achievement test under the Generalized Partial Credit Model (GPCM). The instrumentation research approach of counterbalance design was employed. The sample consisted of 1080 senior secondary school students (SSS3) of 36 schools, who were drawn randomly from the Osun East senatorial district of Osun State, Nigeria. Two instruments were used and data obtained were subjected to Parallel Analysis (PA), Generalized Partial Credit Model (GPCM) and Independent sample t-test. Results showed that the test does not violate unidimensionality with the first Eigenvalue (2.05) from the experimental data was greater than the first random Eigenvalue (1.17) from PA, while other Eigenvalues from the experimental data were less than the rest of Eigenvalues under PA. Also, there existed a significant difference between the step difficulties/overall item difficulty and discrimination/slope index of the two instruments with (t = 3.52, df = 8, p < 0.05) and (t = 3.26, df = 8, p < 0.05) respectively. The authors’ concluded that the developed essay test produced better item statistics estimates compared to NECO-MAT (essay) test. Consequently, it was recommended that public examining bodies in sub-Sahara Africa should embrace an apt polytomous model for the calibration of their test items.