Testlet Model Research Articles

What do music listening tests assess? Applying the Rasch testlet models to the HKDSE data

This study examines the adequacy of a general proficiency in music listening for describing how examinees responded to exam questions on various music genres based on an empirical investigation of a public music listening examination. In the Hong Kong Diploma of Secondary Education Examination (HKDSE), Music consists of three mandatory components: listening, performing, and creating. Notably, the listening component is evaluated through a written examination, allowing for objective scoring. A candidate’s performance on this exam has been summarized as a single score, a method that is effective for evaluation purposes, while the underlying conceptualization may oversimplify the depth of knowledge required for assessment. Consequently, we conducted a reevaluation of the HKDSE Music listening test, employing the Rasch testlet model to explore potential multidimensional structures. Our findings highlighted distinctive attributes in questions related to Cantonese opera, setting them apart from others. We discuss the implications of these findings for music assessment.

Asymptotically Correct Person Fit z-Statistics For the Rasch Testlet Model.

A well-known person fit statistic in the item response theory (IRT) literature is the statistic (Drasgow et al. in Br J Math Stat Psychol 38(1):67-86, 1985). Snijders (Psychometrika 66(3):331-342, 2001) derived , which is the asymptotically correct version of when the ability parameter is estimated. However, both statistics and other extensions later developed concern either only the unidimensional IRT models or multidimensional models that require a joint estimate of latent traits across all the dimensions. Considering a marginalized maximum likelihood ability estimator, this paper proposes and , which are extensions of and , respectively, for the Rasch testlet model. The computation of relies on several extensions of the Lord-Wingersky algorithm (1984) that are additional contributions of this paper. Simulation results show that has close-to-nominal Type I error rates and satisfactory power for detecting aberrant responses. For unidimensional models, and reduce to and , respectively, and therefore allows for the evaluation of person fit with a wider range of IRT models. A real data application is presented to show the utility of the proposed statistics for a test with an underlying structure that consists of both the traditional unidimensional component and the Rasch testlet component.

Cognitive Diagnosis Testlet Model for Multiple-Choice Items

The testlet design is very popular in educational and psychological assessments. This article proposes a new cognitive diagnosis model, the multiple-choice cognitive diagnostic testlet (MC-CDT) model for tests using testlets consisting of MC items. The MC-CDT model uses the original examinees’ responses to MC items instead of dichotomously scored data (i.e., correct or incorrect) to retain information of different distractors and thus enhance the MC items’ diagnostic power. The Markov chain Monte Carlo algorithm was adopted to calibrate the model using the WinBUGS software. Then, a thorough simulation study was conducted to evaluate the estimation accuracy for both item and examinee parameters in the MC-CDT model under various conditions. The results showed that the proposed MC-CDT model outperformed the traditional MC cognitive diagnostic model. Specifically, the MC-CDT model fits the testlet data better than the traditional model, while also fitting the data without testlets well. The findings of this empirical study show that the MC-CDT model fits real data better than the traditional model and that it can also provide testlet information.

A Testlet Diagnostic Classification Model with Attribute Hierarchies.

In this article, a testlet hierarchical diagnostic classification model (TH-DCM) was introduced to take both attribute hierarchies and item bundles into account. The expectation-maximization algorithm with an analytic dimension reduction technique was used for parameter estimation. A simulation study was conducted to assess the parameter recovery of the proposed model under varied conditions, and to compare TH-DCM with testlet higher-order CDM (THO-DCM; Hansen, M. (2013). Hierarchical item response models for cognitive diagnosis (Unpublished doctoral dissertation). UCLA; Zhan, P., Li, X., Wang, W.-C., Bian, Y., & Wang, L. (2015). The multidimensional testlet-effect cognitive diagnostic models. Acta Psychologica Sinica, 47(5), 689. https://doi.org/10.3724/SP.J.1041.2015.00689). Results showed that (1) ignoring large testlet effects worsened parameter recovery, (2) DCMs assuming equal testlet effects within each testlet performed as well as the testlet model assuming unequal testlet effects under most conditions, (3) misspecifications in joint attribute distribution had an differential impact on parameter recovery, and (4) THO-DCM seems to be a robust alternative to TH-DCM under some hierarchical structures. A set of real data was also analyzed for illustration.

Measuring distress in older population: Rasch analysis of the Kessler Psychological Distress Scale

ObjectivesThe 10-item Kessler Psychological Distress Scale (K−10) is a widely applied distress measure; however, its psychometric properties were not established with older populations using advanced methodology. The aim of this study was to examine psychometric properties of the K-10 through application of Rasch methodology and if possible, develop an ordinal-to-interval conversion to improve its reliability in older populations. MethodThe Partial Credit Rasch Model was applied to analyse K-10 scores of the sample including 490 participants (56.3 % females) aged 70 to 90 years and without dementia from the Sydney Memory and Ageing Study (MAS). ResultsThe initial analysis of the K-10 showed poor reliability and significant deviation from the expectations of the Rasch model. The best model fit was evident after correcting disordered thresholds and creating two testlet models to address local dependency between items (χ2(35) = 29.87, p = 0.71). The modified K-10 demonstrated strict unidimensionality, enhanced reliability and scale invariance across personal factors, such as sex, age, and education and permitted development of ordinal-to-interval transformation algorithms. LimitationsOrdinal-to-interval conversion can only be applied for older adults with complete data. ConclusionsThe K-10 satisfied principles of fundamental measurement defined by Rasch model after minor modifications. Clinicians and researchers can transform K-10 raw scores into interval-level data using converging algorithms published here without altering the original scale response format, which increases reliability of the K-10.

The Development of Testlet Assessment Instrument Model Integrated with E-ujian Website to Measure the Higher-Order Thinking Skills

This research and development research aimed to determine the feasibility and characteristics of the Testlet model assessment instrument integrated with the E-ujian website in measuring the Higher-order Thinking Skills (HOTS) of junior high school students. The researchers employed the 4D development model (define, design, develop and disseminate). However, the researchers conduct the development up to the develop stage. The research was conducted at a junior high school with 42 students as respondents. The researchers collected the data through test techniques, questionnaires, interviews, observation, and documentation. This development research utilized the descriptive-quantitative data analysis method through characteristic tests, teacher and students’ questionnaire responses analysis, and analysis of students' HOTS profiles. The students’ HOTS profiles based on the assessment instrument were divided into 53 percent of students in the poor HOTS category, 31 percent of students in the low HOTS category, and 16 percent of students in the high HOTS category. Concluded that the assessment instrument developed was very feasible, with a percentage of 94 percent. The assessment instrument developed had the characteristics of valid, reliable, discriminatory, disproportionate, relevant, representative, impractical, and specific. The Testlet model assessment instrument development can be used as an alternative for the distance learning assessment during the Covid-19 pandemic.

Scale Linking for the Testlet Item Response Theory Model.

In their 2005 paper, Li and her colleagues proposed a test response function (TRF) linking method for a two-parameter testlet model and used a genetic algorithm to find minimization solutions for the linking coefficients. In the present paper the linking task for a three-parameter testlet model is formulated from the perspective of bi-factor modeling, and three linking methods for the model are presented: the TRF, mean/least squares (MLS), and item response function (IRF) methods. Simulations are conducted to compare the TRF method using a genetic algorithm with the TRF and IRF methods using a quasi-Newton algorithm and the MLS method. The results indicate that the IRF, MLS, and TRF methods perform very well, well, and poorly, respectively, in estimating the linking coefficients associated with testlet effects, that the use of genetic algorithms offers little improvement to the TRF method, and that the minimization function for the TRF method is not as well-structured as that for the IRF method.

Comparison of Testlet Effect on Parameter Estimates Using Different Item Response Theory Models

In this study, the testlet effect was calculated for each testlet in the PISA 2018 reading literacy test, and it was examined whether this effect caused a difference in item and ability parameters. The data set was analyzed with a two-parameter logistic item response theory model and a two-parameter logistic testlet model. The results show that variances of testlet effects range from .100 to .432. When the item and ability parameter estimation results of the models were compared, it was determined that the item and ability parameters estimated from the two approaches were highly correlated with each other. It can be said that the item slope and item intercept parameters estimated from different models remained unaffected. However, when the local dependency assumption is not met, it was observed that the standard error values of the two-parameter model for the ability parameter were underestimated. The implications for the analysis and evaluation of the tests based on testlet are discussed. In conclusion, in this study, it was concluded that the testlet effect caused a difference in parameter estimates, but the local dependence among the items was negligible because of the small testlet effects.

Computerized adaptive testing for testlet-based innovative items.

Increasing use of innovative items in operational assessments has shedded new light on the polytomous testlet models. In this study, we examine performance of several scoring models when polytomous items exhibit random testlet effects. Four models are considered for investigation: the partial credit model (PCM), testlet-as-a-polytomous-item model (TPIM), random-effect testlet model (RTM), and fixed-effect testlet model (FTM). The performance of the models was evaluated in two adaptive testings where testlets have nonzero random effects. The outcomes of the study suggest that, despite the manifest random testlet effects, PCM, FTM, and RTM perform comparably in trait recovery and examinee classification. The overall accuracy of PCM and FTM in trait inference was comparable to that of RTM. TPIM consistently underestimated population variance and led to significant overestimation of measurement precision, showing limited utility for operational use. The results of the study provide practical implications for using the polytomous testlet scoring models.

Polytomous Testlet Response Models for Technology-Enhanced Innovative Items: Implications on Model Fit and Trait Inference.

The development of technology-enhanced innovative items calls for practical models that can describe polytomous testlet items. In this study, we evaluate four measurement models that can characterize polytomous items administered in testlets: (a) generalized partial credit model (GPCM), (b) testlet-as-a-polytomous-item model (TPIM), (c) random-effect testlet model (RTM), and (d) fixed-effect testlet model (FTM). Using data from GPCM, FTM, and RTM, we examine performance of the scoring models in multiple aspects: relative model fit, absolute item fit, significance of testlet effects, parameter recovery, and classification accuracy. The empirical analysis suggests that relative performance of the models varies substantially depending on the testlet-effect type, effect size, and trait estimator. When testlets had no or fixed effects, GPCM and FTM led to most desirable measurement outcomes. When testlets had random interaction effects, RTM demonstrated best model fit and yet showed substantially different performance in the trait recovery depending on the estimator. In particular, the advantage of RTM as a scoring model was discernable only when there existed strong random effects and the trait levels were estimated with Bayes priors. In other settings, the simpler models (i.e., GPCM, FTM) performed better or comparably. The study also revealed that polytomous scoring of testlet items has limited prospect as a functional scoring method. Based on the outcomes of the empirical evaluation, we provide practical guidelines for choosing a measurement model for polytomous innovative items that are administered in testlets.

Pengukuran Prestasi Belajar Matematika Siswa dengan Menggunakan Tes Model Testlet ditinjau dari Status Sekolah dan Gender

This study aims to determine the differences in students' mathematics learning achievement as measured using the testlet model test in terms of school status and gender in class VII SMP/MTs students. This study uses a quantitative approach in the form of ex post facto research. This study took samples from 2 public schools and two private schools with a total sample of 109 students. The sampling technique used is purposive sampling. Data collection techniques using test techniques and instruments using a testlet model test based on local wisdom. The instrument's validity uses content validity with the Aiken formula, while the reliability, level of difficulty of distinguishing power, and the effectiveness of distractors are analyzed by the ITEMAN 3.0 application. The data analysis technique used was two-way Analysis of Variance (ANOVA). Based on the results of the data analysis conducted, it can be seen that there is no difference in students' mathematics learning achievement as measured by using the testlet model test when viewed from school status or in terms of gender. The analysis results also show that school status and gender are not related to improving students' mathematics learning achievement when measured using the testlet model test.

Rasch testlet model and bifactor analysis: how do they assess the dimensionality of large-scale Iranian EFL reading comprehension tests?

Rasch testlet and bifactor models are two measurement models that could deal with local item dependency (LID) in assessing the dimensionality of reading comprehension testlets. This study aimed to apply the measurement models to real item response data of the Iranian EFL reading comprehension tests and compare the validity of the bifactor models and corresponding item parameters with unidimensional and multidimensional Rasch models. The data collected from the EFL reading comprehension section of the Iranian national university entrance examinations from 2016 to 2018. Various advanced packages of the R system were employed to fit the Rasch unidimensional, multidimensional, and testlet models and the exploratory and confirmatory bifactor models. Then, item parameters estimated and testlet effects identified; moreover, goodness of fit indices and the item parameter correlations for the different models were calculated. Results showed that the testlet effects were all small but non-negligible for all of the EFL reading testlets. Moreover, bifactor models were superior in terms of goodness of fit, whereas exploratory bifactor model better explained the factor structure of the EFL reading comprehension tests. However, item difficulty parameters in the Rasch models were more consistent than the bifactor models. This study had substantial implications for methods of dealing with LID and dimensionality in assessing reading comprehension with reference to the EFL testing.

Multicomponent framework for students’ subjective well-being in elementary school.

The study proposes a multicomponent model of subjective well-being (SWB) in elementary school. The model includes satisfaction with school, affect toward school, well-being related to communication with peers, and subjective physical well-being. The aim of this study is to verify whether well-being related to different aspects of school life can be explained by a general SWB factor in school. A sample of 1,704 third-grade students (M age = 8.89; SD = 0.4; 48.4% girls) was asked to complete a survey consisting of Likert-type scales. The psychometric analysis investigated the internal structure of the data. Multidimensional models from the Rasch modeling framework of item response theory were applied. A correlated factors model, a bifactor model, and oblique bifactor models were compared. Two models fit the data well: the extended Rasch testlet model and the correlated factor model. The findings indicate that a general SWB factor can be extracted from the data through the extended Rasch testlet model. However, the extended Rasch testlet model provides only 1 usable general factor. For the extraction of detailed information by means of scales, the correlated factor model can be applied. Implications for practitioners and limitations of the study are also discussed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

The Development of Diagnostic Test with Testlet Model to Detect Difficult Learning Students in Temperature and Its Changes Materials

The science learning process in junior high school is expected to lead to the achievement of learning objectives. Learners can experience difficulties in the learning process that must be detected to maximize learning outcomes. This Research & Development study aims to determine the characteristics of the testlet diagnostic model and the profile of students' learning difficulties on the material temperature and its changes. The research design using the 4D model reduced to 3D includes: define, design, and develop. The study was conducted at SMPN 1 Semarang with a total of 39 students. Data collection techniques through test techniques, questionnaires, interviews, and documentation. Data analysis techniques using descriptive-quantitative methods through the test of characteristics (validity, reliability, level of difficulty, distinguishing power), questionnaire analysis of teacher and student responses, and analysis of learning difficulties profiles. The results showed that the diagnostic test of the testlet model can be used to detect learners’ learning difficulties in the temperature and its changes material. The developed testlet test model has valid, reliable, relevant, representative, practical, discriminative, specific, and proportional characteristics. Profile of learning difficulties in the material temperature and its changes, namely, verbal ability, the highest and lowest of 64% and 31% in Competency Achievement Indicators 6 and 3; mastery of concepts, the highest and lowest of 68% and 31% in Competency Achievement Indicators 8 and 5; and the ability to count, the highest and lowest of 75% and 9% in Competency Achievement Indicators 5 and 1.

The Bayesian Covariance Structure Model for Testlets

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items. The inclusion of testlet parameters significantly increases the number of model parameters, which can lead to computational problems. To avoid these problems, a Bayesian covariance structure model (BCSM) for testlets is proposed, where standard IRT models are extended with a covariance structure model to account for dependences among testlet items. In the BCSM, the dependence among testlet items is modeled without using testlet effects. This approach does not imply any sample size restrictions and is very efficient in terms of the number of parameters needed to describe testlet dependences. The BCSM is compared to the well-known Bayesian random effects model for testlets using a simulation study. Specifically for testlets with a few items, a small number of test takers, or weak associations among testlet items, the BCSM shows more accurate estimation results than the random effects model.

Learning achievement of Elementary School student of mathematics using the Testlet model instrument: A comparison between the 2006 Curriculum and the 2013 Curriculum

This study aims to describe and compare elementary school students’ achievement in mathematics using the 2006 curriculum and 2013 curriculum with a testlet model. The research used a survey method and a quantitative approach. The participants were 552 from 14 elementary schools consisting of 273 students from elementary schools applying the KTSP curriculum and 279 from elementary schools applying the 2013 curriculum. The instruments used a testlet model test instruments that had met valid and reliable criteria. The data were analyzed using independent sample t-test. The results of the analysis show that there were not significant differences in the two curricula with t value with equal to 1.246 on the significance level of 0.118. The data obtained using a testlet model were more detailed related to students’ difficulties, students’ ability to answer questions and the level of guessing from students. The results can be used as a basic in enhancing student mathematics learning.

A Bayesian Random Block Item Response Theory Model for Forced-Choice Formats.

Based on the framework of testlet models, the current study suggests the Bayesian random block item response theory (BRB IRT) model to fit forced-choice formats where an item block is composed of three or more items. To account for local dependence among items within a block, the BRB IRT model incorporated a random block effect into the response function and used a Markov Chain Monte Carlo procedure for simultaneous estimation of item and trait parameters. The simulation results demonstrated that the BRB IRT model performed well for the estimation of item and trait parameters and for screening those with relatively low scores on target traits. As found in the literature, the composition of item blocks was crucial for model performance; negatively keyed items were required for item blocks. The empirical application showed the performance of the BRB IRT model was equivalent to that of the Thurstonian IRT model. The potential advantage of the BRB IRT model as a base for more complex measurement models was also demonstrated by incorporating gender as a covariate into the BRB IRT model to explain response probabilities. Recommendations for the adoption of forced-choice formats were provided along with the discussion about using negatively keyed items.

THE DEVELOPMENT OF ASSESSMENT INSTRUMENT USING TESLET MODELS IN STEREOCHEMISTRY MATERIALS

Stereochemistry is the study of molecules in three-dimensional space regarding how atoms in a molecule are arranged in one room relative to the other. Stereochemistry is a difficult course. Therefore, it is necessary to diagnose the difficulty of learning using the testlet model. This study is aimed at developing an appropriate testlet model for the assessment instrument. This research uses the development model of Borg and Gall's assessment instrument with ten stages, but this research is limited to the fifth stage only, namely 1) Research and Information Collection, 2) Planning 3) Develop Preliminary Form of Product 4) Preliminary Field Testing, and 5) Preliminary Product Revision. The data collection techniques were in the form of validation questionnaires from expert teams and students' response questionnaires. The data analysis technique used qualitative and quantitative data. The qualitative data were from the results of a review of validation by a team of experts who are Postgraduate Lecturers of Chemistry Education at Universitas Riau and lecturers of the Chemistry Education Study Program at the Faculty of Tarbiyah and Teaching at UIN Suska Riau. The quantitative data were obtained from the calculation of the mean scores of the validation questionnaire by two lecturers of the Postgraduate of Chemistry Education at Universitas Riau and one lecturer in the Chemistry Education Study Program at the Tarbiyah Faculty and Teacher Training at UIN Suska Riau as the experts. The results of the study show that the mean score of validation from the expert team is 3.97 on the aspect of material with valid category, 3.97 on the aspect of construction with valid category, 3.88 on the aspect of language with valid category; and the results of the students' response questionnaires are 51.58 on the time readability questionnaire and 8.08 on the aspect of time adequacy.

Simultaneously Modeling Differential Testlet Functioning and Differential Item Functioning: Addressing Variance Heterogeneity with a Multigroup One-Parameter Testlet Model

ABSTRACTCurrent methods that simultaneously model differential testlet functioning (DTLF) and differential item functioning (DIF) constrain the variances of latent ability and testlet effects to be equal between the focal and the reference groups. Such a constraint can be stringent and unrealistic with real data. In this study, we propose a multigroup one-parameter testlet model (M1PLTM) that relaxes this equal variance assumption and also allows simultaneous modeling of DTLF and DIF. M1PLTM is a more generalized version of the extant models. The results of a simulation study show that parameters of M1PLTM could be accurately recovered via a Markov chain Monte Carlo (MCMC) method; when data were generated based on M1PLTM, falsely assuming equal variances of latent ability and testlet effects resulted in inferior parameter recovery.

Investigating the Psychometric Impact of Negative Worded Items in Reading Comprehension Passages with a 3PL Cross-Classified Testlet Model

Negative worded (NW) items used in psychological instruments have been studied with the bifactor model to investigate whether the NW items form a secondary factor due to negative wording orthogonal to the measured latent construct, a validation procedure which checks whether NW items form a source of construct irrelevant variance (CIV) and hence constitute a validity threat. In the context of educational testing, however, no such validation attempts have been made. In this study, we studied the psychometric impact of NW items in an English proficiency reading comprehension test using a modeling approach similar to the bifactor model, namely the three-parameter logistic cross-classified testlet response theory (3PL CCTRT) model, to account for both guessing and possible local item dependence due to passage effect in the data set. The findings indicate that modeling the NW items with a separate factor leads to noticeable improvement in model fit, and the factor variance is marginal but nonzero. However, item and ability parameter estimates are highly similar between the 3PL CCTRT model and other models that do not model the NW items. It is concluded that the NW items introduce CIV into the data, but its magnitude is too small to change item and person ability parameter estimates to an extent of practical significance.