3PL Model Research Articles

Analysis of model fit and item parameter of work and energy test using item response theory

One of the important parts in assessing learning outcomes is using a good instrument that is analyzed using an appropriate analytical model and can measure students' abilities accurately. This study aims to determine the model fit and item parameter of work and energy test using item response theory. This research is a quantitative research that was carried out on the responses of 1177 high school students spread across Banten Province. The instrument is a set of work and energy tests consisting of 25 multiple choices. The data analysis used the item response theory approach with statistical methods ranging from determining the fit model to the item characteristics.. The analysis showed the students' responses to the fit energy and effort test with 9 items of 1 PL model, 17 items of 2 PL model, and 16 items of 3 PL model. Based on the percentage, the 2PL model is suitable than 1PL and 3PL. Further analysis determines the item parameter value by referring to the 2PL model, namely the item parameter difficulty level (b) and discrimination (a). The result shows that all items have difficulty in the range of 2.501 to 1.595, and the discrimination was in the range of 0.289 to 1.109. Based on this analysis, it can be concluded that all items in this test are the good item criteria

PSYCHOMETRIC EVALUATION OF FLOOD DISASTER MANAGEMENT QUESTIONNAIRE: CONFIRMATORY FACTOR ANALYSIS AND ITEM RESPONSE THEORY ANALYSIS

Flood disaster is the commonest natural disaster with huge impact on healthcare services in Malaysia. The FloodDMQ-BM© questionnaire was developed as a tool to assess the knowledge, attitude, and practice of healthcare providers regarding patient management during a flood disaster. We aim to further validate the FloodDMQ-BM© questionnaire by using Confirmatory Factor Analysis (CFA) and Item Response Theory (IRT).This cross-sectional study involved doctors, nurses and paramedics working in the Emergency Department of Hospital Universiti Sains Malaysia, Hospital Raja Perempuan Zainab II and Hospital Kuala Krai. Respondents were required to complete the FloodDMQ-BM© questionnaire. The responses were analysed by using CFA and IRT to establish its validity and reliability. A total of 215 respondents participated in this study. CFA analysis with Maximum Likelihood Robust as the estimation method, on the attitude and practice components resulted in good factor loadings (>0.5) in nearly all items and excellent model fit indices values (CFI = 0.96-0.98, TLI = 0.95-0.96, SRMR = 0.04-0.05, RMSEA = 0.07). Meanwhile, IRT analysis on the knowledge section showed a good two-way marginal fit based on S-X2, and a good model fit with RMSEA of 0.08. Based on the 2PL model by using the IRT assessment of the knowledge section, one item in the knowledge section (K3) was removed (chi-squared residual >4) resulting in improved model fit. The included items had well-standardized loadings (>0.3) and marginal reliability of 0. 651.Our results confirmed that the FloodDMQ-BM© questionnaire displayed valid and reliable psychometric properties.

Item-Weighted Likelihood Method for Measuring Growth in Longitudinal Study With Tests Composed of Both Dichotomous and Polytomous Items.

In this paper, a new item-weighted scheme is proposed to assess examinees’ growth in longitudinal analysis. A multidimensional Rasch model for measuring learning and change (MRMLC) and its polytomous extension is used to fit the longitudinal item response data. In fact, the new item-weighted likelihood estimation method is not only suitable for complex longitudinal IRT models, but also it can be used to estimate the unidimensional IRT models. For example, the combination of the two-parameter logistic (2PL) model and the partial credit model (PCM, Masters, 1982) with a varying number of categories. Two simulation studies are carried out to further illustrate the advantages of the item-weighted likelihood estimation method compared to the traditional Maximum a Posteriori (MAP) estimation method, Maximum likelihood estimation method (MLE), Warm’s (1989) weighted likelihood estimation (WLE) method, and type-weighted maximum likelihood estimation (TWLE) method. Simulation results indicate that the improved item-weighted likelihood estimation method better recover examinees’ true ability level for both complex longitudinal IRT models and unidimensional IRT models compared to the existing likelihood estimation (MLE, WLE and TWLE) methods and MAP estimation method, with smaller bias, root-mean-square errors, and root-mean-square difference especially at the low-and high-ability levels.

DIF DETECTION SENSITIVITY OF LORD’S CHI-SQUARE, RAJU’S AREA, LOGISTIC REGRESSION, MANTEL-HAENSZEL, STANDARDIZATION, AND TRANSFORMED ITEM DIFFICULTIES METHODS, IN COMPARISON, USING R.

Differential item functioning (DIF) will always occur as a result of these differences in the person parameter of these individuals being examined even when item parameters remain constant during testing. This postulate of item response theory (IRT) was proven in this work. This study investigated if DIF detection methods will have the same DIF detection sensitivity. Comparative research design formed the framework of the study. Transformed item difficulties (TID), Mantel-Haenszel (MH), standardization, logistic regression, Ragu’s area, and Lord’s chi-square methods were compared. The study used 400 vocational one students (200 male as reference group and 200 female as focal group) in Rivers state, Nigeria. The multiple choice items of 2019 computer science for the junior school certificate examination (JSCE) was adapted as the instrument for data collection, which were administered to students and scored dichotomously. Difficulty and discrimination parameters of the items were analyzed using the 2PL model of IRT with the help of ltm package. Ogives of the items were plotted with ggplot2 package. Individual DIF methods and DichoDif in DifR were used to detect DIF and compare the methods. The results revealed that all the items of the test functioned differently between the reference group and the focal group as shown in the item characteristic curves (ICCs). In comparison of the DIF detection methods, standardization method detected most of the DIF items followed by logistic regression method, and then lord’s chi-square methods. Transformed item difficulties method detected more than mantel-Haenszel method. Raju’s area method could not detect any. In the light of the finding, it was recommended that the best DIF detection methods (possibly combination of them) should be used to identify DIF items in tests. KEYWORDS: Item response theory, differential item functioning, item characteristic curve, item parameters.

Applying Item Response Theory model for evaluating item and test properties of academic potential test for students with disability

Universitas Brawijaya (UB) is one of the pioneers of inclusive education in higher education in Indonesia. One of the innovations in the policies related to inclusive education is affirmative action admissions special for students with disabilities, namely Seleksi Mandiri Penyandang Disabilitas (Independent Selection for Person with Disabilities), which focuses on accommodating admissions selection for students with disabilities who want to enroll in bachelors or vocational programs. A part of this admission selection is the test called the Computer-Based Academic Potential Test. This study aims to evaluate, from a psychometric perspective, the psychometric properties of the potential academic test. The approach used in this study is the item response theory (IRT) framework, which is mostly used for evaluating psychometric quality at both item-level and test levels. This study's IRT model is a two-parameter logistic model that includes difficulty parameter and discrimination parameter. The result of this study exhibited that the three subtests of the Computer-Based Academic Potential Test, in general, have satisfying results from the 2PL model estimation. The result also showed that most of the item difficulties ranged from medium to very difficult.

Analysis of model fit logistic parameter of work and energy material using unidimensional item response theory

This study aims to determine the suitability of the student response analysis on the test of work and energy materials with the logistic parameter model of item response theory. Another aim is to estimate item parameters according to a fit parameter logistic model (PL). This research is a quantitative research that was carried out on the responses of 1177 high school students spread across Banten Province. The instrument is a set of work and energy test consisting of 25 multiple choices. The analysis was carried out using the statistical method. The analysis showed the students' responses to the fit energy and effort test with 9 items of 1 PL model, 17 items of 2 PL model, and 16 items of 3 PL model. Based on the percentage, the 2PL model is suitable than 1PL and 3PL. So it can be concluded that the analysis of the work and energy test instrument fits the 2PL parameter model. Further analysis is the determination of the item parameter value by referring to the 2PL model, namely the item parameter difficulty level (b) and discrimination (a). From the analysis conducted, it was found that the difficulty level was in the range - 2.501 to 1.595 and the discrimination was in the range 0.289 to 1.109. Based on this analysis, it can be concluded that all items in this test are the good item criteria

ANALISIS ESTIMASI KEMAMPUAN SISWA DENGAN PENDEKATAN ITEM RESPONSE THEORY PENSKORAN DIKOTOMUS DAN POLITOMUS

This study compares students' abilities using the Item Response Theory (IRT) approach to dichotomous and polytomous scoring. This research is quantitative descriptive. The research subjects were 1175 high school students in class XI spread across West Java and Banten provinces, consisting of 450 male students and 725 female students. Response data with dichotomous scoring were analyzed using the item response theory approach with the BILOG-MG program, while the polytomous scoring was analyzed using the GPCM approach using the R program. The results of the model fit test showed that the items most fit with the 2PL model. The instrument used is a work and energy material test instrument tested for validity d, reliability, distinguishing power, and difficulty level. The results showed that the average ability with a polytomous score was greater than that of a dichotomous, even though compared to having almost the same ability but with a relatively different distribution. The distribution of students' abilities with polytomous scoring is closer to the normal curve than the dichotomous scoring. The relationship between students' ability scores with these two scoring approaches is shown by a correlation coefficient score of 0.990 and a determination index of 0.9808 with a prediction line y= 0.9735 x +0.0036.

Investigating the Impact of Noneffortful Responses on Individual-Level Scores: Can the Effort-Moderated IRT Model Serve as a Solution?

Suboptimal effort is a major threat to valid score-based inferences. While the effects of such behavior have been frequently examined in the context of mean group comparisons, minimal research has considered its effects on individual score use (e.g., identifying students for remediation). Focusing on the latter context, this study addressed two related questions via simulation and applied analyses. First, we investigated how much including noneffortful responses in scoring using a three-parameter logistic (3PL) model affects person parameter recovery and classification accuracy for noneffortful responders. Second, we explored whether improvements in these individual-level inferences were observed when employing the Effort Moderated IRT (EM-IRT) model under conditions in which its assumptions were met and violated. Results demonstrated that including 10% noneffortful responses in scoring led to average bias in ability estimates and misclassification rates by as much as 0.15 SDs and 7%, respectively. These results were mitigated when employing the EM-IRT model, particularly when model assumptions were met. However, once model assumptions were violated, the EM-IRT model's performance deteriorated, though still outperforming the 3PL model. Thus, findings from this study show that (a) including noneffortful responses when using individual scores can lead to potential unfounded inferences and potential score misuse, and (b) the negative impact that noneffortful responding has on person ability estimates and classification accuracy can be mitigated by employing the EM-IRT model, particularly when its assumptions are met.

An Investigation of Item Position Effects by Means of IRT-Based Differential Item Functioning Methods

In this study, whether item position effects lead to DIF in the condition where different test booklets are used was investigated. To do this the methods of Lord’s chi-square and Raju’s unsigned area with the 3PL model under with and without item purification were used. When the performance of the methods was compared, it was revealed that generally, the method of Lord’s chi-square identified more items with DIF than did the method of Raju’s unsigned area. The differentiation of the booklets with respect to item position resulted in a higher number of items displaying DIF with item purification conditions. Based on the findings of the present study, to avoid the occurrence of DIF due to item position effects, it is recommended to position the same items across different booklets in similar locations when forming different booklets.

A Guide for More Accurate and Precise Estimations in Simulative Unidimensional IRT Models

There is a great deal of research about item response theory (IRT) conducted by simulations. Item and ability parameters are estimated with varying numbers of replications under different test conditions. However, it is not clear what the appropriate number of replications should be. The aim of the current study is to develop guidelines for the adequate number of replications in conducting Monte Carlo simulation studies involving unidimensional IRT models. For this aim, 192 simulation conditions which included four sample sizes, two test lengths, eight replication numbers, and unidimensional IRT models were generated. Accuracy and precision of item and ability parameter estimations and model fit values were evaluated by considering the number of replications. In this context, for the item and ability parameters; mean error, root mean square error, standard error of estimates, and for model fit; M_2, 〖RMSEA〗_2, and Type I error rates were considered. The number of replications did not seem to influence the model fit, it was decisive in Type I error inflation and error prediction accuracy for all IRT models. It was concluded that to get more accurate results, the number of replications should be at least 625 in terms of accuracy of the Type I error rate estimation for all IRT models. Also, 156 replications and above can be recommended. Item parameter biases were examined, and the largest bias values were obtained from the 3PL model. It can be concluded that the increase in the number of parameters estimated by the model resulted in more biased estimates.

Flexibility at the Price of Volatility: Concurrent Calibration in Multistage Tests in Practice Using a 2PL Model

Multistage test (MST) designs promise efficient student ability estimates, an indispensable asset for individual diagnostics in high-stakes educational assessments. In high-stakes testing, annually changing test forms are required because publicly known test items impair accurate student ability estimation, and items of bad model fit must be continually replaced to guarantee test quality. This requires a large and continually refreshed item pool as the basis for high-stakes MST. In practice, the calibration of newly developed items to feed annually changing tests is highly resource intensive. Piloting based on a representative sample of students is often not feasible, given that, for schools, participation in actual high-stakes assessments already requires considerable organizational effort. Hence, under practical constraints, the calibration of newly developed items may take place on the go in the form of a concurrent calibration in MST designs. Based on a simulation approach this paper focuses on the performance of Rasch vs. 2PL modeling in retrieving item parameters when items are for practical reasons non-optimally placed in multistage tests. Overall, the results suggest that the 2PL model performs worse in retrieving item parameters compared to the Rasch model when there is non-optimal item assembly in the MST; especially in retrieving parameters at the margins. The higher flexibility of 2PL modeling, where item discrimination is allowed to vary, seems to come at the cost of increased volatility in parameter estimation. Although the overall bias may be modest, single items can be affected by severe biases when using a 2PL model for item calibration in the context of non-optimal item placement.

Assessing the Accuracy of Parameter Estimates in the Presence of Rapid Guessing Misclassifications.

The presence of rapid guessing (RG) presents a challenge to practitioners in obtaining accurate estimates of measurement properties and examinee ability. In response to this concern, researchers have utilized response times as a proxy of RG and have attempted to improve parameter estimation accuracy by filtering RG responses using popular scoring approaches, such as the effort-moderated item response theory (EM-IRT) model. However, such an approach assumes that RG can be correctly identified based on an indirect proxy of examinee behavior. A failure to meet this assumption leads to the inclusion of distortive and psychometrically uninformative information in parameter estimates. To address this issue, a simulation study was conducted to examine how violations to the assumption of correct RG classification influences EM-IRT item and ability parameter estimation accuracy and compares these results with parameter estimates from the three-parameter logistic (3PL) model, which includes RG responses in scoring. Two RG misclassification factors were manipulated: type (underclassification vs. overclassification) and rate (10%, 30%, and 50%). Results indicated that the EM-IRT model provided improved item parameter estimation over the 3PL model regardless of misclassification type and rate. Furthermore, under most conditions, increased rates of RG underclassification were associated with the greatest bias in ability parameter estimates from the EM-IRT model. In spite of this, the EM-IRT model with RG misclassifications demonstrated more accurate ability parameter estimation than the 3PL model when the mean ability of RG subgroups did not differ. This suggests that in certain situations it may be better for practitioners to (a) imperfectly identify RG than to ignore the presence of such invalid responses and (b) select liberal over conservative response time thresholds to mitigate bias from underclassified RG.

Mathematics Test Development By Item Response Theory Approach And Its Measrument On Elementary School Students

This research is a development and measurement research. This research is conducted to develop a good instrument in measuring elementary school students’ skill by IRT approach which based on; 1) validity and reliability of instrument; 2) assumption test; unidimensional, local independence and parameter invariance; 3) characteristics of item test and 4) measuring students’ ability. This research is developed by the development model of DjemariMardapi, such as: 1) arranging test specification, 2) writing the test, 3) examining the test, 4) conducting trial test, 5) analyzing the item test, 6) improving the test, 7) assembling the test, 8) implementing the test and 9) interpreting the result of test. The research founding; 1) the range of aiken value developed for instrument test of grade 3 is 0.83-1. While the reliability coefficient of instrument test of math skill in grade 3 is 0.883. the result indicates that the item test developed has good validity and reliability. 2) unidimensional test is fulfilled since the test is proven only measure one dominant dimension i.e., the same skill. The assumption test of local independence is also fulfilled due to the value of covariant among interval of skill is small or close to zero. The calculation result between difficulty level from response includes in high category, therefore the assumption of parameter invariance of skill is fulfilled; 3) based on analysis of the three instruments of fit model result, it fits on 2Pl model, so on the parameter estimation of the item, overall package estimates on 2PL model or parameter b and a (difficulty level and trick). Based on the analysis result of parameter item test in grade 3 indicates that the overall item is on good categories of difficulty level and trick; 4) the measurement of students’ skill indicates that the average of students’ skill is 0.451 with the maximum skill score of students is 3.185 and the minimum students’ skill is -2.282. if it is observed from students’ average score, students in the research samples have good average of math skill.

Violation of Conditional Independence in the Many-Facets Rasch Model

ABSTRACT Estimation of parameters for the many-facets Rasch model requires that conditional on the values of the facets, such as person ability, item difficulty, and rater severity, the observed responses within each facet are independent. This requirement has often been discussed for the Rasch models and 2PL and 3PL models, but it becomes more complex with more facets. To show how violation of conditional independence may be exhibited, three scenarios with different types of dependency are developed: (a) raters rating the same work, (b) a residual ability shared by two tasks, and (c) score on one task dependent on observed score on a previous task.

Instrument Test Development of Mathematics Skill on Elementary School

The aims of this research are: 1) producing the test instrument of mathematics skill on elementary school which is valid and reliable, 2) finding out the characteristics of the test instrument of mathematics skill on elementary school. The instrument test development in this research uses the development model of Wilson, Oriondo and Antonio which is modified. The number of testing sample in this research is 160 students in each class. This research results: 1) the validity index of aiken v is 0.979 in grade IV and 0.988 in grade V. The coefficient of instrument skill in class IV and V are 0.883 and 0.954. 2) the compatibility model in this research is it is suitable for 1PL model or parameter b (difficulty level). The result of parameter analysis of test item in class IV and V, shows that the overall item is in good category which is between -2 to 2. The case indicates that the overall item is accepted and reliable to be used for measuring the development of mathematics skill of elementary school students.

Analysis of Quality of Critical Thinking Skills Test Based on Item Response Theory Using R-Program

This research aims to describe the quality of critical thinking skill test of fluid mechanics material in senior high schools in Makassar City. The emphasis is the content validity aspect, the characteristics of each item for the One-Parameter Logistic (1PL) model, the Two-Parameter Logistic (2PL) model, dan the Three-Parameter Logistic (3PL) model. It is a descriptive quantitative research with the subject of all student responses to the critical thinking skills test of fluid mechanics material at senior high schools in Makassar City. There were 726 student responses. The data were collected online by google form and analyzed by using descriptive quantitative technique. The results showed that the critical thinking skill test of fluid mechanics material had fulfilled the content validity. Analysis of the quality of CTS test items showed that the models of 1PL, 2PL, and 3PL were all consistent, showing that the test items were mostly able to discriminate the high abilitytestees from the low abilitytestees. Of the three logistic model approaches used to estimate the parameters of item difficulty level, item discrimination, and guess factor, the 3PL model is better than the other twomodels.

3 ve 4PL Madde Tepki Kuramı Modellerine Göre Farklı Yetenek Kestirim Yöntemlerinin Karşılaştırılması

This research analyzed the two-category Item Response Theory (IRT) models as part of different ability estimation methods. The research was carried out in consideration of responses to 20 items under the Mathematics subtest of TEOG (National Transition from Primary to Secondary Education) exam by the 8th-grade students in 2015-2016. The study group consisted of 400 students who were randomly selected from the students participated in the TEOG exam. Ability estimations and standard error values for these estimations were calculated based on the data. These estimations were compared by two-way analysis of variance (ANOVA) for repeated measurements According to the research findings; it was revealed that the four-parameter logistic (4PL) item model fit better. In terms of ability estimation methods, the accuracy of Weighted Likelihood Estimation (WLE) was higher than Maximum A Posteriori (MAP) and Expected A Posteriori (EAP). WLE and MAP ability estimation model gave lower standard error values compared to the 4PL and 3PL model, respectively. The highest marginal reliability coefficient value for the 3PL model was calculated using estimations made according to MAP while estimations made according to WLE were used for the 4PL model. According to the research findings, it was concluded that the accuracy of ability scores obtained by the WLE estimation method under the 4PL model was higher

Emergency preparedness for heat illness in China: A cross-sectional observational study

BackgroundThe morbidity and mortality rates from heat illness have increased due to a higher number of heatwaves. Effective urgent care of heat illness is crucial for optimizing patient outcomes. However, few studies have examined the emergency preparedness measures required for treating such patients. MethodsFrom December 23, 2019, to January 23, 2020, a content-validated instrument containing the Perceived Emergency Preparedness Scale for heat illness (heatPEPS) was administered to emergency nurses in China through WeChat. Some of these nurses were retested two weeks later. SPSS 26, IRTPRO 4.2, and NVivo 12 Plus were used for data analysis. ResultsIn total, 46.4% (200/431) of the participants returned valid responses. With dichotomous scoring, a high score for heatPEPS (mean 7.29; SD 1.667) was elicited. The reduced 9-item heatPEPS had a perfect fit with the 2PL model (M2 = 27.24, p > 0.05; RMSEA = 0.01) and acceptable internal (α = 0.68) and test-rest reliability (intraclass correlation = 0.56). Many participants (74%) were dissatisfied with their heat illness-related knowledge and skills, suggesting an area that could be improved for better emergency preparedness. ConclusionEmergency departments appear to be well-prepared; however, this is subject to social desirability bias. The 9-item heatPEPS is a reliable and valid tool to measure emergency preparedness for heat illness.

Item and scale properties of the Oral Health Literacy Adults Questionnaire assessed by item response theory.

a) To evaluate the item and scale properties of the Oral Health Literacy Adults Questionnaire (OHL-AQ) in an adult general population. b) To determine precision or accuracy of the respondents' estimated scores along the Oral Health Literacy (OHL) spectrum using item response theory (IRT) modeling. Survey data were collected from a convenience sample of 405 adult attendees of the 2014 Minnesota State Fair. We used the two-parameter logistic (2PL) model for the item response theory (IRT) analyses of OHL-AQ data and calibrated items to estimate model-based item difficulty and discrimination parameters. Item and scale properties were also assessed by plotting and interpreting item characteristic curves (ICCs), test characteristic curve (TCC), and test information function (TIF). Based on interpretation of model coefficients, statistical testing, and model fit criteria, we deemed the 2PL model superior and selected this model to examine item and scale properties. Scale reliability was shown to be good through the test information function (TIF). TIF from our analysis showed that higher levels of OHL were measured less precisely than lower levels of OHL. We demonstrated OHL-AQ as a whole has promising psychometric properties. However, for equiprecise measurement across the scale range, the scale needs more items for measuring higher levels of OHL.

PND63 Knowledge and Understanding of Alzheimer'S Disease: Revisiting the Psychometric Properties of the Adks Questionnaire

The Alzheimer´s Disease Knowledge Scale (ADKS) is a 30-item instrument to assess what people know about AD covering risk factors, diagnosis, symptoms, course, functional impact, caregiving, and management. The aim of this study was to explore scale item characteristics with item response theory (IRT) procedures. A noninterventional, web-based study was conducted among employees of Roche Spain. We compared ADKS scores between participants with and without relatives diagnosed with AD. A nonparametric IRT procedure, Mokken analysis, was performed to explore its underlying structure. A parametric IRT analysis was then used to compare the fitting of two different models for dichotomous items, the one-parameter logistic model (1PL) that assumes the items show different difficulty but share the same discrimination parameter, and the two-parameter logistic model (2PL) that assumes the items show different difficulty and discrimination parameters. A total of 447 participants completed the ADKS. The mean ADKS score was 21.2 (SD 2.8). Participants with a first-degree relative with AD (N=42, 9%) scored higher (mean difference 1.06, p-value = 0.02; standardized mean difference 0.38, indicating a small to moderate effect). Mokken analysis showed that most items (22/30) did not fit to any scale and could be considered scale independent. Two items fitted a knowledge domain related with depression, another two items were related with AD prevention and recovery, and four items fitted a risk factors domain. The likelihood ratio test (LRT) between 1PL and 2PL models was significant (LRT 198.91, df 29, p-value < 0.0001), favoring 2PL. The overall ADKS score distinguishes knowledge about dementia among people with or without AD relatives. IRT analyses suggest an independence among items that is highly relevant for a general knowledge scale. In summary, the 30-item ADKS, as originally reported, could be used to assess the public´s knowledge and to develop and guide educational interventions if needed.