Psychometric Analysis of Student Social Skill Instruments (S3I) with the Rasch Model

Differential item functioning (DIF) analysis is important in terms of test fairness. While DIF analyses have mainly been conducted with manifest grouping variables, such as gender or race/ethnicity, it has been recently claimed that not only the grouping variables but also contextual variables pertaining to examinees should be considered in DIF analyses. This study adopted propensity scores to incorporate the contextual variables into the gender DIF analysis. In this study, propensity scores were used to control for the contextual variables that potentially affect the gender DIF. Subsequent DIF analyses with the Mantel-Haenszel (MH) procedure and the Logistic Regression (LR) model were run with the propensity score applied reference (males) and focal groups (females) through propensity score matching. The propensity score embedded MH model and LR model detected fewer number of gender DIF than the conventional MH and LR models. The propensity score embedded models, as a confirmatory approach in DIF analysis, could contribute to hypothesizing an inference on the potential cause of DIF. Also, salient advantages of propensity score embedded DIF analysis models are discussed.

Adolescent dating violence (ADV) is a global public health problem that has a serious impact on adolescents' physical, psychological, and social development. This study aimed to explore gender disparities in Indonesian adolescents' knowledge of dating violence using the Rasch Model and Differential Item Function Analysis. A total of 250 junior high school students in Yogyakarta, consisting of 107 males and 143 females, participated. The ADV knowledge measurement instrument consisted of 16 previously tested items for validity and reliability. Results showed that female students had a higher level of knowledge than male students, especially in identifying emotional and physical violence. Differential item function (DIF) analysis revealed that two items showed differences in perception based on gender, with female students focusing more on physical violence. In contrast, male students tended to view physical violence as a more common behaviour. This study highlights the importance of more inclusive and gender-sensitive educational programs to increase adolescents' knowledge of different forms of dating violence. The findings provide important insights for the development of interventions that can help prevent dating violence among adolescents.

Rasch modelling was conducted by applying the lifetime NODS criteria using data from a national representative sample aged from 15 to 74 years in Norway (N = 5235). To a large extent, the results replicated previous findings, supporting the view that the DSM-IV gambling symptoms have a unidimensional structure. Differential item functioning (DIF) analysis indicated that younger gamblers (aged 15–25 years) were more likely to endorse the symptom of ‘chasing’ than older gamblers. Likewise, DIF analysis indicated that female gamblers were more likely to report ‘escape’ at lower levels of problem gambling than males. Moreover, the results showed that younger gamblers were less likely to endorse symptoms of ‘withdrawal’ and ‘loss of control’ than older gamblers. As the results may be explained by age and gender-specific correlates (or behaviour), future studies should include extended age groups (e.g. 12–75 years) or conduct Rasch modelling and DIF analysis on specific adolescent gambling instruments.

Investigating differential item functioning across interaction variables in listening comprehension assessment

BackgroundRasch analysis with a focus on Differential Item Functioning (DIF) is increasingly used for examination of psychometric properties of health outcome measures. To take account of DIF in order to retain precision of measurement, split of DIF-items into separate sample specific items has become a frequently used technique. The purpose of the paper is to present and summarise recent advances of analysis of DIF in a unified methodology. In particular, the paper focuses on the use of analysis of variance (ANOVA) as a method to simultaneously detect uniform and non-uniform DIF, the need to distinguish between real and artificial DIF and the trade-off between reliability and validity. An illustrative example from health research is used to demonstrate how DIF, in this case between genders, can be identified, quantified and under specific circumstances accounted for using the Rasch model.MethodsRasch analyses of DIF were conducted of a composite measure of psychosomatic problems using Swedish data from the Health Behaviour in School-aged Children study for grade 9 students collected during the 1985–2014 time periods.ResultsThe procedures demonstrate how DIF can be identified efficiently by ANOVA of residuals, and how the magnitude of DIF can be quantified and potentially accounted for by resolving items according to identifiable groups and using principles of test equating on the resolved items. The results of the analysis also show that the real DIF in some items does affect person measurement estimates.ConclusionsFirstly, in order to distinguish between real and artificial DIF, the items showing DIF initially should not be resolved simultaneously but sequentially. Secondly, while resolving instead of deleting a DIF item may retain reliability, both options may affect the content validity negatively. Resolving items with DIF is not justified if the source of the DIF is relevant for the content of the variable; then resolving DIF may deteriorate the validity of the instrument. Generally, decisions on resolving items to deal with DIF should also rely on external information.

Ensuring fairness in instruments like survey questionnaires or educational tests is crucial. One way to address this is by a Differential Item Functioning (DIF) analysis, which examines if different subgroups respond differently to a particular item, controlling for their overall latent construct level. DIF analysis is typically conducted to assess measurement invariance at the item level. Traditional DIF analysis methods require knowing the comparison groups (reference and focal groups) and anchor items (a subset of DIF-free items). Such prior knowledge may not always be available, and psychometric methods have been proposed for DIF analysis when one piece of information is unknown. More specifically, when the comparison groups are unknown while anchor items are known, latent DIF analysis methods have been proposed that estimate the unknown groups by latent classes. When anchor items are unknown while comparison groups are known, methods have also been proposed, typically under a sparsity assumption – the number of DIF items is not too large. However, DIF analysis when both pieces of information are unknown has not received much attention. This paper proposes a general statistical framework under this setting. In the proposed framework, we model the unknown groups by latent classes and introduce item-specific DIF parameters to capture the DIF effects. Assuming the number of DIF items is relatively small, an L1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L_1$$\\end{document}-regularised estimator is proposed to simultaneously identify the latent classes and the DIF items. A computationally efficient Expectation-Maximisation (EM) algorithm is developed to solve the non-smooth optimisation problem for the regularised estimator. The performance of the proposed method is evaluated by simulation studies and an application to item response data from a real-world educational test.

This research aims to analyze the opinions of classroom teachers on the social skills of refugee students. The qualitative method linked to case study design was used in the research. Within the scope of the research, interviews were conducted with 22 classroom teachers who have knowledge about the concept of social skills; who work in Sivas which was selected with regard to the method of convenience sampling; and who have refugee students in their classes which were selected with regard to the method of criteria sampling. Data were collected by using semi-structured interview and observation forms. Descriptive analysis method was used in the analysis of the data. As a result of the research, it was observed that refugee students have language problems, and they are shy and introvert. It was determined that communication and social skills of the students are weak; yet, they obey school and classroom rules. The number of teachers who stated that refugee students’ ability to solve the problems they face is sufficient and the number of teachers who stated that refugee students’ ability to solve the problems they face is not sufficient are very close to each other. While the teachers state that they do not do any extra exercise in the classes to increase social skills of refugee students and they do not find themselves competent as they do not have knowledge on this subject, they also suggest that students should learn Turkish before they attend school. A variety of suggestions were made in the light of the research findings.

This study aimed to evaluate the psychometric properties of the INAP-L instrument by applying the Rasch model. The psychometric evaluation includes item calibration and differential item functioning (DIF) assessment. Participants in this study were 6153 high school students (2326 boys and 3827 girls) aged 14-19 years (mean age = 15.76, SD = 0.78) from 280 schools spread across 34 provinces in Indonesia. The results of the Rasch model analysis show that the assumptions of unidimensionality and local independence are met. The internal consistency analysis of the INAP-L instrument (PSR = 0.80; Ordinal α = 0.81) and item-person targeting showed quite good results. At the item level, it was discovered that four of the forty items did not fit the Rasch model. Meanwhile, the gender-based DIF analysis revealed that all items were free from gender DIF and that two out of forty items showed DIF based on school type. It can be concluded that the INAP-L instrument has good psychometric properties. Furthermore, multilevel analysis was performed to determine the effect of clustering in the data, and it was discovered that INAP-L has a multilevel data structure with ICC = 0.185 and a design effect > 2.00. This means that future research on the relationship among variables on the INAP-L score must consider a multilevel approach. Even though the developer has already published an interim report on INAP-L, the findings of this study can be used as a reference in improving the instrument before conducting INAP-L in the future.

BackgroundDifferential item functioning (DIF) methods can be used to determine whether different subgroups respond differently to particular items within a health-related quality of life (HRQoL) subscale, after allowing for overall subgroup differences in that scale. This article reviews issues that arise when testing for DIF in HRQoL instruments. We focus on logistic regression methods, which are often used because of their efficiency, simplicity and ease of application.MethodsA review of logistic regression DIF analyses in HRQoL was undertaken. Methodological articles from other fields and using other DIF methods were also included if considered relevant.ResultsThere are many competing approaches for the conduct of DIF analyses and many criteria for determining what constitutes significant DIF. DIF in short scales, as commonly found in HRQL instruments, may be more difficult to interpret. Qualitative methods may aid interpretation of such DIF analyses.ConclusionsA number of methodological choices must be made when applying logistic regression for DIF analyses, and many of these affect the results. We provide recommendations based on reviewing the current evidence. Although the focus is on logistic regression, many of our results should be applicable to DIF analyses in general. There is a need for more empirical and theoretical work in this area.

BackgroundThe Middlesex Elderly Assessment of Mental State (MEAMS) was developed as a screening test to detect cognitive impairment in the elderly. It includes 12 subtests, each having a 'pass score'. A series of tasks were undertaken to adapt the measure for use in the adult population in Turkey and to determine the validity of existing cut points for passing subtests, given the wide range of educational level in the Turkish population. This study focuses on identifying and validating the scoring system of the MEAMS for Turkish adult population.MethodsAfter the translation procedure, 350 normal subjects and 158 acquired brain injury patients were assessed by the Turkish version of MEAMS. Initially, appropriate pass scores for the normal population were determined through ANOVA post-hoc tests according to age, gender and education. Rasch analysis was then used to test the internal construct validity of the scale and the validity of the cut points for pass scores on the pooled data by using Differential Item Functioning (DIF) analysis within the framework of the Rasch model.ResultsData with the initially modified pass scores were analyzed. DIF was found for certain subtests by age and education, but not for gender. Following this, pass scores were further adjusted and data re-fitted to the model. All subtests were found to fit the Rasch model (mean item fit 0.184, SD 0.319; person fit -0.224, SD 0.557) and DIF was then found to be absent. Thus the final pass scores for all subtests were determined.ConclusionThe MEAMS offers a valid assessment of cognitive state for the adult Turkish population, and the revised cut points accommodate for age and education. Further studies are required to ascertain the validity in different diagnostic groups.

Increasingly, tests are being translated and adapted into different languages. Differential item functioning (DIF) analyses are often used to identify non‐equivalent items across language groups. However, few studies have focused on understanding why some translated items produce DIF. The purpose of the current study is to identify sources of differential item and bundle functioning on translated achievement tests using substantive and statistical analyses. A substantive analysis of existing DIF items was conducted by an 11‐member committee of testing specialists. In their review, four sources of translation DIF were identified. Two certified translators used these four sources to categorize a new set of DIF items from Grade 6 and 9 Mathematics and Social Studies Achievement Tests. Each item was associated with a specific source of translation DIF and each item was anticipated to favor a specific group of examinees. Then, a statistical analysis was conducted on the items in each category using SIBTEST. The translators sorted the mathematics DIF items into three sources, and they correctly predicted the group that would be favored for seven of the eight items or bundles of items across two grade levels. The translators sorted the social studies DIF items into four sources, and they correctly predicted the group that would be favored for eight of the 13 items or bundles of items across two grade levels. The majority of items in mathematics and social studies were associated with differences in the words, expressions, or sentence structure of items that are not inherent to the language and/or culture. By combining substantive and statistical DIF analyses, researchers can study the sources of DIF and create a body of confirmed DIF hypotheses that may be used to develop guidelines and test construction principles for reducing DIF on translated tests.

Differential item functioning (DIF) analysis is a way of determining whether test items function differently across subgroups of test takers after controlling for ability level. DIF results are used to evaluate tests' validity arguments. This study uses Rasch measurement to examine the Michigan English Language Assessment Battery listening test for DIF across gender subgroups. After establishing the unidimensionality and local independence of the data, the authors used two methods to test for DIF: (a) a t-test uniform DIF analysis, which showed that two test items displayed substantive DIF, and favored different gender subgroups; and (b) nonuniform DIF analysis, which revealed several test items with significant DIF, many of which favored low-ability male test takers. A possible explanation for gender-ability DIF is that lower ability male test takers are more likely to attempt lucky guesses, particularly on multiple-choice items with unattractive distracters, and that having only two distracters makes this strategy likely to succeed.

In this study, we used the Rasch measurement to investigate the fairness of the listening section of a national computerized high-stakes English test for differential item functioning (DIF) across gender subgroups. The computerized test format inspired us to investigate whether the items measure listening comprehension differently for females and males. Exploring the functioning of novel task types including multimodal materials such as videos and pictures was especially interesting. Firstly, the unidimensionality and local independence of the data were examined as preconditions for DIF analysis. Secondly, the authors explored the performance of female and male students through DIF analysis using the Rasch measurement. The uniform DIF analysis showed that 25 items (out of 30 items) displayed DIF and favored different gender subgroups, whereas the effect size was not meaningful. The non-uniform DIF analysis revealed several items exhibiting DIF with a moderate to large effect size, favoring various gender and ability groups. Explanations for DIF are hypothesized. Finally, implications of the study regarding test development and fairness are discussed.

Lack of students 'social skills makes students' mathematics learning less optimal. The learning model that is carried out is often not maximizing the learning process-oriented towards mathematical communication skills. This makes the teacher strive for a learning method or strategy to foster students' social skills and mathematical communication skills. The Pair Check method is one of the learning methods expected to train and improve students' social skills and mathematical communication skills. The subjects of this study were 24 students of class VII B MTs Riyadlul Ulum Bendungan. The results showed that: (1) students social skills had a percentage of 74.05% with the "most" criteria, which means that most of the students had quite good social skills. At the same time, students' mathematical communication skills have a percentage of 49.55% in the less category. (2) the application of the pair check method can improve students' social skills and mathematical communication. This can be seen from the post-tests average value, which is greater than the pretest, namely pretest posttest or 13.88 19.58. (3) the application of the pair check method has implications for the social skills and mathematical communication skills of students at MTs Riyadlul Ulum Bendungan. This can be seen from the results of the questionnaire and the test results after the pair check method was applied. It was better than the questionnaire results, and the test results before the pair check method were applied

Differential item functioning (DIF) analyses can be used to explore translation, cultural, gender or other differences in the performance of quality of life (QoL) instruments. These analyses are commonly performed using "baseline" or pretreatment data. We previously reported DIF analyses to examine the pattern of item responses for translations of the European Organisation for Research and Treatment of Cancer (EORTC) QLQ-C30 QoL instrument, using only data collected prior to cancer treatment. We now compare the consistency of these results with similar analyses of on-treatment and off-treatment assessments and explore whether item relationships differ from those at baseline. Logistic regression DIF analyses were used to examine the translation of each item in each multi-item scale at the three time points, after controlling for the overall scale score and other covariates. The consistency of results at the three time points was explored. For most EORTC QLQ-C30 subscales, the DIF results were very consistent across the three time points. Results for the Nausea and Vomiting scale varied the most across assessments. The results indicated that DIF analyses were stable across each time point and that the same DIF effects were usually found regardless of the treatment status of the respondent.