Item Bias Research Articles

MCMC stopping rules in latent variable modelling.

Bayesian analysis relies heavily on the Markov chain Monte Carlo (MCMC) algorithm to obtain random samples from posterior distributions. In this study, we compare the performance of MCMC stopping rules and provide a guideline for determining the termination point of the MCMC algorithm in latent variable models. In simulation studies, we examine the performance of four different MCMC stopping rules: potential scale reduction factor (PSRF), fixed-width stopping rule, Geweke's diagnostic, and effective sample size. Specifically, we evaluate these stopping rules in the context of the DINA model and the bifactor item response theory model, two commonly used latent variable models in educational and psychological measurement. Our simulation study findings suggest that single-chain approaches outperform multiple-chain approaches in terms of item parameter accuracy. However, when it comes to person parameter estimates, the effect of stopping rules diminishes. We caution against relying solely on the univariate PSRF, which is the most popular method, as it may terminate the algorithm prematurely and produce biased item parameter estimates if the cut-off value is not chosen carefully. Our research offers guidance to practitioners on choosing suitable stopping rules to improve the precision of the MCMC algorithm in models involving latent variables.

Rasch Analysis of the Chinese Version of the Nociception Coma Scale-Revised in Patients with Prolonged Disorders of Consciousness.

The aim of this study was to analyze the Chinese version of the Nociception Coma Scale-Revised in patients with prolonged disorders of consciousness within the framework of Rasch modeling, including investigating the invariance of total scores across different etiologies of disorders of consciousness. Prospective psychometric study. Patients with prolonged disorders of consciousness from the Rehabilitation and Neurology units in hospital. None. The Nociception Coma Scale-Revised was undertaken by trained raters and the Coma Recovery Scale-Revised was used to assess patients' consciousness. The psychometric properties within the Rasch model including item-person targeting, reliability and separation, item fit, unidimensionality, and differential item functioning were assessed. 84 patients with prolonged disorders of consciousness (mean age 53 years; mean injury 5 months; 42 with Minimally Conscious State and 42 with Unresponsive Wakefulness Syndrome) of 252 observations were enrolled in the study. Through the procedure of repeated assessment and differential item function, a lower item bias Rasch set was purified. The Rasch model assumptions were examined and met, with item reliability and validity meeting the recommended threshold. The Chinese version of the Nociception Coma Scale-Revised demonstrated unidimensionality, good reliability and separation, and good item fit, but dissatisfied person fit and item-person targeting. The verbal subscale showed a notable discrepancy between person responses and the difficulty of the items, suggesting limited clinical significance.

An interplay between cross-cultural and psychometric factors in the Montreal Cognitive Assessment: Experience from the language of a small nation

The present study aimed to test the hypothesis that the total word length on the Memory subtest of the Czech version of the MoCA, which is 12 syllables compared to the English version of 7 syllables, would have a significant effect on Delayed Recall scores compared to the newly created well-balanced version of the test (further MoCA-WLE). In the original Czech version of MoCA, we replaced the 12-syllable word list in the Memory subtest with a 7-syllable list (MoCA-WLE) to make it equivalent to the standard English version in this respect. We analyzed data from 83 participants in the original MoCA group (70.63 ± 7.01 years old, 14.61 ± 3.17 years of education, 30.12% males) and 83 participants in the MoCA-WLE group (70.72 ± 6.95 years old, 14.93 ± 3.48 years of education, 30.12% males). We did not find evidence for a significant word-length effect in the original MoCA versus MoCA-WLE Delayed Recall in either the Mann–Whitney U test (W = 3418.0, p = .932) or multilevel binomial regression (b = 0.10, 95% posterior probability interval [−0.46, 0.68]). The present study shows cross-cultural limits in the adaptation of the test material. The results underline the caveats of such an approach to test adaptation. Fortunately, 12-syllables in the MoCA Memory Czech version versus the original 7-syllable list did not show a detectable word-length effect. We did not find evidence for differential item functioning or cultural item bias. The original MoCA Czech version is psychometrically comparable to the original English version.

Harnessing AI for Educational Measurement: Standards and Emerging Frontiers

The surge of AI in education raises concerns about measurement downsides. Calls for clear standards are warranted. Fortunately, the psychometrics field has a long history of developing relevant standards—like sample invariance and item bias avoidance—crucial for reliable, valid, and interpretable assessments. This established body of knowledge, not unlike traffic laws for self-driving cars, should guide AI assessment development. Measuring new constructs necessitates stronger construct validity research. Instead of rewriting the rulebook, our focus should be on educating AI developers about these standards. This commentary specifically addresses the concern of empowering instructors not with high-stakes testing but with effective item writing through AI. We explore the potential of AI to transform item development, a key area highlighted by researchers. While AI tools offer exciting possibilities for tackling educational challenges, equipping instructors to leverage them effectively remains paramount.

Disentangled variational auto-encoder for multimodal fusion performance analysis in multimodal sentiment analysis

Multimodal Sentiment Analysis (MSA) holds extensive applicability owing to its capacity to analyze and interpret users' emotions, feelings, and perspectives by integrating complementary information from multiple modalities. However, inefficient and unbalanced cross-modal information fusion substantially undermines the accuracy and reliability of MSA models. Consequently, a critical challenge in the field now lies in effectively assessing the information integration capabilities of these models to ensure balanced and equitable processing of multimodal data. In this paper, a Disentanglement-based Variable Auto-Encoder (DVAE) is proposed for systematically assessing fusion performance and investigating the factors that facilitate multimodal fusion. Specifically, a distribution constraint module is presented to decouple the fusion matrices and generate multiple low-dimensional and trustworthy disentangled latent vectors that adhere to the authentic unimodal input distribution. In addition, a combined loss term is modified to effectively balance inductive bias, signal reconstruction, and distribution constraint items to facilitate the optimization of neural network weights and parameters. Utilizing the proposed evaluation method, we can evaluate the fusion performance of multimodal models by contrasting the classification degradation ratio derived from disentangled hidden representations and joint representations. Experiments conducted with eight state-of-the-art multimodal fusion methods on the CMU-MOSEI and CMU-MOSEI benchmark datasets demonstrate that DVAE is capable of effectively evaluating the effects of multimodal fusion. Moreover, the comparative experimental results indicate that the equalizing effect among various advanced mechanisms in multimodal sentiment analysis, as well as the single-peak characteristic of the ground label distribution, both contribute significantly to multimodal data fusion.

Identification of Students’ Comprehension of Radiation Concept Using Rasch Analysis

This community service was conducted using a survey method. The survey aimed to test the understanding of high school students regarding the dangers of radiation exposure from the surrounding environment. Participants in this activity were 48 students (35 science students and 13 social science students) from a high school in Majalaya, Bandung Regency. The instrument used was a two-tier diagnostic test consisting of 4 essay questions related to physics concepts concerning radiation exposure. The analysis of the test results was divided into two parts: conception analysis and Rasch analysis. The conception analysis was done by interpreting students' answers and categorizing them into five conception categories: Full Understanding (FU), Partial Understanding (PU), Misconception (MC), Not Understanding (NU), and No Coding (NC). This categorization was presented in percentage form. The Rasch analysis was conducted based on the distribution of students' answers and item bias detection to identify the spread of students' answers to each question item and the biases present. First, the conception analysis results showed that 53% of students were included in the FU category, while only 5% were in the Not Understanding (NU) category. This indicates that students' understanding of the dangers of radiation exposure is quite good, as more than half of the participants are in the FU category. Second, the Rasch analysis results based on the distribution of students' answers showed that more students could answer basic comprehension questions compared to classification and analysis-based questions. Additionally, the Rasch analysis results based on item bias detection indicated that there was no gender or academic major bias in the four question items.

A Comparison of Limited Information Estimation Methods for the Two-Parameter Normal-Ogive Model with Locally Dependent Items

The two-parameter normal-ogive (2PNO) model is one of the most popular item response theory (IRT) models for analyzing dichotomous items. Consistent parameter estimation of the 2PNO model using marginal maximum likelihood estimation relies on the local independence assumption. However, the assumption of local independence might be violated in practice. Likelihood-based estimation of the local dependence structure is often computationally demanding. Moreover, many IRT models that model local dependence do not have a marginal interpretation of item parameters. In this article, limited information estimation methods are reviewed that allow the convenient and straightforward handling of local dependence in estimating the 2PNO model. In detail, pairwise likelihood, weighted least squares, and normal-ogive harmonic analysis robust method (NOHARM) estimation are compared with marginal maximum likelihood estimation that ignores local dependence. A simulation study revealed that item parameters can be consistently estimated with limited information methods. At the same time, marginal maximum likelihood estimation resulted in biased item parameter estimates in the presence of local dependence. From a practical perspective, there were only minor differences regarding the statistical quality of item parameter estimates of the different estimation methods. Differences between the estimation methods are also compared for two empirical datasets.

Using moderated nonlinear factor models to adjust for differential item functioning in the Student-Teacher Relationship Scale from kindergarten to Grade 6

Prior research has demonstrated that children form developmentally salient relationships with teachers and that these relationships are uniquely predictive of subsequent functioning both in and outside of school. However, prior work estimating trajectories and predictors of teacher-student relationship quality has failed to test and adjust for bias in questionnaire items. The present study used longitudinal data from the NICHD Study of Early Child Care and Youth Development (SECCYD; N = 1140) to test and adjust for measurement bias in the Student-Teacher Relationship Scale (STRS; Pianta, 2001) across grades (K–6) and sociodemographic characteristics (i.e., birth sex, race/ethnicity, family income-to-needs ratio, and maternal education) to generate less biased estimates of trajectories of teacher-student relationship quality. Results identified differential item functioning for three of seven STRS items assessing conflict and three of eight STRS items assessing closeness, with items functioning differentially across child grade, birth sex, race/ethnicity, and maternal education level. Comparisons of growth models using non-adjusted and adjusted STRS scores highlight substantive differences between scoring approaches, such that the effects of race/ethnicity, maternal education, and maternal sensitivity on teacher-student relationship quality were masked prior to adjusting for item bias. These findings demonstrate the importance of testing and correcting for item bias in questionnaire-based assessments of teacher-student relationship quality to ensure valid conclusions.

Psychometric Assessment of an Item Bank for Adaptive Testing on Patient-Reported Experience of Care Environment for Severe Mental Illness: Validation Study.

The care environment significantly influences the experiences of patients with severe mental illness and the quality of their care. While a welcoming and stimulating environment enhances patient satisfaction and health outcomes, psychiatric facilities often prioritize staff workflow over patient needs. Addressing these challenges is crucial to improving patient experiences and outcomes in mental health care. This study is part of the Patient-Reported Experience Measure for Improving Quality of Care in Mental Health (PREMIUM) project and aims to establish an item bank (PREMIUM-CE) and to develop computerized adaptive tests (CATs) to measure the experience of the care environment of adult patients with schizophrenia, bipolar disorder, or major depressive disorder. We performed psychometric analyses including assessments of item response theory (IRT) model assumptions, IRT model fit, differential item functioning (DIF), item bank validity, and CAT simulations. In this multicenter cross-sectional study, 498 patients were recruited from outpatient and inpatient settings. The final PREMIUM-CE 13-item bank was sufficiently unidimensional (root mean square error of approximation=0.082, 95% CI 0.067-0.097; comparative fit index=0.974; Tucker-Lewis index=0.968) and showed an adequate fit to the IRT model (infit mean square statistic ranging between 0.7 and 1.0). DIF analysis revealed no item biases according to gender, health care settings, diagnosis, or mode of study participation. PREMIUM-CE scores correlated strongly with satisfaction measures (r=0.69-0.78; P<.001) and weakly with quality-of-life measures (r=0.11-0.21; P<.001). CAT simulations showed a strong correlation (r=0.98) between CAT scores and those of the full item bank, and around 79.5% (396/498) of the participants obtained a reliable score with the administration of an average of 7 items. The PREMIUM-CE item bank and its CAT version have shown excellent psychometric properties, making them reliable measures for evaluating the patient experience of the care environment among adults with severe mental illness in both outpatient and inpatient settings. These measures are a valuable addition to the existing landscape of patient experience assessment, capturing what truly matters to patients and enhancing the understanding of their care experiences. ClinicalTrials.gov NCT02491866; https://clinicaltrials.gov/study/NCT02491866.

Evaluation of the Posttraumatic Stress Disorder Checklist for DSM-5 across United States and India samples: An item response theory analysis using the graded response model.

The Posttraumatic Stress Disorder (PTSD) Checklist for DSM-5 (PCL-5) is frequently used to assess PTSD symptoms. Extending its psychometric investigations across distinct samples (United States and India), the aims of the present study included investigating the item characteristics, person fit, and differential item functioning (DIF) across the two samples. We (a) conducted item analysis using the graded response model to examine item characteristics (thresholds and discrimination parameters) and (b) examined person fit to determine participants' response patterns. The U.S. sample included 176 trauma-exposed individuals seeking mental health treatment, and the Indian sample included 148 trauma-exposed ex-military personnel. Results indicated low discrimination for Item 8 and low but acceptable discrimination for Item 17 for the U.S. and Indian samples, respectively. Across both samples, the most unused response categories were the middle-point or extreme (higher severity) categories (all response categories were better utilized in the Indian sample), and half the participants exhibited person misfit. Supplemental DIF analysis indicated that five items exhibited DIF indicating potential item bias; however, effect sizes for DIF were negligible. Although the PCL-5 demonstrated strong psychometric properties and showed promise for use across the samples differing on cultural and demographic characteristics, some of the items and the number of categories used to measure them could be revisited to create a more efficient instrument. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Assessment of adult cognitive abilities in Greece: A differential item functioning study of the General Ability Measure for Adults (GAMA)

The purpose of this study was to evaluate the generalizability and possible adaptation for use of a non-verbal measure of intelligence developed in the United States, the General Ability Measure for Adults (GAMA; Naglieri &amp; Bardos, 1997) in Greece. As a validity evidence the study examined the differential item function of its questions in order to explore potential item bias utilizing the disproportionate group probabilities of participants’ correctly endorsing test items. The analysis was performed using a logistic regression procedure with samples from the United States (n = 2,369) and Greece (n = 1,273). The findings indicate a small (&lt;1%) number of items that work differentially between these two cultural groups. Implications for the development and weighting of cross-cultural intelligence assessment tests using non-verbal measures are discussed.

Invariance and item bias of the Mental Health Continuum Short-Form for South African university first-year students

Over the last decade, higher education institutions (HEIs) have become increasingly interested in student well-being. However, since the student population is very diverse in South Africa, questionnaires measuring the well-being of students must be psychometrically sound for different cultural and demographic groups. This study aimed to determine the psychometric properties of the Mental Health Continuum Short-Form (MHC-SF), including factorial validity, measurement invariance, item bias and internal consistency. The sample consisted of 1285 first-year university students. The three-factor structure of the MHC-SF was confirmed, indicating that emotional, social and psychological well-being are three independent factors. Invariance results showed that the MHC-SF produced similar results across campuses and gender sub-groups, although partial invariance was present among language groups. Item bias was present for different sub-groups, but the practical impact was negligible. Reliability scores indicated that all three dimensions are reliable in this sample. This study’s findings could help higher education institutions with preliminary results on the validity and reliability of a widely used well-being measure to assess university students’ subjective well-being and could aid in investigating and measuring first-year students’ overall well-being during their transition to tertiary education.Contribution: This study contributes to creating knowledge about fair and unbiased measurement of student well-being across different sub-groups in South Africa.

Minimizing Social Desirability in Questionnaires of Non-Cognitive Measurements

Data obtained through questionnaires sometimes respond to the items presented by social norms, so sometimes they do not suit themselves. High social desirability (SD) in non-cognitive measurements will cause item bias. Several ways are used to reduce item bias, including freeing respondents from not writing their names or being anonymous, explaining to the participants to respond to each statement honestly, as they are or according to themselves, and responding to the questionnaire online or offline. This research aims to prove that several methods can minimize the possibility of item bias SD and academic dishonesty (AD). The research was carried out with an experimental study using a factorial design. There were 309 respondents who were willing to be involved in this research. Data analysis was carried out using multivariate ANOVA. The research results show differences for all variables, Self-Deceptive Enhancement (SDE), Impression Management (IM), and AD in the anonymous group. There are differences in AD in the groups that provide a complete explanation and do not explain, and there is an interaction between the average AD based on the anonymous and explanation group.

Minimizing Social Desirability in Questionnaires of Non-Cognitive Measurements

Data obtained through questionnaires sometimes respond to the items presented by social norms, so sometimes they do not suit themselves. High social desirability (SD) in non-cognitive measurements will cause item bias. Several ways are used to reduce item bias, including freeing respondents from not writing their names or being anonymous, explaining to the participants to respond to each statement honestly, as they are or according to themselves, and responding to the questionnaire online or offline. This research aims to prove that several methods can minimize the possibility of item bias SD and academic dishonesty (AD). The research was carried out with an experimental study using a factorial design. There were 309 respondents who were willing to be involved in this research. Data analysis was carried out using multivariate ANOVA. The research results show differences for all variables, Self-Deceptive Enhancement (SDE), Impression Management (IM), and AD in the anonymous group. There are differences in AD in the groups that provide a complete explanation and do not explain, and there is an interaction between the average AD based on the anonymous and explanation group.

Popularity-Debiased Graph Self-Supervised for Recommendation

The rise of graph neural networks has greatly contributed to the development of recommendation systems, and self-supervised learning has emerged as one of the most important approaches to address sparse interaction data. However, existing methods mostly focus on the recommendation’s accuracy while neglecting the role of recommended item diversity in enhancing user interest and merchant benefits. The reason for this phenomenon is mainly due to the bias of popular items, which makes the long-tail items (account for a large proportion) be neglected. How to mitigate the bias caused by item popularity has become one of the hot topics in current research. To address the above problems, we propose a Popularity-Debiased Graph Self-Supervised for Recommendation (PDGS). Specifically, we apply a penalty constraint on item popularity during the data enhancement process on the user–item interaction graph to eliminate the inherent popularity bias. We generate item similarity graphs with the popularity bias removed to construct a self-supervised learning task under multiple views, and we design model optimization strategies from the perspectives of popular items and long-tail items to generate recommendation lists. We conduct a large number of comparison experiments, as well as ablation experiments, on three public datasets to verify the effectiveness and the superiority of the model in balancing recommendation accuracy and diversity.

Antibiotic knowledge assessment questionnaire in undergraduate pharmacy students: A Rasch analysis of validity evidence

Background: Antibiotic knowledge is crucial for undergraduate pharmacy students who are future healthcare professionals. However, a valid and reliable instrument to assess their knowledge is scarce. This study aimed to develop and validate an Antibiotic Knowledge Assessment Questionnaire (AKAQ). Methods: The AKAQ had three domains and 29 items, encompassing general antibiotic knowledge, antibiotic resistance, and antibiotic stewardship. Rasch analysis was used to assess psychometric properties, including validity parameters (item and person fit and structural validity), reliability (person and item reliability, Cronbach’s Alpha value), item-person interaction, and item bias, using differential item functioning (DIF) based on tems. Results: This study included 500 undergraduate pharmacy students from 90 Indonesian universities. The validity of the questionnaire was demonstrated, except for one item. Person means infit and outfit for MNSQ were 1.02 and 0.95, whereas ZSTD were 0.11 and 0.08, respectively. Items means infit and outfit for MNSQ were 1.01 and 0.96, while ZSTD were 0.11 and -0.23. Item and person reliabilities were acceptable at 0.99 and 0.68. Cronbach’s alpha reliability was acceptable at 0.71. Two items were biased by term. Conclusion: The AKAQ is a valid, reliable, and standard instrument for assessing the antibiotic knowledge levels of undergraduate pharmacy students.

Benefits of the Curious Behavior of Bayesian Hierarchical Item Response Theory Models-An in-Depth Investigation and Bias Correction.

When using Bayesian hierarchical modeling, a popular approach for Item Response Theory (IRT) models, researchers typically face a tradeoff between the precision and accuracy of the item parameter estimates. Given the pooling principle and variance-dependent shrinkage, the expected behavior of Bayesian hierarchical IRT models is to deliver more precise but biased item parameter estimates, compared to those obtained in nonhierarchical models. Previous research, however, points out the possibility that, in the context of the two-parameter logistic IRT model, the aforementioned tradeoff has not to be made. With a comprehensive simulation study, we provide an in-depth investigation into this possibility. The results show a superior performance, in terms of bias, RMSE and precision, of the hierarchical specifications compared to the nonhierarchical counterpart. Under certain conditions, the bias in the item parameter estimates is independent of the bias in the variance components. Moreover, we provide a bias correction procedure for item discrimination parameter estimates. In sum, we show that IRT models create a unique situation where the Bayesian hierarchical approach indeed yields parameter estimates that are not only more precise, but also more accurate, compared to nonhierarchical approaches. We discuss this beneficial behavior from both theoretical and applied point of views.

Response bias in Likert-style psychological items – an example from a large-scale travel survey in China

This paper addresses the challenge of ensuring response quality when using item sets with Likert scales in travel surveys. Particularly for capturing psychology, such item sets play an important role in travel behavior research. A challenge with this kind of data is the identification of response bias. An example is straightlining, which describes selecting the same response category for each item. Since there is no universal indicator in the literature to identify unusual or strategic response patterns, we apply various indicators, compare the results and develop a new indicator based on correlations which encounters plausible straightlining.

Evaluation of Item Bias Using Differential Item Functioning (DIF) Technique in NECO Conducted Economics Examination in Taraba State, Nigeria

Differential item functioning is an approach that is widely used to find out items that are bias. This study investigated items that are bias using differential item functioning approach in relation to gender, school ownership (private and public schools) and school location (urban and rural schools) using National Examinations Council (NECO) Economics questions for June/July 2022/2022. The research design employed in this study was exposit facto research design. The study sample comprised students in Taraba State, Nigeria. One Hundred (100) students were used for study. And the test contains 60 items which was administered to the students. Logistic regression was used to analysis the data. The research findings showed that out of sixty items in NECO Economics questions 11 items were biased in relation to gender, 7 items were biased in relation to school ownership and 9 items in relation to school location. The implication of these findings is that NECO economics examinations questions have presences of differential item functioning (DIF). From the result of the findings, it was then recommended that test experts and developers should explore the use of DIF approach to detect biased items.

Developing and Validating Critical-Heutagogy Learning (CHL) to Enhance Lifelong Learning: A Rasch Analysis Approach

This research is dedicated to the development and validation of Critical-Heutagogy Learning (CHL), an innovative pedagogical approach designed to foster lifelong learning among university students. In an era characterized by rapid changes, the imperative for lifelong learning is paramount, necessitating individuals to continually update their knowledge and skills to remain adaptable and successful in diverse environments. To evaluate the validity and reliability of CHL, this study employs Rasch analysis. The development phase of CHL encompassed an exhaustive review of relevant literature, consultations with subject matter experts, and an iterative design methodology. This process was critical to ensure that CHL adhered to established principles of lifelong learning while promoting critical thinking and heutagogical practices. For the validation phase, CHL tools were administered to a heterogeneous group of students across multiple educational contexts. The gathered data underwent rigorous Rasch analysis to ascertain the framework's reliability, evaluate item fit, and pinpoint areas necessitating enhancements. The findings indicate that CHL is both a valid and reliable approach, with a high percentage (89.5%) of student respondents affirming its effectiveness in fostering engagement in learning and enhancing autonomous learning competencies. Additionally, this study delves into the interactions between bias and differential item functioning. The outcomes of this research offer a valuable contribution to the field of education, presenting a rigorously validated framework that supports lifelong learning. This is achieved through the promotion of critical thinking and the facilitation of independent learning skills, thus aligning with contemporary educational objectives and needs.