Item Response Theory Approach Research Articles

Development and Validation of a Multidimensional Perceived Media Convergence Scale: A Classical Test Theory and Item Response Theory Approach

Development and Validation of a Multidimensional Perceived Media Convergence Scale: A Classical Test Theory and Item Response Theory Approach

Evidence of validity of a Jealousy Scale in Peruvian youth and adults: An item response theory approach

Introduction/Objective: In romantic relationships, jealousy can influence how individuals feel emotionally and interact with their partners. This study analysed the metric properties of the Brief Jealousy Scale (BJS) in young adults from Peru using current methods. Method: A total of 297 individuals were selected to participate in the study. They averaged 26.52 years of age (with a variation of 7.75 years), and 74.10% were women. The study used a technique called Item Response Theory (IRT) with a two-parameter model to confirm the structure and reliability of the BJS. Result: The BJS demonstrated high reliability and fit through goodness-of-fit indices. DIF analysis indicated slight gender biases in some items, reflecting differences in the perception of jealousy between men and women. Conclusions: The study highlights the importance and usefulness of the Jealousy Scale in relationships, as it helps to understand relationship dynamics and guides interventions. Future studies should address issues such as participant selection and study design and examine whether the scale can be used in different cultures.

Trajectories of healthy aging and their association with prodromal parkinson disease: The HELIAD study

Background: The aim of the present study was to investigate the association of prodromal PD (pPD) with trajectories of healthy aging, according to its latest definition by the WHO. Methods: In a sample of 1,226 older adults (704 women), PD diagnosis was reached through standard clinical research procedures. Probability of pPD was calculated according to the International Parkinson and Movement Disorder Society’s research criteria for PD-free participants. A healthy aging metric was introduced using an item response theory approach (IRT) based on information from validated questionnaires assessing functionality. Four trajectories of healthy aging were created based on whether the healthy aging status of participants was above or below the median at baseline and follow up: High-High, High-Low, Low-High and Low-Low. Results: 34.3% belonged to the High-High group, 15.7% to the High-Low, 18.6% to the Low-High and 31.4% to the Low-Low group. Participants with possible/probable pPD were 78% less likely to belong in High-High trajectory of healthy aging as compared to those without pPD (OR = 0.22, 95%CI 0.06-0.79, p-value = 0,02). Conclusion: Our findings suggest an inverse association of pPD probability with healthy aging among older adults; Further research is needed to investigate the clinical implications of this association.

Common measures or common metrics? the value of IRT-based common metrics

There is a clear need to harmonize outcome measurement. Some authors propose to express scores as T scores to facilitate interpretation of PROM results in clinical practice. While this is a step in the right direction, there are important limitations to the acceptance of the T score metric as a common metric when T scores are based on raw sum scores of ordinal items: Such T scores of different instruments are not exactly comparable because they are not interval scaled; T scores of different measures are only on the same scale if exactly the same reference group is used; and the T sore metric cannot be maintained because it is reference population-dependent and needs to be updated regularly. These limitations can be overcome by using an item response theory (IRT)-based metric. Items from different measures can be placed on the same IRT metric to make scores comparable on an interval scale. The PROMIS initiative used IRT to develop item banks for measuring various health outcomes. Other PROMs have been linked to the PROMIS metric. Although PROMIS uses a T-score metric for practical reasons, the underlying PROMIS metric is actually an IRT metric. An IRT approach also enables further development of an item bank while preserving the underlying metric. Therefore, IRT-based metrics should be considered as common metrics for the future.

Performance Evaluation of Oral Health Teams in Brazil: An Item Response Theory Approach

Access to quality healthcare is an essential component of health policy. This cross-sectional study describes the actions performed by Brazilian Oral Health Teams (OHTs) analyzed in the National Program for Improving Access and Quality of Primary Care and the relationship of contextual aspects. A total of 22,993 OHTs that participated in the third cycle of the referred program were evaluated using a structured questionnaire. Thirteen procedures (items) related to spontaneous dental care and preventive, surgical, restorative, prosthetic, and oral cancer prevention or diagnosis were assessed. Item response theory was used to estimate the performance scores of OHTs, based on 13 items. The relationship between performance scores and contextual variables in Brazilian regions was analyzed by thematic maps and Spearman correlation (p < 0.05). The highest difficulty parameters were for questions related to prosthetics (b = 0.879) and actions for oral cancer (b = 2.922). The 13 items were more appropriate to discriminate the teams with lower performance and relatively ineffective in differentiating those with better performance. A direct relationship with the Human Development Index (rs = 0.249; p = 0.004) and an indirect relationship with the Gini Index (rs = −0.482; p < 0.001) were found. Contextual aspects presented a relationship with the performance of the Brazilian OHTs. The evaluated items showed some potential to discriminate the performance of OHTs since many of the evaluated dental procedures are performed by most of the teams.

Numeric rating scale for pain should be used in an ordinal but not interval manner. A retrospective analysis of 346,892 patient reports of the quality improvement in postoperative pain treatment registry.

To assess postoperative pain intensity in adults, the numeric rating scale (NRS) is used. This scale has shown acceptable psychometric features, although its scale properties need further examination. We aimed to evaluate scale properties of the NRS using an item response theory (IRT) approach. Data from an international postoperative pain registry (QUIPS) was analyzed retrospectively. Overall, 346,892 adult patients (age groups: 18-20 years: 1.6%, 21-30 years: 6.7%, 31-40 years: 8.3%, 41-50 years: 13.2%, 51-60 years: 17.1%, 61-70 years: 17.3%, 71-80 years: 16.4%, 81-90 years: 3.9%, >90: 0.2%) were included. Among the patients, 55.7% are female and 38% had preoperative pain. Three pain items (movement pain, worst pain, least pain) were analyzed using 4 different IRT models: partial credit model (PCM), generalized partial credit model (GPCM), rating scale model (RSM), and graded response model (GRM). Fit indices were compared to decide the best fitting model (lower fit indices indicate a better model fit). Subgroup analyses were done for sex and age groups. After collapsing the highest and the second highest response category, the GRM outperformed other models (lowest Bayesian information criterion) in all subgroups. Overlapping categories were found in category boundary curves for worst and minimum pain and particularly for higher pain ratings. Response category widths differed depending on pain intensity. For female, male, and age groups, similar results were obtained. Response categories on the NRS are ordered but have different widths. The interval scale properties of the NRS should be questioned. In dealing with missing linearity in pain intensity ratings using the NRS, IRT methods may be helpful.

Day Scholars Food Insecurity Experience Scale-Survey Module (DSFIES-SM): Psychometric Analysis

Background: School feeding programs’ evaluation requires the measurement of food insecurity, a more objective indicator, within school in low-income countries. The Global Child Nutrition Foundation (GCNF) uses subjective indicators to report school feeding coverage rates across many countries that participate in the global survey of school meal programs all year round. Aim: To test the methodological feasibility of measuring a school food insecurity construct as a direct indicator of the effectiveness and efficiency of school feeding programs. Methods: Two-stage sampling was used during the selection of schools and 128 schoolchildren with a mean ( SD) age of 10.5(1.58) were recruited. Item Response Theory (IRT) and Classical Test Theory (CTT) approaches were utilized during DSFIES-SM development. Psychometric analysis was utilized to assess the psychometric properties of the measure of school food insecurity and to establish the construct-level reliability, convergent and discriminant validity of the DSFIES-SM. Results: DSFIES-SM generated acceptable item-level reliabilities, ranging from .75 to .78. The construct-level reliability of the DSFIES-SM was indicated by Cronbach’s α of .78. Composite reliability was at .77. Fit measures and tests of model fitness for confirmatory factor analysis (CFA) (RMSEA < 0.0001; TLI = 1.06; SRMR = 0.043; CFI = 1.00, p = .85) confirmed that the data fitted the model perfectly. Conclusion: DSFIES-SM consists of twelve questions with dichotomous yes/no responses. DSFIES-SM has good convergent and discriminant validity. Findings suggest that the items compose a statistical scale designed to cover a range of severity of school food insecurity. Future replications to establish other forms of validity across different cultural contexts in low-income countries can be of benefit to the present research.

Standard Assessment Model for Scientific Integration-Interconnection as Material for Evaluation of Vision and Mission Achievement

This study aims to develop a standardized assessment model of scientific integration-interconnection, which begins with conducting a literature review and needs assessment, setting goals, developing initial products, validating contents, testing readability, testing instruments, analyzing results, and ending with an interpretation of the results of the assessment. The method used is Research and Development. Testing of the assessment instruments was carried out for each element of the Tridharma of Higher Education, namely 76 lesson plans, 139 research and development works, and 41 community service works. The scientific integration-interconnection assessment model has characteristics according to the structure of the second order model, with the first order being 4 scientific integration-interconnection domains (philosophical domain, material domain, methodological domain, and strategic domain) and the second order is 15 aspects of scientific integration-interconnection. Data analysis techniques using Aiken-V, Exploratory Factor Analysis/EFA and Confirmatory Factor Analysis/CFA, Omega, Item Response Theory approach of polytomous scoring, and descriptive statistics. The assessment model is feasible, fulfilling the criteria of content validity, construct validity (p-value = 0.11949 and RMSEA = 0.027), estimated instrument reliability (α = 0.816), and construct reliability (ɷ =0.939). The scoring used the GRM approach with good quality in terms of the results of the 94% item fit test with an item discrimination index of 0.655 < DISCi < 1.153 and the average item difficulty level of -1.965 < DIFFi < -0.662.

Analsis Kemampuan Peserta Didik pada Sumatif Akhir Semester (SAS) Menggunakan Pendekatan Item Response Theory (IRT) Berbantuan Modular Object-Oriented Dynamic Learning Environment (MOODLE)

This article aims to determine the quality of instruments and the ability of end-of-semester summative learners in science subjects’ grade 7 at MTsN 1 Ketapang using the Item Response Theory (IRT) approach assisted by Modular Object-Oriented Dynamic Learning Environment (MOODLE). Respondents involved in this study were 287 learners with 25 question items. The application used for item response theory is Software Quest. The results of the analysis using Quest Software show that the reliability reliability is very high, validity is good, and the level of difficulty is good. Based on the analysis using item response theory, teachers can know the quality of the assessment of the questions developed and the ability of students. The use of analysis with item response theory is recommended to be used by every teacher to help teachers in knowing the quality of the assessment questions developed and the ability of students. every teacher so that it can help teachers in knowing the quality of the assessment instrument and the ability of student.

Item response theory approach to ethnocentrism

IntroductionAlthough ethnocentrism is one of the fundamental concepts in the social sciences, its study has been impeded by a diversity of conceptualizations and measures. In recent years, a growing number of political scientists and psychologists have undertaken in-depth research into ethnocentrism. In addition, researchers have recently proposed a comprehensive reconceptualization of ethnocentrism and developed a new Ethnocentrism scale. There is strong evidence for this scale's reliability and validity in indexing ethnocentrism, but like most measures in psychology and political science, this scale is based on classical test theory. Item response theory (IRT) is a powerful psychometric technique that can provide a much more sophisticated test of test performance and is currently under-utilized in research.MethodsWe performed IRT to assess the psychometric properties of the Ethnocentrism scale on a sample of 4,187 participants.ResultsThe scale's items had strong psychometric properties to capture the ethnocentrism latent construct, particularly in the below average to above average range. Men required marginally lower levels of ethnocentrism to endorse less socially acceptable items than women (items relating to superiority, purity, or exploitativeness). When compared to liberals, conservatives responded more readily to nearly all ethnocentrism items. Given this variation, the IRT approach highlighted that future measurements must adjust for differential item functioning, albeit more for political orientation than gender identity.DiscussionThe findings detail how IRT can enhance measurement in political science and demonstrate the implications for how gender and political ideology may affect the differential performance of items.

“Use the Force!” Adaptation of Response Formats

Abstract. The present paper features the adaptation of an existing Big Five questionnaire with a rating scale (RS) response format into a measure using a multidimensional forced choice (MFC) response format. Rating scale response formats have been criticized for their proneness to intentional and unintentional response distortions. Multidimensional forced choice response formats were suggested as a solution to mitigate several types of response sets and response styles by design. The Big Five Inventory of Personality in Occupational Situations (B5PS) is a situation-based questionnaire designed for personnel selection and development purposes which would benefit from fake-proof response formats. MFC response formats require special effort during test construction and calibration which will be laid out here. Changing the response format has severe consequences on item design and scoring. An inherent issue with MFC formats derives from their inability to yield interpersonal comparative results from standard (sum) scoring. This issue can be solved with item response theory (IRT)-based calibration during test construction. The Thurstonian IRT approach (TIRT) was developed by Brown and Maydeu-Olivares (2011) , and aspects of MFC item design and TIRT calibrations are explored in this paper. Evidence on structural and construct validity are presented alongside recommendations on the test development processes. The results support the feasibility of the concept of MFC test construction with TIRT calibration in a contextualized and situation-based item format.

Measuring skill-based health literacy in chronic airway disease patients: the development and psychometric evaluation of the Vancouver airways health literacy tool (VAHLT).

This article describes the development of the Vancouver airways health literacy tool (VAHLT), a novel measure of skill-based health literacy specific to chronic airway diseases (CADs). Across several phases, psychometric characteristics of the VAHLT were examined and used to guide its development. An initial pool of 46 items was developed using input from patients, clinicians, researchers, and policy-makers. An initial patient sample (N = 532) was evaluated and used to inform item revisions. A revised 44-item pool was then evaluated using a second sample, the results of which aided in the selection of a final set of 30 items. The finalized 30-item VAHLT was then psychometrically evaluated using the second sample (N = 318). An item response theory approach was utilized to evaluate the VAHLT by assessing model fit, item parameter estimates, test and item information curves, and item characteristic curves. Reliability was assessed using ordinal coefficient alpha. We additionally assessed differential item functioning between asthma and COPD diagnoses. The VAHLT demonstrated a unidimensional structure and reasonably discriminated patients in the lower range of health literacy estimates. The tool demonstrated strong reliability (α = .920). Two of the 30 items were found to exhibit non-negligible differential item functioning. This study presents compelling evidence of validity in several areas for the VAHLT, including content and structural validity. Further external validation studies are needed and forthcoming. Overall, this work represents a strong first step towards a novel, skill-based, and disease-specific measure of CAD-related health literacy.

The development and validation of the Chinese safety climate scale using the item-response theory approach.

To develop a valid and practical tool to measure the safety climate in China, and further raise awareness of it in Chinese industries, we developed the Chinese safety climate scale (including two subscales at the levels of organization and group separately) based on the work of Huang et al. in 2017. A descriptive survey with the convenience sampling method was conducted in Shanghai Disney Resort, China. A sample of 1,570 employees was involved in the final data analyses. The item response theory (IRT) analyses with graded response models were conducted using. The unidimensionality and local independence assumption were held. The Cronbach's α of organization- and group- level safety climate subscale was 0.912 and 0.937, respectively. The confirmatory factor analysis (CFA) showed good model fit for a one-factor model of the organization-level safety climate subscale, χ2 (df = 20) = 129.158, p < 0.001, CFI = 0.994, TLI = 0.992, NFI = 0.993, IFI = 0.994, RMSEA = 0.059, 90% CI = (0.050, 0.069), and SRMR = 0.048. A one-factor model also fits well for the group-level safety climate subscale, χ2 (df = 44) = 219.727, p < 0.001, CFI = 0.996, TLI = 0.9925, NFI = 0.995, IFI = 0.996, RMSEA = 0.050, 90% CI = (0.044, 0.057), and SRMR = 0.046. Discrimination and difficulty parameters showed that all items effectively spanned the range of the latent trait and could successfully separate participants at different safety climate levels. Items in the organization-level and group-level Chinese safety climate subscales had significantly different discrimination parameters, fitted well with the models, and had a substantive relationship with the latent traits. The Chinese safety climate scale was reliable and valid overall. They can facilitate the research and survey regarding the safety climate in China.

Validation of a Chinese Version of the Digital Stress Scale and Development of a Short Form Based on Item Response Theory Among Chinese College Students.

In 2021, Hall et al developed the Digital Stress Scale (DSS), but its psychometric characteristics were only tested using classical test theory (CTT). In this study, we use item response theory (IRT) and CTT to develop and verify a Chinese version of the DSS and its short version, which can improve the reliability and effectiveness of the digital stress measurement tool for Chinese college students. In this study, we developed a Chinese version of the DSS (DSS-C) and recruited 1506 Chinese college students as participants to analyze its psychometric characteristics based on CTT and item response theory methods. First, we used CTT, including common method bias, construct validity, criterion-related validity, internal consistency, test-retest reliability and measurement invariance. Then, we adopted the IRT approach to examine the item parameters, item characteristics, item information, differential item function, test information, and test reliability of the DSS-C. Finally, a short form (DSS-C-S) was constructed, and the psychometric characteristics of the DSS-C-S were examined. (1) The five-factor structure of the DSS-C was verified. The DSS-C shows good internal reliability, test-retest reliability, criterion-related validity and measurement invariance between urban college students and rural college students. (2) All 24 items had reasonable discrimination parameters and location parameters and were DIF-free by gender. Except for Items 4 and 10, all the other items had high information and measurement reliability at medium θ levels. (3) Compared with the DSS-C, the 22-item short form also has good reliability and validity and maintains sufficient measurement accuracy while reducing items of poor quality. Both the DSS-C and the DSS-C-S have good psychometric characteristics and are accurate and effective tools for measuring the digital stress of Chinese college students.

Validation of the Bangla Communication Scale among Bangladeshi adolescents: A Classical Test Theory and Item Response Theory approach.

Assessing communication skills is necessary to facilitate pro-communication skills development programs. The 23-item Communication Scale (CS) is the most widely used tool for this purpose. Since there is a scarcity of validated tools to assess communication skills among Bangladeshi adolescents, we translated this questionnaire into Bangla and validated it on a Bangladeshi adolescent sample. We conducted two independent rounds of large-scale surveys that yielded data from 621 Bangladeshi adolescents (AgeMean±SD =16.44±1.32), of which 378 were males, and 244 were females. The participants completed the Bangla CS. A subset of the participants (n=160) also completed the Bangla Beck's Hopelessness Scale (BBHS)-a measure of hopelessness. Exploratory factor analysis on the first-round data (n=340) discarded six items and retained 17 items and revealed a unidimensional factor structure. Confirmatory Factor Analysis on the second-round data (n=281) supported the unidimensional structure (CFI=0.94, TLI=0.93). Measurement invariance analysis indicated that the unidimensional structure was robust across gender (143 males vs 139 females). The scale exhibited a negative correlation with BBHS revealing the scale's concurrent validity (r =-0.16, p <0.01). The scale exhibited satisfactory reliability (ωt = 0.79). The Item Response Theory-based analysis revealed that the scale was reliable (> 0.70) across a sizable range of communication skills continuum (θ = - 5.3 to 2.3) and had excellent marginal reliability (0.80). All items had adequate discriminating power (0.90 ± 0.20). The psychometric analysis of the 17-item Bangla-CS indicated that the scale is reliable and valid. We recommend that researchers and mental health practitioners utilize this scale to evaluate communication skills among Bangladeshi adolescents.

Development and psychometric testing of the perception of nursing profession instrument.

Perceptions of the nursing profession influence career choices in nursing. An unrealistic perception might lead students to drop out of nursing education programmes. Objective measurement of the nursing applicants' perceptions at the student selection stage could enhance their career choices in nursing. To develop and psychometrically evaluate the Perception of Nursing Profession Instrument (PNPI). Mixed method design. Two versions of the PNPI were developed during the years 2016-2022. The first version was based on documents describing the nursing profession and the second version was based on an integrative literature review, a focus groups study and a document analysis of descriptions of the nursing profession. The meta-ethnographic approach was used to synthesize the results and form a theoretical framework for developing the PNPI (60 items). Item content validity was evaluated by an expert panel of nurses (n = 7). The psychometric properties of the instrument were analysed using the item response theory approach. The development process resulted in the 40-item PNPI with the following subscales: the content of nursing work, the career in nursing, the nature of nursing work and the characteristics of a nurse. The psychometric analysis revealed unidimensionality and goodness of fit to the partial credit model; however, the item difficulty was not well matched with the participants' abilities. The PNPI is a novel instrument for objectively measuring perceptions of the nursing profession. For further development, item difficulty must be enhanced to improve the measurement accuracy of the nursing applicants' perceptions of the nursing profession. Perceptions of the nursing profession influence career choices, but there is a lack of objective assessment instruments that can be used in nursing student selection setting to measure the perception. The results of this study offer an instrument to measure perception, while also suggesting ideas for further development.

A Spanish translation of Zohar and Luria's safety climate scale and a test of measurement equivalence

The purpose of this study is to provide a Spanish-language version of Zohar and Luria’s (Zohar and Luria, 2005) commonly-used safety climate scale using a rigorous translation–back translation process. To refine the translated Spanish-version safety climate scale and ensure its content and face validity, we also conducted six cognitive interviews with Spanish-speaking workers. Both confirmatory factor analysis (CFA) and item response theory (IRT) approaches were adopted to test measurement equivalence to ensure the comparability of safety climate scores obtained from the English- and Spanish-language versions of safety climate measures. Data were collected from 12,148 frontline employees within four companies from multiple industries (e.g., construction, and oil and gas). Results from CFA indicated that, although item residuals were heterogeneous between the two versions of the safety climate scales, strong support was found for measurement equivalence with factorial invariance as well as invariant threshold for item rating. Results from IRT also supported strong measurement equivalence of the two versions of the safety climate scales. This study provides a much-needed safety-climate assessment tool for Spanish-speaking workers and paves the way for future research investigating factors related to safety climate for a growing Spanish workforce.

ANALYSIS OF ISLAMIC SUBJECT TEST QUALITY USING THE ITEM RESPONSE THEORY OF RASCH MODEL

The objective of this study is to use the Rasch Item Response Theory (IRT) approach to assess and characterize multiple choice exams and AKM Islamic Religious Education Subjects developed by students. This is a descriptive quantitative study that included 30 student replies to multiple choice test instruments and AKM with five possible solutions. The Rasch Model Item Response Theory (IRT) technique was used to analyze quantitative data using the QUEST program and the content validity instrument by constructing the Aiken's V formula. The study revealed that 13 items fit, 2 items did not fit, and all items were valid. The reliability coefficient on the item summary shows the "enough" criteria and the reliability coefficient on the estimation case summary shows the "weak" criteria, the item difficulty level shows 13 items in the good category and 2 items in the poor category, and the results of validation by experts processed by experts using the Aiken's V formula show that all items are classified as valid.

An item response theory approach to the measurement of working memory capacity.

Complex span tasks are perhaps the most widely used paradigm to measure working memory capacity (WMC). Researchers assume that all types of complex span tasks assess domain-general WM. However, most research supporting this claim comes from factor analysis approaches that do not examine task performance at the item level, thus not allowing comparison of the characteristics of verbal and spatial complex span tasks. Item response theory (IRT) can help determine the extent to which different complex span tasks assess domain-general WM. In the current study, spatial and verbal complex span tasks were examined using IRT. The results revealed differences between verbal and spatial tasks in terms of item difficulty and block difficulty, and showed that most subjects with below-average ability were able to answer most items correctly across all tasks. In line with previous research, the findings suggest that examining domain-general WM by using only one task might elicit skewed scores based on task domain. Further, visuospatial complex span tasks should be prioritized as a measure of WMC if resources are limited.

Constructing and validating a self-assessment scale for Chinese college English-major students’ feedback knowledge repertoire in EFL academic writing: Item response theory and factor analysis approaches

Constructing and validating a self-assessment scale for Chinese college English-major students’ feedback knowledge repertoire in EFL academic writing: Item response theory and factor analysis approaches