Item Response Theory Approach Research Articles

Learner Competence Across Cognitive Levels In Undergraduate Medical Education: An Item Response Approach.

Objective: The current study endeavors to evaluate competence of learners using item response theory approach in undergraduate medical education. Methodology: The responses of the examinees from first professional year (MBBS) were evaluated for Anatomy discipline to gauge learner competence through knowledge, understanding and application in the subject using item response theory approach (IRT) which is an effective tool to measure the validity (discrimination and difficulty index) of assessment, while KR-20 is used to gauge reliability of the items with statistical software STATA 17. The quality of question and level of examinee responses provide skill level and competence of learners. Results: The current study found that the C1 and C3 level items from the Anatomy discipline are mostly acceptable and effectively gauge learner competence while flawed C2 level items need complete restructuring. The study also identified weak areas of average learner for items in each cognitive level. Conclusion: The study found significant role of IRT in effectively estimating competence of the examinees across cognitive levels for each discipline and in building of an effective test pool through assessment of learner ability and construction of quality SBAs to achieve valid and reliable assessment tool. Keywords: Difficulty index; Discrimination index; Single best answer questions; Cognitive levels; Item characteristic curve.

Understanding Vietnamese chicken farmers' knowledge and practices related to antimicrobial resistance using an item response theory approach.

Antimicrobial resistance (AMR) poses a threat to animal and human health, as well as food security and nutrition. Development of AMR is accelerated by over- and misuse of antimicrobials as seen in many livestock systems, including poultry production. In Vietnam, high AMR levels have been reported previously within poultry production, a sector which is dominated by small-scale farming, even though it is intensifying. This study focuses on understanding small- and medium-scale chicken farmers' knowledge and practices related to AMR by applying an item response theory (IRT) approach, which has several advantages over simpler statistical methods. Farmers representing 305 farms in Thai Nguyen province were interviewed from November 2021 to January 2022, using a structured questionnaire. Results generated with IRT were used in regression models to find associations between farm characteristics, and knowledge and practice levels. Descriptive results showed that almost all farmers could buy veterinary drugs without prescription in the local community, that only one third of the farmers received veterinary professional advice or services, and that the majority of farmers gave antibiotics as a disease preventive measure. Regression analysis showed that multiple farm characteristics were significantly associated to farmers' knowledge and practice scores. The study highlights the complexity when tailoring interventions to move towards more medically rational antibiotic use at farms in a setting with high access to over-the-counter veterinary drugs and low access to veterinary services, since many on-farm factors relevant for the specific context need to be considered.

Optimizing Educational Assessment: The Practicality of Computer Adaptive Testing (CAT) with an Item Response Theory (IRT) Approach

This research aims to develop a Computer Adaptive Test (CAT) system using the Items Response Theory (IRT) approach. This study is part of developing a web-based system using the Research and Development (R&D) method, employing the Four-D (4-D) model. At its core, this system is similar to a Computer-Based Test (CBT). Still, the critical difference lies in its ability to randomize and provide questions that align with the test-taker's skill levels using the Items Response Theory (IRT) algorithm. The system employs the 3-PL model from the Items Response Theory, considering the difficulty level of questions, the discriminative power of questions, and the likelihood of guessing or interference in the questions. The examination system randomly assigns questions to students based on their responses to previous questions, ensuring that each test-taker receives a unique question sequence. The exam concludes when a test-taker accurately estimates their ability, i.e., SE <= 0.01, or when all questions have been answered. The outcome of this research is a Computer Adaptive Test (CAT) system based on the Items Response Theory (IRT), which can be used to assess students' learning outcomes. This research was implemented in the Multimedia Department of SMK Negeri 1 Gunung Talang, with 90 students as the research sample. The evaluation of the practicality of this system received very high scores, indicating that the Computer Adaptive Test (CAT) system based on the Items Response Theory (IRT) is considered highly practical and effective in achieving the established measurement goals.

HBMIRT: A SAS macro for estimating uni- and multidimensional 1- and 2-parameter item response models in small (and large!) samples

Item response theory (IRT) has evolved as a standard psychometric approach in recent years, in particular for test construction based on dichotomous (i.e., true/false) items. Unfortunately, large samples are typically needed for item refinement in unidimensional models and even more so in the multidimensional case. However, Bayesian IRT approaches with hierarchical priors have recently been shown to be promising for estimating even complex models in small samples. Still, it may be challenging for applied researchers to set up such IRT models in general purpose or specialized statistical computer programs. Therefore, we developed a user-friendly tool – a SAS macro called HBMIRT – that allows to estimate uni- and multidimensional IRT models with dichotomous items. We explain the capabilities and features of the macro and demonstrate the particular advantages of the implemented hierarchical priors in rather small samples over weakly informative priors and traditional maximum likelihood estimation with the help of a simulation study. The macro can also be used with the online version of SAS OnDemand for Academics that is freely accessible for academic researchers.

Conditional Dependence across Slow and Fast Item Responses: With a Latent Space Item Response Modeling Approach.

There recently have been many studies examining conditional dependence between response accuracy and response times in cognitive tests. While most previous research has focused on revealing a general pattern of conditional dependence for all respondents and items, it is plausible that the pattern may vary across respondents and items. In this paper, we attend to its potential heterogeneity and examine the item and person specificities involved in the conditional dependence between item responses and response times. To this end, we use a latent space item response theory (LSIRT) approach with an interaction map that visualizes conditional dependence in response data in the form of item-respondent interactions. We incorporate response time information into the interaction map by applying LSIRT models to slow and fast item responses. Through empirical illustrations with three cognitive test datasets, we confirm the presence and patterns of conditional dependence between item responses and response times, a result consistent with previous studies. Our results further illustrate the heterogeneity in the conditional dependence across respondents, which provides insights into understanding individuals' underlying item-solving processes in cognitive tests. Some practical implications of the results and the use of interaction maps in cognitive tests are discussed.

Psychometric properties of the FTCD and Brief WISDM: support for validity in a legal-system-involved sample.

The Fagerstrom Test of Cigarette Dependence (FTCD) and Brief Wisconsin Index of Smoking Dependence Motives (WISDM) are widely-used measures of smoking dependence. The FTCD was previously found to have 1- and 2-factor structures and Brief WISDM has been found to have an 11-factor and 11-factor hierarchical structure. As such, the current study sought to further investigate the psychometric properties of the FTCD and Brief WISDM with a novel criminal-legal system-involved sample using both a factor-analytic and an Item Response Theory (IRT) approach. Data from 517 criminal-legal system-involved adults (i.e., 18 years of age or older) who smoke from Alabama, USA were analyzed. Confirmatory factor analyses (CFA) were conducted on 1-factor and 2-factor structures of the FTCD and 1-factor, 11-factor, and 11-factor hierarchical structures of the Brief WISDM. IRT analyses investigating item discrimination and threshold parameters were also conducted on the Brief WISDM. The CFA showed poor fit for a single-factor structure, and mixed results for two 2-factor results for the FTCD. CFA also showed poor fit for a single-factor, and mixed results for the 11-factor model. Initial IRT investigations using the 11-factor model showed strong item discrimination, but non-ordered threshold parameters. Two-factor structures for the FTCD and the 11-factor model for the Brief WISDM were partially supported in a criminal-legal population, suggesting continued support for the multidimensional structure of the measures. Additionally, exploratory IRT analyses suggested good discrimination across the use spectrum for the Brief WISDM. The Fagerstrom Test of Cigarette Dependence (FTCD) and Brief Wisconsin Index of Smoking Dependence Motives WISDM are two widely used measures of nicotine dependence, though previous research has shown mixed results for their internal consistency and factor structure. The current study used a unique sample of criminal-legal-involved participants who generally have moderate to high levels of nicotine dependence. The current study found that the FTCD displayed poor internal consistency, poor fit for a single-factor model, but mixed support for two two-factor models. The Brief-WISDM was found to have strong internal consistency, poor fit for a single-factor model, but mixed fit for an 11-factor model and good item discrimination.

On the intricate relationship between data and theory, and the potential gain afforded by capturing very low levels of media trust: Commentary on Mangold (2024)

ABSTRACT In his paper ‘Improving media trust research through better measurement: An item response theory perspective', Frank Mangold (2024) adopts an item response theory approach to rethink and reconceptualise an existing measure of media trust, originally developed to distinguish perceptions of news media as (1) balanced, (2) fair, and (3) current. Applying an Item Response Theory approach, Mangold argues that the three factors do not capture different dimensions, but different ranges of media trust. In this commentary, we highlight the importance of capturing truly low levels of trust. We then turn to theoretical implications for trust research (e.g., discussing the theoretical relevance of the currency factor) and practical implications (e.g., discussing even more efficient measurement and quantifying the gain in measurement precision). We finally suggest some opportunities for future research that arise from Mangold's work.

Item Response Theory: A Modern Measurement Approach to Reliability and Precision for Counseling Researchers

The purpose of this article is to discuss reliability and precision through the lens of a modern measurement approach, item response theory (IRT). Reliability evidence in the field of counseling is primarily generated using Classical Test Theory (CTT) approaches, although recent studies in the field of counseling have shown the benefits of using IRT approaches to explore measurement precision. We discuss the theoretical foundations and assumptions of CTT and IRT, and examine how modern measurement theory (i.e. IRT) poses advantages for capturing measurement precision. We use an example analysis to demonstrate the indices of measurement precision in IRT approaches (e.g. standard errors, person-fit, item-fit, targeting, residual-based fit statistics). Finally, we discuss practical and clinical implications for counseling researchers for using IRT approaches to measure precision, including insights into precision for persons, items, and measurement invariance as well as the utility of brief and adaptive scales.

Psychometric Properties of the Utrecht Work Engagement Scale-9 in a School Counselor Sample Using Classical Test Theory and Item Response Theory

The purpose of this study is to explore the psychometric properties of the Utrecht Work Engagement Scale-9 (UWES-9) using both Classical Test Theory (CTT) and Item Response Theory (IRT) approaches, with a sample of practicing school counselors (N = 356). The results of analyzing the UWES-9 included information on item-level statistics, internal consistency, structural validity, and concurrent validity. IRT findings based on the Graded Response Model (GRM) indicated the usefulness of the UWES-9 in measuring school counselors’ work engagement.

Development and Validation of a Multidimensional Perceived Media Convergence Scale: A Classical Test Theory and Item Response Theory Approach

Development and Validation of a Multidimensional Perceived Media Convergence Scale: A Classical Test Theory and Item Response Theory Approach

Trajectories of healthy aging and their association with prodromal parkinson disease: The HELIAD study

Background: The aim of the present study was to investigate the association of prodromal PD (pPD) with trajectories of healthy aging, according to its latest definition by the WHO. Methods: In a sample of 1,226 older adults (704 women), PD diagnosis was reached through standard clinical research procedures. Probability of pPD was calculated according to the International Parkinson and Movement Disorder Society’s research criteria for PD-free participants. A healthy aging metric was introduced using an item response theory approach (IRT) based on information from validated questionnaires assessing functionality. Four trajectories of healthy aging were created based on whether the healthy aging status of participants was above or below the median at baseline and follow up: High-High, High-Low, Low-High and Low-Low. Results: 34.3% belonged to the High-High group, 15.7% to the High-Low, 18.6% to the Low-High and 31.4% to the Low-Low group. Participants with possible/probable pPD were 78% less likely to belong in High-High trajectory of healthy aging as compared to those without pPD (OR = 0.22, 95%CI 0.06-0.79, p-value = 0,02). Conclusion: Our findings suggest an inverse association of pPD probability with healthy aging among older adults; Further research is needed to investigate the clinical implications of this association.

Common measures or common metrics? the value of IRT-based common metrics

There is a clear need to harmonize outcome measurement. Some authors propose to express scores as T scores to facilitate interpretation of PROM results in clinical practice. While this is a step in the right direction, there are important limitations to the acceptance of the T score metric as a common metric when T scores are based on raw sum scores of ordinal items: Such T scores of different instruments are not exactly comparable because they are not interval scaled; T scores of different measures are only on the same scale if exactly the same reference group is used; and the T sore metric cannot be maintained because it is reference population-dependent and needs to be updated regularly. These limitations can be overcome by using an item response theory (IRT)-based metric. Items from different measures can be placed on the same IRT metric to make scores comparable on an interval scale. The PROMIS initiative used IRT to develop item banks for measuring various health outcomes. Other PROMs have been linked to the PROMIS metric. Although PROMIS uses a T-score metric for practical reasons, the underlying PROMIS metric is actually an IRT metric. An IRT approach also enables further development of an item bank while preserving the underlying metric. Therefore, IRT-based metrics should be considered as common metrics for the future.

Performance Evaluation of Oral Health Teams in Brazil: An Item Response Theory Approach

Access to quality healthcare is an essential component of health policy. This cross-sectional study describes the actions performed by Brazilian Oral Health Teams (OHTs) analyzed in the National Program for Improving Access and Quality of Primary Care and the relationship of contextual aspects. A total of 22,993 OHTs that participated in the third cycle of the referred program were evaluated using a structured questionnaire. Thirteen procedures (items) related to spontaneous dental care and preventive, surgical, restorative, prosthetic, and oral cancer prevention or diagnosis were assessed. Item response theory was used to estimate the performance scores of OHTs, based on 13 items. The relationship between performance scores and contextual variables in Brazilian regions was analyzed by thematic maps and Spearman correlation (p < 0.05). The highest difficulty parameters were for questions related to prosthetics (b = 0.879) and actions for oral cancer (b = 2.922). The 13 items were more appropriate to discriminate the teams with lower performance and relatively ineffective in differentiating those with better performance. A direct relationship with the Human Development Index (rs = 0.249; p = 0.004) and an indirect relationship with the Gini Index (rs = −0.482; p < 0.001) were found. Contextual aspects presented a relationship with the performance of the Brazilian OHTs. The evaluated items showed some potential to discriminate the performance of OHTs since many of the evaluated dental procedures are performed by most of the teams.

Numeric rating scale for pain should be used in an ordinal but not interval manner. A retrospective analysis of 346,892 patient reports of the quality improvement in postoperative pain treatment registry.

To assess postoperative pain intensity in adults, the numeric rating scale (NRS) is used. This scale has shown acceptable psychometric features, although its scale properties need further examination. We aimed to evaluate scale properties of the NRS using an item response theory (IRT) approach. Data from an international postoperative pain registry (QUIPS) was analyzed retrospectively. Overall, 346,892 adult patients (age groups: 18-20 years: 1.6%, 21-30 years: 6.7%, 31-40 years: 8.3%, 41-50 years: 13.2%, 51-60 years: 17.1%, 61-70 years: 17.3%, 71-80 years: 16.4%, 81-90 years: 3.9%, >90: 0.2%) were included. Among the patients, 55.7% are female and 38% had preoperative pain. Three pain items (movement pain, worst pain, least pain) were analyzed using 4 different IRT models: partial credit model (PCM), generalized partial credit model (GPCM), rating scale model (RSM), and graded response model (GRM). Fit indices were compared to decide the best fitting model (lower fit indices indicate a better model fit). Subgroup analyses were done for sex and age groups. After collapsing the highest and the second highest response category, the GRM outperformed other models (lowest Bayesian information criterion) in all subgroups. Overlapping categories were found in category boundary curves for worst and minimum pain and particularly for higher pain ratings. Response category widths differed depending on pain intensity. For female, male, and age groups, similar results were obtained. Response categories on the NRS are ordered but have different widths. The interval scale properties of the NRS should be questioned. In dealing with missing linearity in pain intensity ratings using the NRS, IRT methods may be helpful.

Day Scholars Food Insecurity Experience Scale-Survey Module (DSFIES-SM): Psychometric Analysis

Background: School feeding programs’ evaluation requires the measurement of food insecurity, a more objective indicator, within school in low-income countries. The Global Child Nutrition Foundation (GCNF) uses subjective indicators to report school feeding coverage rates across many countries that participate in the global survey of school meal programs all year round. Aim: To test the methodological feasibility of measuring a school food insecurity construct as a direct indicator of the effectiveness and efficiency of school feeding programs. Methods: Two-stage sampling was used during the selection of schools and 128 schoolchildren with a mean ( SD) age of 10.5(1.58) were recruited. Item Response Theory (IRT) and Classical Test Theory (CTT) approaches were utilized during DSFIES-SM development. Psychometric analysis was utilized to assess the psychometric properties of the measure of school food insecurity and to establish the construct-level reliability, convergent and discriminant validity of the DSFIES-SM. Results: DSFIES-SM generated acceptable item-level reliabilities, ranging from .75 to .78. The construct-level reliability of the DSFIES-SM was indicated by Cronbach’s α of .78. Composite reliability was at .77. Fit measures and tests of model fitness for confirmatory factor analysis (CFA) (RMSEA < 0.0001; TLI = 1.06; SRMR = 0.043; CFI = 1.00, p = .85) confirmed that the data fitted the model perfectly. Conclusion: DSFIES-SM consists of twelve questions with dichotomous yes/no responses. DSFIES-SM has good convergent and discriminant validity. Findings suggest that the items compose a statistical scale designed to cover a range of severity of school food insecurity. Future replications to establish other forms of validity across different cultural contexts in low-income countries can be of benefit to the present research.

Standard Assessment Model for Scientific Integration-Interconnection as Material for Evaluation of Vision and Mission Achievement

This study aims to develop a standardized assessment model of scientific integration-interconnection, which begins with conducting a literature review and needs assessment, setting goals, developing initial products, validating contents, testing readability, testing instruments, analyzing results, and ending with an interpretation of the results of the assessment. The method used is Research and Development. Testing of the assessment instruments was carried out for each element of the Tridharma of Higher Education, namely 76 lesson plans, 139 research and development works, and 41 community service works. The scientific integration-interconnection assessment model has characteristics according to the structure of the second order model, with the first order being 4 scientific integration-interconnection domains (philosophical domain, material domain, methodological domain, and strategic domain) and the second order is 15 aspects of scientific integration-interconnection. Data analysis techniques using Aiken-V, Exploratory Factor Analysis/EFA and Confirmatory Factor Analysis/CFA, Omega, Item Response Theory approach of polytomous scoring, and descriptive statistics. The assessment model is feasible, fulfilling the criteria of content validity, construct validity (p-value = 0.11949 and RMSEA = 0.027), estimated instrument reliability (α = 0.816), and construct reliability (ɷ =0.939). The scoring used the GRM approach with good quality in terms of the results of the 94% item fit test with an item discrimination index of 0.655 < DISCi < 1.153 and the average item difficulty level of -1.965 < DIFFi < -0.662.

Item response theory approach to ethnocentrism

IntroductionAlthough ethnocentrism is one of the fundamental concepts in the social sciences, its study has been impeded by a diversity of conceptualizations and measures. In recent years, a growing number of political scientists and psychologists have undertaken in-depth research into ethnocentrism. In addition, researchers have recently proposed a comprehensive reconceptualization of ethnocentrism and developed a new Ethnocentrism scale. There is strong evidence for this scale's reliability and validity in indexing ethnocentrism, but like most measures in psychology and political science, this scale is based on classical test theory. Item response theory (IRT) is a powerful psychometric technique that can provide a much more sophisticated test of test performance and is currently under-utilized in research.MethodsWe performed IRT to assess the psychometric properties of the Ethnocentrism scale on a sample of 4,187 participants.ResultsThe scale's items had strong psychometric properties to capture the ethnocentrism latent construct, particularly in the below average to above average range. Men required marginally lower levels of ethnocentrism to endorse less socially acceptable items than women (items relating to superiority, purity, or exploitativeness). When compared to liberals, conservatives responded more readily to nearly all ethnocentrism items. Given this variation, the IRT approach highlighted that future measurements must adjust for differential item functioning, albeit more for political orientation than gender identity.DiscussionThe findings detail how IRT can enhance measurement in political science and demonstrate the implications for how gender and political ideology may affect the differential performance of items.

“Use the Force!” Adaptation of Response Formats

Abstract. The present paper features the adaptation of an existing Big Five questionnaire with a rating scale (RS) response format into a measure using a multidimensional forced choice (MFC) response format. Rating scale response formats have been criticized for their proneness to intentional and unintentional response distortions. Multidimensional forced choice response formats were suggested as a solution to mitigate several types of response sets and response styles by design. The Big Five Inventory of Personality in Occupational Situations (B5PS) is a situation-based questionnaire designed for personnel selection and development purposes which would benefit from fake-proof response formats. MFC response formats require special effort during test construction and calibration which will be laid out here. Changing the response format has severe consequences on item design and scoring. An inherent issue with MFC formats derives from their inability to yield interpersonal comparative results from standard (sum) scoring. This issue can be solved with item response theory (IRT)-based calibration during test construction. The Thurstonian IRT approach (TIRT) was developed by Brown and Maydeu-Olivares (2011) , and aspects of MFC item design and TIRT calibrations are explored in this paper. Evidence on structural and construct validity are presented alongside recommendations on the test development processes. The results support the feasibility of the concept of MFC test construction with TIRT calibration in a contextualized and situation-based item format.

Measuring skill-based health literacy in chronic airway disease patients: the development and psychometric evaluation of the Vancouver airways health literacy tool (VAHLT).

This article describes the development of the Vancouver airways health literacy tool (VAHLT), a novel measure of skill-based health literacy specific to chronic airway diseases (CADs). Across several phases, psychometric characteristics of the VAHLT were examined and used to guide its development. An initial pool of 46 items was developed using input from patients, clinicians, researchers, and policy-makers. An initial patient sample (N = 532) was evaluated and used to inform item revisions. A revised 44-item pool was then evaluated using a second sample, the results of which aided in the selection of a final set of 30 items. The finalized 30-item VAHLT was then psychometrically evaluated using the second sample (N = 318). An item response theory approach was utilized to evaluate the VAHLT by assessing model fit, item parameter estimates, test and item information curves, and item characteristic curves. Reliability was assessed using ordinal coefficient alpha. We additionally assessed differential item functioning between asthma and COPD diagnoses. The VAHLT demonstrated a unidimensional structure and reasonably discriminated patients in the lower range of health literacy estimates. The tool demonstrated strong reliability (α = .920). Two of the 30 items were found to exhibit non-negligible differential item functioning. This study presents compelling evidence of validity in several areas for the VAHLT, including content and structural validity. Further external validation studies are needed and forthcoming. Overall, this work represents a strong first step towards a novel, skill-based, and disease-specific measure of CAD-related health literacy.

The development and validation of the Chinese safety climate scale using the item-response theory approach.

To develop a valid and practical tool to measure the safety climate in China, and further raise awareness of it in Chinese industries, we developed the Chinese safety climate scale (including two subscales at the levels of organization and group separately) based on the work of Huang et al. in 2017. A descriptive survey with the convenience sampling method was conducted in Shanghai Disney Resort, China. A sample of 1,570 employees was involved in the final data analyses. The item response theory (IRT) analyses with graded response models were conducted using. The unidimensionality and local independence assumption were held. The Cronbach's α of organization- and group- level safety climate subscale was 0.912 and 0.937, respectively. The confirmatory factor analysis (CFA) showed good model fit for a one-factor model of the organization-level safety climate subscale, χ2 (df = 20) = 129.158, p < 0.001, CFI = 0.994, TLI = 0.992, NFI = 0.993, IFI = 0.994, RMSEA = 0.059, 90% CI = (0.050, 0.069), and SRMR = 0.048. A one-factor model also fits well for the group-level safety climate subscale, χ2 (df = 44) = 219.727, p < 0.001, CFI = 0.996, TLI = 0.9925, NFI = 0.995, IFI = 0.996, RMSEA = 0.050, 90% CI = (0.044, 0.057), and SRMR = 0.046. Discrimination and difficulty parameters showed that all items effectively spanned the range of the latent trait and could successfully separate participants at different safety climate levels. Items in the organization-level and group-level Chinese safety climate subscales had significantly different discrimination parameters, fitted well with the models, and had a substantive relationship with the latent traits. The Chinese safety climate scale was reliable and valid overall. They can facilitate the research and survey regarding the safety climate in China.