Multidimensional Item Response Theory Research Articles

Investigating Subscores of VERA 3 German Test Based on Item Response Theory/Multidimensional Item Response Theory Models

In this study, the psychometric properties of the listening and reading subtests of the German VERA 3 test were examined using Item Response Theory (IRT) and Multidimensional Item Response Theory (MIRT) models. Listening and reading subscores were estimated using unidimensional Rasch, 1PL, and 2PL models, and total scores on the German test (listening + reading) were estimated using unidimensional and multidimensional IRT models. Various MIRT models were used, and model fit was compared in a cross-validation study. The results of the study showed that unidimensional models of the reading and listening subtests and the German test provided a good overall model-data fit, however, multidimensional models of the subtests provided a better fit. The results demonstrated that, although the subtest scores also fit adequately independently, estimating the scores of the overall test with a model (e.g., bifactor) that includes a general factor (construct) in addition to the subfactors significantly improved the psychometric properties of the test. A general factor was identified that had the highest reliability values; however, the reliabilities of the specific factors were very low. In addition to the fit of the model data, the fit of the persons with IRT/MIRT models was also examined. The results showed that the proportion of person misfit was higher for the subtests than for the overall tests, but the overfit was lower. NA-German students, who did not speak German all-day, had the highest proportion of misfits with all models.

Avoiding bias from sum scores in growth estimates: An examination of IRT-based approaches to scoring longitudinal survey responses.

A huge portion of what we know about how humans develop, learn, behave, and interact is based on survey data. Researchers use longitudinal growth modeling to understand the development of students on psychological and social-emotional learning constructs across elementary and middle school. In these designs, students are typically administered a consistent set of self-report survey items across multiple school years, and growth is measured either based on sum scores or scale scores produced based on item response theory (IRT) methods. Although there is great deal of guidance on scaling and linking IRT-based large-scale educational assessment to facilitate the estimation of examinee growth, little of this expertise is brought to bear in the scaling of psychological and social-emotional constructs. Through a series of simulation and empirical studies, we produce scores in a single-cohort repeated measure design using sum scores as well as multiple IRT approaches and compare the recovery of growth estimates from longitudinal growth models using each set of scores. Results indicate that using scores from multidimensional IRT approaches that account for latent variable covariances over time in growth models leads to better recovery of growth parameters relative to models using sum scores and other IRT approaches. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Development of a computerized adaptive substance use disorder scale for screening, measurement and diagnosis – The CAT-SUD-E

IntroductionThe Computerized Adaptive Test for Substance Use Disorder (CAT-SUD), an adaptive test based on multidimensional item response theory, has been expanded to include 7 specific Diagnostic and Statistical Manual, 5th edition (DSM-5) defined SUDs. Initial testing of the new measure, the CAT-SUD expanded (CAT-SUD-E) is reported here. Methods275 Community-dwelling adults (ages 18–68) responded to public and social-media advertisements. Participants virtually completed both the CAT-SUD-E and the Structured Clinical Interview for DSM-5, Research Version (SCID) to assess the validity of the CAT-SUD-E in determining whether participants met criteria for specific DSM-5 SUDs. Diagnostic classifications were based on 7 SUDs, each with 5 items, for current and lifetime SUDs. ResultsFor SCID-based presence of any lifetime SUD, predictions based on the overall CAT-SUD-E diagnosis and severity score were AUC=0.92, 95% CI = 0.88, 0.95 for current and AUC=0.94, 95% CI = 0.91, 0.97 for lifetime. For individual diagnoses, classification accuracy for current SUDs ranged from an AUC=0.76 for alcohol to AUC=0.92 for nicotine/tobacco. Classification accuracy for lifetime SUDs ranged from an AUC=0.81 for hallucinogens to AUC=0.96 for stimulants. Median CAT-SUD-E completion time was under 4 min. ConclusionsThe CAT-SUD-E quickly produces similar results as lengthy structured clinical interviews for overall SUD and substance-specific SUDs, with high precision and accuracy, through a combination of fixed-item responses for diagnostic classification and adaptive SUD severity measurement. The CAT-SUD-E harmonizes information from mental health, trauma, social support and traditional SUD items to provide a more complete characterization of SUD and provides both diagnostic classification and severity measurement.

Patient-Reported Outcome Measures: An Exploration of Their Utility in Functional Assessment and Rehabilitation

Patient reported outcome measures (PROMs) have had a place in medicine for many years. However, their importance has grown rapidly in recent years due to a number of factors-not the least of which is an increased recognition of importance of the patient' assessment of their care and outcomes. Despite their growing use, few of us have a good understanding of when they should be favored over alternative measures, how they are developed, the benefits of computerized administration, or how to leverage their incorporation into the electronic health record (EHR) and clinical decision making. This supplement is not intended to cover all aspects of the use of PROMs in clinical and rehabilitation care. Rather it's goal is to provide an overview of their use and development, as well as the clinical relevance of their increasingly electronic, computerized, and EHR-based administration. It's papers also highlight how advances in computing capabilities have permitted the use of newer modeling capabilities, such as multidimensional item response theory, to radically expand the capabilities of computerized PROMs.

Multidimensional Computerized Adaptive Testing Simulations in R

Computerized Adaptive Testing (CAT) is a beneficial test technique that decreases the number of items that need to be administered by taking items in accordance with individuals' own ability levels. After the CAT applications were constructed based on the unidimensional Item Response Theory (IRT), Multidimensional CAT (MCAT) applications have gained momentum with the improvement of multidimensional IRT (MIRT) models in recent years. Researchers often benefit from simulation studies in order to design the final adaptive testing application and to test the effectiveness of adaptive testing applications they developed with different methods. Recently, R has become one of the most widely used programming languages in Monte Carlo Simulation studies since it is a free and open-source software. The aims of this study are to present the MCAT simulation process step by step in the R environment, to examine the effects of the conditions that researchers can handle during the simulation process according to two different dimensional models, and to examine the effect of treating multidimensional structures as unidimensional structures on simulation results. In this direction, datasets generated in accordance with within-item dimensionality and between-item dimensionality models, MCAT simulation studies were constructed with different customizations, and MCAT simulation results were compared with unidimensional CAT simulation results. All commands required for each simulation example were explained and results were shared for each condition.

Psychometric assessment of scales for measuring loneliness and social isolation: an analysis of the household, income and labour dynamics in Australia (HILDA) survey

BackgroundLoneliness and social isolation are increasingly recognised as global public health threats, meaning that reliable and valid measures are needed to monitor these conditions at a population level. We aimed to determine if robust and practical scales could be derived for conditions such as loneliness and social isolation using items from a national survey.MethodsWe conducted psychometric analyses of ten items in two waves of the Household, Income and Labour Dynamics in Australia Survey, which included over 15,000 participants. We used the Hull method, exploratory structural equation modelling, and multidimensional item response theory analysis in a calibration sample to determine the number of factors and items within each factor. We cross-validated the factor structure using confirmatory factor analysis in a validation sample. We assessed construct validity by comparing the resulting sub-scales with measures for psychological distress and mental well-being.ResultsCalibration and cross-validation consistently revealed a three-factor model, with sub-scales reflecting constructs of loneliness and social isolation. Sub-scales showed high reliability and measurement invariance across waves, gender, and age. Construct validity was supported by significant correlations between the sub-scales and measures of psychological distress and mental health. Individuals who met threshold criteria for loneliness and social isolation had consistently greater odds of being psychologically distressed and having poor mental health than those who did not.ConclusionsThese derived scales provide robust and practical measures of loneliness and social isolation for population-based research.

Psychometric approaches to analyzing C-tests

The C-test is a type of gap-filling test designed to efficiently measure second language proficiency. The typical C-test consists of several short paragraphs with the second half of every second word deleted. The words with deleted parts are considered as items nested within the corresponding paragraph. Given this testlet structure, it is commonly taken for granted that the C-test design may violate the local independence assumption. However, this assumption has not been fully investigated in the C-test research to date, including the evaluation of alternative psychometric models (i.e., unidimensional and multidimensional) to calibrate and score the C-test. This study addressed each of these issues using a large data set of responses to an English-language C-test. First, we examined the local item independence assumption via multidimensional item response theory (IRT) models, Yen’s Q3, and Jackknife Slope Index. Second, we evaluated several IRT models to determine optimal approaches to scoring the C-test. The results support an interpretation of unidimensionality for the C-test items within a paragraph, with only minor evidence of local item dependence. Furthermore, the two-parameter logistic (2PL) IRT model was found to be the most appropriate model for calibrating and scoring the C-test. Implications for designing, scoring, and analyzing C-tests are discussed.

An Examination of the Latent Constructs in Risk Tools for Individuals Who Sexually Offend: Applying Multidimensional Item Response Theory to the Static-2002R.

Multidimensional item response theory (MIRT) was used to study the construct validity of the Static-2002R, an actuarial scale for the assessment of reoffending among adult men who sexually offended. Using a sample of 2,569 individuals with a history of sexual crime, exploratory factor analysis (EFA) extracted three factors: Persistence/Paraphilia, General Criminality, and Youthful Stranger Aggression. MIRT confirmed the factor structure identified in the EFA model and provided item-level data on discrimination and difficulty. All Static-2002R items showed moderate to very high discrimination and covered a wide range of risk levels (i.e., difficulty). MIRT analyses attested to the construct validity of the scale, as no items were identified as problematic and the resulting factor structure was consistent with that of earlier studies. Considering the stability of results pertaining to the factor structure of the Static-2002R and the advantages of dimensional scoring, we recommend the integration of dimensional scores in the scale.

Assessment of German and Austrian students' Educational Research Literacy: validation of a competency test based on cross-national comparisons

Educational Research Literacy (ERL) is the ability to access, comprehend, and consider scientific information and to apply the resulting conclusions to problems connected with educational decisions. It is crucial for the process of data-based decision making and–corresponding to the consecutive phases–defined as the conglomeration of different facets of competence, including information literacy, statistical literacy, and evidencebased reasoning. However, the engagement with research in educational contexts appears to have some difficulties. This is even more remarkable as the state of knowledge about actual teacher competency levels remains unsatisfactory, even though test instruments for assessing research literacy have been developed in recent years. This paper addresses the question of whether such a test developed in the specific context of German study programs in (teacher) education can be applied to other national contexts, in this case to Austrian teacher education. An investigation of the construct validity under consideration of the psychometric structure and group differences on item level is necessary for ensuring the fairness of cross-national comparisons. Based on multidimensional item response theory models, samples from Germany (n = 1360 students, 6 universities) and Austria (n = 295 students, 2 universities) are investigated in terms of measurement invariance between the two countries. A comparable psychometric structure and at least partial measurement invariance with no particular advantage for either sample could be demonstrated. This is an indication that the presented test instrument can be validly applied to assess the research literacy of teacher training students in both countries.

Validity and reliability evidence for the scale of distance education satisfaction of medical students based on item response theory (IRT)

BackgroundThe objective of this study is to reveal on the basis of the item response theory (IRT) the validity and reliability evidence for the data obtained from the scale prepared to determine the satisfaction with distance education in students studying in medical schools.MethodsThis is a quantitative study exploring IRT and measurement invariance evidence in developing a scale. The scale whose IRT evidence was explored was the Distance Education Satisfaction Scale (DESS). The data were obtained from 1332 medical school students who were studying at various universities. The data were analysed using the confirmatory factor analysis (CFA), multidimensional and unidimensional IRT, and measurement invariance.ResultsA 20-item construct with 3 sub-factors was found for DESS. This construct was unable to pass the iteration limit in the multidimensional IRT analysis. A unidimensional IRT was used assuming that the 3 sub-factors were locally independent.ConclusionsThe least informative items were item 23, 24 and 25 in Factor 1, item 3 in Factor 2, and items 13 and 18 in Factor 3. The most informative items of DESS were those that had adaptive, useful expressions that had meaningful content and were able to provide educator support, which are the properties emphasized in the literature with respect to satisfaction with distance education. A measurement invariance test made based on gender revealed that DESS satisfied measurement invariance by meeting the compliance indexes required for configural, metric, scalar and strict invariance as recommended in the literature. The results showed that it is possible to make comparisons on the basis of gender using DESS.

Evaluating Stan’s Variational Bayes Algorithm for Estimating Multidimensional IRT Models

Bayesian estimation of multidimensional item response theory (IRT) models in large data sets may come with impractical computational burdens when general-purpose Markov chain Monte Carlo (MCMC) samplers are employed. Variational Bayes (VB)—a method for approximating the posterior distribution—poses a potential remedy. Stan’s general-purpose VB algorithms have drastically improved the accessibility of VB methods for a wide psychometric audience. Using marginal maximum likelihood (MML) and MCMC as benchmarks, the present simulation study investigates the utility of Stan’s built-in VB function for estimating multidimensional IRT models with between-item dimensionality. VB yielded a marked speed-up in comparison to MCMC, but did not generally outperform MML in terms of run time. VB estimates were trustworthy only for item difficulties, while bias in item discriminations depended on the model’s dimensionality. Under realistic conditions of non-zero correlations between dimensions, VB correlation estimates were subject to severe bias. The practical relevance of performance differences is illustrated with data from PISA 2018. We conclude that in its current form, Stan’s built-in VB algorithm does not pose a viable alternative for estimating multidimensional IRT models.

Multidimensional item Response theory to assess the psychometric properties of persian version of dutch eating behavior questionnaire (DEBQ) in university students

Multidimensional item Response theory to assess the psychometric properties of persian version of dutch eating behavior questionnaire (DEBQ) in university students

Reintroducing the PVCT: A revised measure of the perceived value of certification among registered nurses

Background and PurposeThe Perceived Value of Certification Tool (PVCT) was developed in 2003 to measure nurses’ perceived value of certification. We redrafted the original PVCT to correct inherent agreeableness of the items and improve measurement model fit. MethodsWe used exploratory factor analysis, unidimensional and multidimensional item response theory, and reliability procedures to find the best-fitting dimensional structure, compare the original version with the revised version, and compare the perceived value of certification among nurses from nine nursing organizations using the redrafted PVCT. ResultsWe found the new PVCT was less agreeable and demonstrated improved measurement model fit. We found significant differences in nurses’ perceptions of the intrinsic and extrinsic value of certification. ConclusionThe new PVCT provides a reliable and valid instrument for determining the perceived value of nursing specialty certification.

Longitudinal measurement invariance of the Behavioral Health Measure in a clinical sample.

The practice of routine outcome monitoring (ROM) has grown in popularity and become a fixture in feedback-supported clinical practice and research. However, if the interpretation of an ROM measure changes over time, treatment outcome scores may be inaccurate and produce erroneous or misguided interpretations of client progress and therapist efficacy. The current study examined whether factorial invariance held when using the Behavioral Health Measure (BHM-20) longitudinally in a clinical sample (n = 12,467). Using multidimensional item response theory-based models for the investigation of the BHM-20 factor structure, at a single time point and then longitudinally. Based on the original factor structure of the BHM-20 a unidimensional model, a three-factor orthogonal model, and a three-factor correlated model were fit to the data, indicating poor model fit with the proposed three-factor or unidimensional models. Next, using exploratory factor analysis and subsequent multidimensional item response theory procedures, a new 4-factor (General Distress, Life Functioning, Anxiety, and Alcohol/Drug Use) model was proposed with improved model-fit statistics. Finally, when testing the longitudinal invariance of the BHM-17 over 10 sessions of treatment, it was found to be fully consistent. The current study proposes the use of a 17-item, 4-factor model for a new understanding of the BHM-17. Implications for use in ROM and limitations are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Higher-order thinking test of science for college students using multidimensional item response theory analysis

Higher-order thinking test of science for college students using multidimensional item response theory analysis

Latent variable sdelection in multidimensional item response theory models using the expectation model selection algorithm.

The aim of latent variable selection in multidimensional item response theory (MIRT) models is to identify latent traits probed by test items of a multidimensional test. In this paper the expectation model selection (EMS) algorithm proposed by Jiang etal. (2015) is applied to minimize the Bayesian information criterion (BIC) for latent variable selection in MIRT models with a known number of latent traits. Under mild assumptions, we prove the numerical convergence of the EMS algorithm for model selection by minimizing the BIC of observed data in the presence of missing data. For the identification of MIRT models, we assume that the variances of all latent traits are unity and each latent trait has an item that is only related to it. Under this identifiability assumption, the convergence of the EMS algorithm for latent variable selection in the multidimensional two-parameter logistic (M2PL) models can be verified. We give an efficient implementation of the EMS for the M2PL models. Simulation studies show that the EMS outperforms the EM-based L1 regularization in terms of correctly selected latent variables and computation time. The EMS algorithm is applied to a real data set related to the Eysenck Personality Questionnaire.

Application of Bi-factor MIRT and Higher-order CDM Models to an In-house EFL Listening Test for Diagnostic Purposes

ABSTRACT The present study examined the performance of the bi-factor multidimensional item response theory (MIRT) model and higher-order (HO) cognitive diagnostic models (CDM) in providing diagnostic information and general ability estimation simultaneously in a listening test. The data used were 1,611 examinees’ item-level responses to an in-house EFL listening test in China and five content experts’ item-attribute coding results of the test form. The bi-factor MIRT model was compared with five CDMs with and without a higher-order structure in terms of model fit, attribute classification and general ability estimation. The results showed that the bi-factor MIRT model provided the best model-data fit, followed by the HO-G-DINA model, the saturated G-DINA model, and other reduced CDMs. The HO-G-DINA model produced attribute classification results more similar to the G-DINA model, whereas the bi-factor MIRT model offered better results in discriminating examinees’ general listening ability. The findings of this study highlighted the feasibility of using the bi-factor MIRT model as an attractive alternative for diagnostic assessment, especially in language assessment where attributes are assumed to be continuous.

The Automated Test Assembly and Routing Rule for Multistage Adaptive Testing with Multidimensional Item Response Theory

AbstractDesigning a multidimensional adaptive test (M‐MST) based on a multidimensional item response theory (MIRT) model is critical to make full use of the advantages of both MST and MIRT in implementing multidimensional assessments. This study proposed two types of automated test assembly (ATA) algorithms and one set of routing rules that can facilitate the development of an M‐MST. Different M‐MST designs were developed based on the proposed ATA algorithms and routing rules and were evaluated through two sets of simulation studies. Study 1 used simulated item banks and considered a variety of testing factors, and results from which can inform us the theoretical performance of the proposed M‐MST designs. Study 2 was designed based on a real multidimensional assessment. In addition, a MCAT was simulated as a baseline design to demonstrate the advantage of the proposed M‐MST design in each study. Our simulation results indicate that the proposed ATA algorithms and routing rule can generate M‐MSTs with a good‐quality control and the same or even better ability estimation results than the MCAT given the same condition. This demonstrates the advantage of using M‐MST for multidimensional assessment especially for the one that needs to satisfy many nonstatistical constraints.

Exercise training improves cognition in chronic stroke: A 6‐month randomized controlled trial

AbstractBackgroundA stroke doubles one’s risk for dementia. Impairment in multiple domains of cognition are common following stroke. Thus, stroke survivors represent a target population in need of intervention strategies to promote cognitive function and prevent dementia. We assessed whether targeted exercise training or cognitive and social enrichment activities would promote cognitive function in community‐dwelling adults with chronic stroke (i.e., > 12 months since stroke event).MethodsA three‐arm parallel, single‐blinded, 6‐month randomized controlled trial with 120 adults with chronic stroke who were randomized (2:2:3) to: 1) targeted exercise training (EX; n=34)) modeled after the Fitness And Mobility Exercise (FAME) program (Eng, Top Geriatr Rehabil, 2010); 2) cognitive and social enrichment activities (Cog‐Plus; n=34); and 3) an active control group (CON; n=52). Each experimental arm was group‐based and 2x/week. The primary outcome was the ADAS‐Cog‐Plus, a global measure of cognitive performance using multidimensional item response theory to summarize scores from the 13‐item ADAS‐Cog and other standard cognitive assessments (i.e., Trail Making Test Parts A and B, Digit Span Forward and Backward, and Animal and Vegetable Fluency). Lower ADAS‐Cog‐Plus scores represent better cognitive performance; ADAS‐Cog‐Plus scores of approximately −1.0 indicate healthy cognitive performance, 0.0 indicate mild cognitive impairment, and 1.0 indicate dementia. The primary analysis using linear mixed model compared changes in ADAS‐Cog‐Plus performance from baseline to month 6, with baseline ADAS‐Cog‐Plus and Mini‐Mental State Examination scores as covariates. Two planned simple contrasts (i.e., EX vs. CON; Cog‐Plus vs. CON) were performed using the Dunnett test.ResultMean baseline Fugl‐Meyer score was 81.21 (SD=23.85; range: 6‐100) and mean baseline Montreal Cognitive Assessment score was 21.89 points out of 30. An overall attrition rate of 14% was observed (EX=6; Cog‐Plus=5; CON=6). Participants in the EX group had significantly better ADAS‐Cog Plus performance than CON at trial completion (estimated mean difference: ‐0.23; 95%CI:[‐0.43, ‐0.03]; p=.024). There was no significant difference in ADAS‐Cog Plus performance between the Cog‐Plus and CON group at trial completion (estimated mean difference: ‐0.10; 95% CI:[‐0.30, 0.10]; p=.314).ConclusionTargeted exercise training may be efficacious in improving cognition in adults with chronic stroke and should be promoted among individuals with chronic stroke.

Multidimensional item response theory and the Brief Electricity and Magnetism Assessment

The Conceptual Survey of Electricity and Magnetism and the Brief Electricity and Magnetism Assessment measure similar conceptual domains with different coverage and with items of different intellectual complexity.