Item Response Theory Procedures Research Articles

Rasch modelling vs. item facility: implications on the validity of assessments of Asian EFL/ESL vocabulary knowledge and lexical sophistication modelling

This study examines how second/foreign language (L2) word difficulty estimates derived from item response theory (IRT) and classical test theory (CTT) frameworks are virtually identical in the context of vocabulary testing. This conclusion is reached via a two-stage process: (a) psychometric assessments of both approaches and (b) L2 word difficulty modelling with lexical sophistication. Using data collected from first/native language (L1) Japanese EFL learners, both approaches led to similar conclusions in terms of the psychometric properties of the construct. Furthermore, CTT- and IRT/Rasch-derived estimates for word difficulty yielded nearly identical results in the predictive models. Although the “CTT-vs-IRT” debates in the past few decades have concluded with a middle ground agreed upon in most educational settings, this study acts as a useful demonstration to L2 vocabulary researchers who appear to rely heavily on Rasch/IRT analysis. The findings have practical applications to the area of L2 word difficulty research. Namely, that Rasch (an IRT procedure) alone might not be sufficient for validation, although it could be preferable when conducting inferential testing because of its potential utility for reducing chances of Type II errors.

Validity evidence for the coping strategy indicator-short version (CSI-S) among psychology students

The abbreviated measurement of coping strategies is useful for monitoring and identifying the effects of stress. The Coping strategy indicator-Short version (CSI-S, including the dimensions of seeking support, problem solving and avoidance strategies) is a new adaptation of the full version of this indicator, and additional evidence of its validity is needed. Psychology students (n = 125) from a public university in Lima, Peru, were recruited to help provide such evidence of validity in terms of internal structure, reliability and associations with other variables (perceived stress and general efficacy in cope with difficulties), which were evaluated using nonparametric item response theory procedures. Support-seeking and problem-solving items from the Mokken scale and the avoidance scale exhibited limitations. The correlations between the scales were moderate or low and exhibited theoretical consistency, and the relationship with perceived stress highlighted the predictive capacity of avoidance and problem-solving strategies. In general, the CSI-S exhibits suitable psychometric properties; however, the avoidance score requires further examination or reconstruction of its items.

Scale Development Based on Item Response Theory: A Systematic Review

Classical test theory (CTT) and item response theory (IRT) are two fundamental approaches used in scale development research. Although CTT is the preferred methodology in scale development research in the Republic of Turkey, the IRT methodology has started to gain traction in recent years. In this study, we used the systematic review methodology to review scale development research in Turkey that used the IRT procedures. Results revealed that most of the studies were conducted in 2017. Most of the studies have a sufficient sample size. Exploratory factor analysis was used more than confirmatory factor analysis. Some studies failed to choose the proper IRT model, report and interpret item parameters, check IRT assumptions, and report and interpret information functions. We recommend some practical considerations to researchers in terms of IRT assumptions.

The Online Flow Questionnaire: An Item Response Theory Examination.

Online flow refers to the rate of an individual's absorbance in an Internet activity in which they are engaged. It has been implicated with both the effectiveness of cyberhealth and online education applications, as well as excessive Internet use. One way of assessing it is the Online Flow Questionnaire (OFQ). Consequently, this study investigated the optimum measurement of online flow, as well as OFQ cutoff points, using Item Response Theory (IRT). A large sample of high school students from Greece (N = 1579, Mage = 16.12, SDage = 0.85; 50.5 percent females) completed the OFQ and the data were analyzed using IRT procedures. The analysis indicated that items in the OFQ possess differing levels of discrimination and difficulty, although all items were generally within acceptable ranges. An OFQ cutoff score of 5 represented an extremely high level of online flow experiences. The findings suggest that the OFQ generally functions as an acceptable marker of overall online flow. However, the current format of the OFQ appears to inhibit its ability to detect variability at the extreme low and high ends of the latent trait in the population assessed. Future revisions (potentially employing computerized adaptive tests) of the OFQ response format may enhance its utility.

Longitudinal measurement invariance of the Behavioral Health Measure in a clinical sample.

The practice of routine outcome monitoring (ROM) has grown in popularity and become a fixture in feedback-supported clinical practice and research. However, if the interpretation of an ROM measure changes over time, treatment outcome scores may be inaccurate and produce erroneous or misguided interpretations of client progress and therapist efficacy. The current study examined whether factorial invariance held when using the Behavioral Health Measure (BHM-20) longitudinally in a clinical sample (n = 12,467). Using multidimensional item response theory-based models for the investigation of the BHM-20 factor structure, at a single time point and then longitudinally. Based on the original factor structure of the BHM-20 a unidimensional model, a three-factor orthogonal model, and a three-factor correlated model were fit to the data, indicating poor model fit with the proposed three-factor or unidimensional models. Next, using exploratory factor analysis and subsequent multidimensional item response theory procedures, a new 4-factor (General Distress, Life Functioning, Anxiety, and Alcohol/Drug Use) model was proposed with improved model-fit statistics. Finally, when testing the longitudinal invariance of the BHM-17 over 10 sessions of treatment, it was found to be fully consistent. The current study proposes the use of a 17-item, 4-factor model for a new understanding of the BHM-17. Implications for use in ROM and limitations are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Development of the R library “jrt”: Automated item response theory procedures for judgment data and their application with the consensual assessment technique.

Development of the R library “jrt”: Automated item response theory procedures for judgment data and their application with the consensual assessment technique.

Quantifying the patient´s perspective in neuromyelitis optica spectrum disorder: Psychometric properties of the SymptoMScreen questionnaire.

BackgroundThe assessment of self-reported outcomes in neuromyelitis optica spectrum disorder (NMOSD) is limited by the lack of validated disease-specific measures. The SymptoMScreen (SyMS) is a patient-reported questionnaire for measuring symptom severity in different domains affected by multiple sclerosis (MS), but has not been thoroughly evaluated in NMOSD. The aim of this study was to assess the psychometric properties of the SyMS in a sample of patients with NMOSD.MethodsA non-interventional, cross-sectional study in adult subjects with NMOSD (Wingerchuk 2015 criteria) was conducted at 13 neuroimmunology clinics applying the SyMS. A non-parametric item response theory procedure, Mokken analysis, was performed to assess the underlying dimensional structure and scalability of items and overall questionnaire. All analyses were performed with R (v4.0.3) using the mokken library.ResultsA total of 70 patients were studied (mean age: 47.5 ± 15 years, 80% female, mean Expanded Disability Status Scale score: 3.0 [interquartile range 1.5, 4.5]). Symptom severity was low (median SyMS score: 19.0 [interquartile range 10.0, 32.0]). The SyMS showed a robust internal reliability (Cronbach’s alpha: 0.90 [95% confidence interval 0.86, 0.93]) and behaved as a unidimensional scale with all items showing scalability coefficients > 0.30. The overall SyMS scalability was 0.45 conforming to a medium scale according to Mokken’s criteria. Fatigue and body pain were the domains with the highest scalability coefficients. The SyMS was associated with disability (rho: 0.586), and physical and psychological quality of life (rho: 0.856 and 0.696, respectively).ConclusionsThe SyMS shows appropriate psychometric characteristics and may constitute a valuable and easy-to-implement option to measure symptom severity in patients with NMOSD.

Quantifying Knowledge of Alzheimer's Disease: An Analysis of the Psychometric Properties of the Alzheimer's Disease Knowledge Scale.

IntroductionThe Alzheimer’s Disease Knowledge Scale (ADKS) is one of the most popular instruments for assessing a person’s knowledge regarding Alzheimer’s disease (AD). The objective of this study was to explore ADKS item characteristics with item response theory (IRT) procedures.MethodsA noninterventional web-based study was conducted. A nonparametric IRT procedure, Mokken analysis, was used to explore the underlying latent structure of the ADKS and ADKS item characteristics regarding scalability and violations of the monotone homogeneity (MH) model. A random-effects meta-analysis was implemented that combined ADKS scores from independent studies.ResultsA total of 447 employees of a pharmaceutical company participated in the study. The mean ADKS score was 21.2 (SD 2.8). Mokken analysis showed that most ADKS items (22 of 30) do not fit to any scale and can be considered to be scale independent. Two items (#1: particularly prone to depression; #20: depression can be mistaken for AD) fit to a domain relating to depression, another two items (#2: mental exercise can prevent AD development; #8: benefit of psychotherapy) can be related to potential prevention and improvement, and four items (#12: poor nutrition can make the symptoms worse; #18: high cholesterol may increase the risk of AD; #26: high blood pressure may increase the risk of AD; #27: genes can only partially account for AD development) fit to a risk factor domain. As expected from those results, neither the overall scale (H = 0.033) nor its items showed appropriate scalability index values, suggesting that ADKS does not fit to a MH model. Eleven items showed violations of the assumptions of the MH model. The meta-analytical average score was 21.78 (95% CI 20.67–22.90), with healthcare professionals and caregivers showing the highest levels of AD knowledge.ConclusionAlthough the ADKS does not present a unidimensional structure, its independent items together provide a comprehensive spectrum of information regarding AD knowledge.

PND63 Knowledge and Understanding of Alzheimer'S Disease: Revisiting the Psychometric Properties of the Adks Questionnaire

The Alzheimer´s Disease Knowledge Scale (ADKS) is a 30-item instrument to assess what people know about AD covering risk factors, diagnosis, symptoms, course, functional impact, caregiving, and management. The aim of this study was to explore scale item characteristics with item response theory (IRT) procedures. A noninterventional, web-based study was conducted among employees of Roche Spain. We compared ADKS scores between participants with and without relatives diagnosed with AD. A nonparametric IRT procedure, Mokken analysis, was performed to explore its underlying structure. A parametric IRT analysis was then used to compare the fitting of two different models for dichotomous items, the one-parameter logistic model (1PL) that assumes the items show different difficulty but share the same discrimination parameter, and the two-parameter logistic model (2PL) that assumes the items show different difficulty and discrimination parameters. A total of 447 participants completed the ADKS. The mean ADKS score was 21.2 (SD 2.8). Participants with a first-degree relative with AD (N=42, 9%) scored higher (mean difference 1.06, p-value = 0.02; standardized mean difference 0.38, indicating a small to moderate effect). Mokken analysis showed that most items (22/30) did not fit to any scale and could be considered scale independent. Two items fitted a knowledge domain related with depression, another two items were related with AD prevention and recovery, and four items fitted a risk factors domain. The likelihood ratio test (LRT) between 1PL and 2PL models was significant (LRT 198.91, df 29, p-value < 0.0001), favoring 2PL. The overall ADKS score distinguishes knowledge about dementia among people with or without AD relatives. IRT analyses suggest an independence among items that is highly relevant for a general knowledge scale. In summary, the 30-item ADKS, as originally reported, could be used to assess the public´s knowledge and to develop and guide educational interventions if needed.

Measuring brand personality using emoji: findings from Mokken scaling

Brand personality (BP) measurement is a research area that has been attracting a tremendous amount of attention. An examination of extant BP scales reveals that: (a) all of them gauge BP through verbal stimuli; (b) most of them were formulated in one language and tested in one country; and that (c) the majority of them were developed using Classical Test Theory and factor analytic procedures as “the” measurement framework and the method of choice, respectively. This study explores the possibility of using emoji (a new universal nonverbal language) to measure BP. It adopts a Nonparametric Item Response Theory procedure known in personality research as Mokken scaling. Data were collected from 416 respondents from 10 countries via an online 30-emoji survey assessing four global brands. Results indicate that 17 of the 30 emoji form four strong and reliable Mokken scales gauging dimensions Agreeableness, Conscientiousness/Openness, Extraversion, and Neuroticism. The proposed emoji-based scale represents a valuable measure that is suited to national as well as cross-national BP research.

Automated Assessment of Complex Programming Tasks Using SIETTE

This paper presents an innovative method to tackle the automatic evaluation of programming assignments with an approach based on well-founded assessment theories (Classical Test Theory (CTT) and Item Response Theory (IRT)) instead of heuristic assessment as in other systems. CTT and/or IRT are used to grade the results of different items of evidence obtained from students’ results. The methodology consists of considering program proofs as items, calibrating them, and obtaining the score using CTT and/or IRT procedures. These procedures measure overall validity reliability as well as diagnose the quality of each proof (item). The evidence is obtained through program proofs. The SIETTE system collects and processes all data to calculate the student knowledge level. This innovative method for programming task evaluation makes it possible to deploy the whole artillery developed in this research field over the last few decades. To the best of our knowledge, this is a new and original contribution in the area of programming assessment.

Assessing Stigma in Multiple Sclerosis: Psychometric Properties of the Eight-Item Stigma Scale for Chronic Illness (SSCI-8).

The stigma associated with neurologic disorders plays a part in poor health-related quality of life. The eight-item Stigma Scale for Chronic Illness (SSCI-8) is a brief self-assessment tool for measuring perceived level of stigma. The psychometric performance of the SSCI-8 in people with multiple sclerosis (MS) was assessed. A multicenter, cross-sectional study in adults with relapsing-remitting or primary progressive MS was performed. A nonparametric item response theory procedure, Mokken analysis, was done to preliminarily study the dimensional structure of the SSCI-8. A confirmatory factor analysis (CFA) model was then fit, and the behavior and information covered by the eight items were assessed by parametric item response theory analysis. A total of 201 patients (mean ± SD age, 43.9 ± 10.5 years; 60.2% female; 86.1% with relapsing-remitting MS) were studied. The Mokken analysis found that the SSCI-8 is a unidimensional strong scale (scalability index H = 0.56) with high reliability (Cronbach α = 0.88). The CFA model confirmed the unidimensionality (comparative fit index = 0.975, root mean square error of approximation = 0.077). The information covered by the SSCI-8 items ranges from 3.79 to 13.52, for a total of 66.56. More than half (66%) of the SSCI-8 overall information is conveyed by four items: 1 ("Some people avoided me"), 2 ("I felt left out of things"), 3 ("People avoided looking at me"), and 7 ("People were unkind to me"). The SSCI-8 shows appropriate psychometric characteristics and is, therefore, a useful instrument for assessing stigma in people with MS.

Psychopathic features across development: Assessing longitudinal invariance among Caucasian and African American youths

ObjectivesPsychopathy is associated with severe forms of antisocial and violent behavior in adults. There is also a rapidly growing body of research focused on extending features of adult psychopathy downward to youth. To date however, the degree to which these features can be consistently and comparatively assessed at these younger ages, remains unclear. This study addresses this issue by investigating measurement invariance of underlying features of psychopathy across childhood and adolescence in a racially diverse sample of youth. MethodsThree cohorts of youth (n = 1517) were assessed annually from childhood to adolescence (ages ∼7–16). Underlying features of psychopathy commonly assessed in youth (e.g. lack of guilt, impulsivity) were examined within a longitudinal bi-factor framework using multi-dimensional item-response theory (IRT) techniques. Differential item functioning was used to assess invariance across development and participant’s race (African-American and Caucasian), using two distinct approaches: (1) traditional item-response theory (IRT) methods; and (2) a recently developed Bayesian structural equation modeling (BSEM) approach. ResultsPsychopathy features assessed in this study exhibited measurement consistency across development (∼ages 7–16) and were found to tap into the same underlying construct as intended across measurement occasions, and equivalently for African-American and Caucasian youth. Results were similar when assessed using traditional IRT procedures for longitudinal invariance testing and when implementing the more recent BSEM methodology. ConclusionsFindings provide the first evidence that features of psychopathy can be assessed consistently in youth and improve our understanding of important developmental and sociocultural factors associated these features during earlier periods of development.

Using Procedure Based on Item Response Theory to Evaluate Classification Consistency Indices in the Practice of Large-Scale Assessment.

In spite of the growing interest in the methods of evaluating the classification consistency (CC) indices, only few researches are available in the field of applying these methods in the practice of large-scale educational assessment. In addition, only few studies considered the influence of practical factors, for example, the examinee ability distribution, the cut score location and the score scale, on the performance of CC indices. Using the newly developed Lee's procedure based on the item response theory (IRT), the main purpose of this study is to investigate the performance of CC indices when practical factors are taken into consideration. A simulation study and an empirical study were conducted under comprehensive conditions. Results suggested that with negatively skewed distribution, the CC indices were larger than with other distributions. Interactions occurred among ability distribution, cut score location, and score scale. Consequently, Lee's IRT procedure is reliable to be used in the field of large-scale educational assessment, and when reporting the indices, it should be treated with caution as testing conditions may vary a lot.

PROC IRT: A SAS Procedure for Item Response Theory

This article reviews the procedure for item response theory (PROC IRT) procedure in SAS/STAT 14.1 to conduct item response theory (IRT) analyses of dichotomous and polytomous datasets that are unidimensional or multidimensional. The review provides an overview of available features, including models, estimation procedures, interfacing, input, and output files. A small-scale simulation study evaluates the IRT model parameter recovery of the PROC IRT procedure. The use of the IRT procedure in Statistical Analysis Software (SAS) may be useful for researchers who frequently utilize SAS for analyses, research, and teaching.

An Approach to Scoring and Equating Tests With Binary Items: Piloting With Large-Scale Assessments.

This article describes an approach to test scoring, referred to as delta scoring (D-scoring), for tests with dichotomously scored items. The D-scoring uses information from item response theory (IRT) calibration to facilitate computations and interpretations in the context of large-scale assessments. The D-score is computed from the examinee's response vector, which is weighted by the expected difficulties (not "easiness") of the test items. The expected difficulty of each item is obtained as an analytic function of its IRT parameters. The D-scores are independent of the sample of test-takers as they are based on expected item difficulties. It is shown that the D-scale performs a good bit better than the IRT logit scale by criteria of scale intervalness. To equate D-scales, it is sufficient to rescale the item parameters, thus avoiding tedious and error-prone procedures of mapping test characteristic curves under the method of IRT true score equating, which is often used in the practice of large-scale testing. The proposed D-scaling proved promising under its current piloting with large-scale assessments and the hope is that it can efficiently complement IRT procedures in the practice of large-scale testing in the field of education and psychology.

A two-step item response theory procedure for a better measurement of marketing constructs

In order to broaden the set of methods for developing better multi-item marketing scales, this article proposes an item response theory (IRT) procedure that consists of two successive steps. The first exploratory step uses a non-parametric IRT model for dimensionality detection and scale purification. The second confirmatory step employs a parametric IRT model for evaluating, and likely refining, the resulting scale. This two-step IRT procedure also involves a wide range of new parametric and non-parametric IRT-based interpretive tools (that is, goodness-of-fit indices, scale-score reliability coefficients and item response/information functions) that allows scale developers and users to work in a highly informed manner. To illustrate the benefits of the proposed procedure, the article presents an extended application to the topical construct of emotional attachment.

The Validity of the Instrument to Evaluate Social Network in the Ageing Population: The Collaborative Research on Ageing in Europe Social Network Index

The aim of the study was to create a simplified, easy implementable multidimensional instrument to assess all relevant elements of the structure and function of social network within individuals across different European countries and to provide the tool for health professionals and policy makers. The analysis was based on the sample of 10 446 non-institutionalized adult population from Finland, Poland and Spain. The Social Network Questionnaire Collaborative Research on Ageing in Europe Social Network Index (COURAGE-SNI) was part of the COURAGE questionnaire. The indicators of the functioning of social network ties (close relations), frequency of direct contact and general support were evaluated. Functions were assess within the main structural components as spouse, parents, children, grandchildren, other relatives, friends, coworkers and neighbours. The exploratory factor analysis revealed five main latent components of social network with one component composed of hierarchical part. The confirmatory factor analysis provided an acceptable fit for the model. The generalize partial credit model was used to calculate factor scores for five components of the COURAGE-SNI considering the social networks of 'spouse/partner', 'parents', 'other family members', 'neighbours' and 'friends and co-workers'. The scores for every component were recalculated so as to provide the social network saturation ranged from 0 (the lowest) to 100% (the highest possible). Finally, the COURAGE-SNI score was obtained as the sum of weighted information calculated by the item response theory procedure for every aforementioned component. In summary, the COURAGE-SNI showed good reliability and content validity and seems to be a promising tool for the assessment of the social network phenomenon across European countries. The Courage-SNI is a new tool to assess the construct of social network in population studies. The Courage-SNI is an instrument useful to identify high risk groups or populations whose social network is poorer.

An Exploration of the Latent Constructs of the STATIC-99

Although actuarial instruments are ubiquitously used in the field of sex offender recidivism risk assessment, there is limited empirical information about the underlying constructs from which they are derived. The following study utilized a nonparametric item response theory procedure, a Mokken analysis, and nonlinear factor analysis (Normal Ogive Harmonic Analysis Robust Method) to explore the underlying constructs of the STATIC-99 scores obtained for male sexual offenders (N = 451) referred for an evaluation to determine if they met criteria for civil commitment under a state's Sexually Violent Persons Law. The results from the analyses indicated that the STATIC-99 comprises the two previously identified constructs associated with sexual deviancy and antisocial behaviors as well as a third, additional construct, associated with the items pertaining to age and past marital-type relationships. These findings support Hanson and Thornton's assertion that sexual offender recidivism risk is multifactorial and not the result of a single underlying trait. Implications for future research are discussed.

The Nursing Student Self-Efficacy Scale

For nurse educators to optimize instruction that supports students' accurate estimates of their ability, a sound measure of nursing self-efficacy is needed. To provide evidence for an adequate item response theory model fit to scores measuring nursing students' self-efficacy to care for critically ill patients and to use the theory to provide evidence for reliable and valid interpretation of self-efficacy scores. The Nursing Student Self-Efficacy Scale was administered to 421 nursing students. Data regarding students' self-efficacy beliefs for psychomotor skill performance and communication skills were analyzed using the generalized partial credit item response theory model. All items showed acceptable model-data fit using the chi-square test with an alpha of .001. Review of item information functions showed that 40% of the items provided high information about self-efficacy and 20% provided moderately high information. Comparison of trait level distributions with reliability information given by the two subscales showed that the psychomotor subscale measured self-efficacy most accurately for second and third semester students and the communication subscale measured efficacy most accurately for first and second semester students. Significant mean differences in self-efficacy scores from both subscales were found between students in successive semesters of the program. Data analysis provided some evidence for reliable interpretation of scores indicating self-efficacy beliefs of nursing students. Comparison of scores from students in different semesters provided validity evidence; the instrument could be used to discriminate between beginning students and more advanced students possessing different levels of self-efficacy. When used for scale development, item response theory procedures can be more informative about item and test reliability than classical true score theory.