Rasch Fit Statistics Research Articles

Adaptation and measurement invariance of the 13-item version of Patient Activation Measure across Japanese young adult cancer survivors during and after treatment: A cross-sectional observational study.

The 13-item version of the Patient Activation Measure (PAM-13) is a frequently used measure that gauges the level of self-management in an individual. However, its applicability across Japanese young adult (YA) cancer survivors during and after their treatment remains unclear. This study confirmed the psychometric properties and measurement invariance of the Japanese version of PAM-13 across them during and after treatment. We used cross-sectional observational data collected through an online survey from 500 survivors in January 2022. We determined feasibility, internal consistency, concurrent validity against physical fatigue and depression, and known-groups validity regarding educational level. Structural validity was also found using Rasch analysis for survivors both during and after treatment. Furthermore, measurement invariance of the PAM-13 was examined using multiple-group structural equation modeling. Rasch fit statistics were acceptable for the unidimensional structure of PAM-13. It was found to be internally consistent for survivors during (McDonald's omega: 0.88, item-total correlations: 0.48-0.62) and after treatment (McDonald's omega: 0.90, item-total correlations: 0.32-0.72). The PAM-13 was concurrently valid with physical fatigue (Pearson's product-moment correlation coefficients: -0.25 and -0.18 for survivors during and after treatment, respectively) and depression (Pearson's product-moment correlation coefficients: -0.20 and -0.19 for survivors during and after treatment, respectively). Known-groups validity showed that survivors after treatment with a higher educational level reported a higher patient activation score than those with a lower educational level (p = 0.001); however, there was no difference due to the education level between survivors during treatment. The configural and metric invariance of the PAM-13 were confirmed, but scalar invariance was rejected. It was found that the PAM-13 is applicable for Japanese YA cancer survivors during and after treatment. However, given the lack of scalar invariance in the PAM-13, the scores of particular items between YA cancer survivors during and after treatment should be interpreted with caution.

Exploring Item Bank Stability through Live and Simulated Datasets

LanguageCert manages the construction of its tests, exams and assessments using a sophisticated item banking system which contains large amounts of test material that is described, inter alia, in terms of content characteristics such as macroskills, grammatical and lexical features and measurement characteristics such as Rasch difficulty estimates and fit statistics. In order to produce content and difficulty equivalent test forms, it is vital that the items in any LanguageCert bank manifest stable measurement characteristics. The current paper is one of two linked studies exploring the stability of one of the item banks developed by LanguageCert [Note 1]. This particular bank has been used as an adaptive test bank and comprises 820 calibrated items. It has been administered to over 13,000 test takers, each of whom have taken approximately 60 items. The purpose of these two exploratory studies is to examine the stability of this adaptive test item bank from both statistical and operational perspectives. The study compares test taker performance in the live dataset with over 13,000 test takers (where each test taker takes approximately 60 items) with a simulated ‘full’ dataset generated using model-based imputation. Simulation regression lines showed a good match and Rasch fit statistics were also good: thus indicating that items comprising the adaptive item bank are of high quality both in terms of content and statistical stability. Potential future stability was confirmed by results obtained from a Bayesian ANOVA. As mentioned above, such item bank stability is important when item banks are used for multiple purposes, in this context for adaptive testing and the construction of linear tests. The current study therefore lays the ground work for a follow-up study where the utility of this adaptive test item bank is verified by the construction, administration and analysis of a number of linear tests.

Detection of Outliers in Anchor Items Using Modified Rasch Fit Statistics

ABSTRACT The existence of outliers in the anchor items can be detrimental to the estimation of examinee ability and undermine the validity of score interpretation across forms. However, in practice, anchor item performance can become distorted due to various reasons. This study compares the performance of modified INFIT and OUTFIT Rasch statistics with the Logit Difference approach with 0.3 and 0.5 as the predetermined cutoff values, and the Robust z statistic with 1.645 and 2.7 as the cutoff values through a simulation study by varying the sample size, proportion of outliers, item difficulty drift direction, and group difference magnitude. The results suggest that both modified INFIT and OUTFIT statistics perform very similarly and outperform the other methods in all aspects, including sensitivity of flagging outliers, specificity of flagging non-outliers, recovery of translation constant, and recovery of examinee ability in all simulated conditions.

Identifying Guessing in English Language Tests via Rasch Fit Statistics: An Exploratory Study

This article explores the issue of identifying guessers – with a specific focus on multiple-choice tests. Guessing has long been considered a problem due to the fact that it compromises validity. A test taker scoring higher than they should through guessing does not provide a picture of their actual ability. After an initial description of issues associated with guessing, the article then outlines approaches which have been taken to either discourage test takers from guessing or which attempt statistically to handle the problem. From this, the article moves to a novel way of identifying potential guessers: from the post hoc use of Rasch fit statistics. Two datasets, each consisting of approximately 200 beginner level English language test takers were split into two. In each dataset, half the test takers’ answers were randomised – to approximate guessing. Results obtained via a Rasch analysis of the data was then passed to an analyst who used the Rasch fit statistics to identify possible guessers. On each dataset, 80% of guessers were identified.

Is the Patient Activation Measure a valid measure of osteoarthritis self-management attitudes and capabilities? Results of a Rasch analysis

BackgroundThe Patient Activation Measure (PAM-13) was developed using Rasch analysis to assess knowledge, skills and confidence in the management of one’s health. Previous studies report positive relationships between PAM-13 scores, self-management behaviours and longitudinal health outcomes in adults with chronic disease. There is little extant measurement property evidence for the use of PAM-13 in specific osteoarthritis (OA) populations. This study tested measurement properties of the PAM-13 in people living with hip and knee OA.MethodsItem response frequency analysis was conducted. Rasch analysis evaluated the fit of the PAM-13 data to the Rasch model. Model-data fit was evaluated using infit and outfit statistics; person/item reliability and person separation indices were computed. Unidimensionality was evaluated using Principal Components Analysis of Rasch residuals and the data were assessed for item redundancy. Differential Item Functioning (DIF) examined bias in respondent subgroups and correlations tested relationships between PAM-13 and other patient-reported outcomes.ResultsTwo-hundred-and-seventeen PAM-13 surveys were completed; there were no missing responses, floor or ceiling effects. Person and item reliability were acceptable (0.98 and 0.87 respectively) with good separation (person separation index 2.58). Unidimensionality was evaluated, with 49.4% of the variance explained by the first eigenvector. There was evidence of potential local response-dependence. The Rasch fit statistics were acceptable (except for item-2). There were some issues identified with targeting of the PAM-13 items to people with higher ability and the item difficulty order was different to that proposed in original cohorts. Significant DIF was identified for sex and educational level for a small number of items. PAM-13 scores were moderately correlated with depressive symptoms on the Depression Anxiety Stress Scale and Assessment of Quality of Life-6D. There were small correlations between PAM-13 and Knee injury and Osteoarthritis Outcome Score pain and activities of daily living scores.ConclusionsThis study provides some evidence of adequate person and item reliability, unidimensionality, and construct validity to support the use of PAM-13 to measure patient activation in people living with hip and knee OA. Possible limitations regarding targeting, different item difficulty order, DIF and local response dependence should be investigated in future research.

Using Overt and Covert Items in Self-Report Personality Tests: Susceptibility to Faking and Identifiability of Possible Fakers.

Self-report personality tests widely used in clinical, medical, forensic, and organizational areas of psychological assessment are susceptible to faking. Several approaches have been developed to prevent or detect faking, which are based on the use of faking warnings, ipsative items, social desirability scales, and validity scales. The approach proposed in this work deals with the use of overt items (the construct is clear to test-takers) and covert items (the construct is obscure to test-takers). Covert items are expected to be more resistant to faking than overt items. Two hundred sixty-seven individuals were presented with an alexithymia scale. Two experimental conditions were considered. Respondents in the faking condition were asked to reproduce the profile of an alexithymic individual, whereas those in the sincere condition were not asked to exhibit a particular alexithymia profile. The items of the scale were categorized as overt or covert by expert psychotherapists and analyzed through Rasch models. Respondents in the faking condition were able to exhibit measures of alexithymia in the required direction. This occurred for both overt and covert items, but to a greater extent for overt items. Differently from overt items, covert items defined a latent variable whose meaning was shared between respondents in the sincere and faking condition, and resistant to deliberate distortion. Rasch fit statistics indicated unexpected responses more often for respondents in the faking condition than for those in the sincere condition and, in particular, for the responses to overt items by individuals in the faking condition. More than half of the respondents in the faking condition showed a drift rate (difference between the alexithymia levels estimated on the responses to overt and covert items) significantly larger than that observed in the respondents in the sincere condition.

Development and validation of an instrument to measure understanding in daily lesson plan

A valid, reliable and practical instrument is needed in measuring any kind of understanding. The purpose of this study is to develop and validate an instrument to measure the understanding in the daily lesson plan (DLP). The whole population is chosen as samples which involves 241 students enrolling in the bachelor degree of education in the final semester of the 2015/2016 session in the Malaysian institution of higher learning. The instrument (K-RPH) is developed based on the principles in the Madeline Hunter Model of Mastery Learning and The ASSURE Model. The instrument consists of 32 items with 160 rubrics with four main constructs namely analysing DLP, determining the objective of DLP, preparing learning experiences and evaluating and reflecting. Content validity is conducted by two experts in the field from the university. The instrument validity and reliability have been assessed using the Rasch Measurement Model by identifying the Rasch fit statistics, item difficulty, unidimensionality, item dispersion and reliability as well as the item distribution map. The Rasch analysis shows that the item reliability is 0.99 while the Cronbach Alpha is 0.96. All the items fit the model as their MNSQ values are between 0.75 and 1.49. The dispersion of items from K-RPH data is 10.29 which indicates the existence of 11 to 12 item strata group. No item shows a negative point measure correlation or less than 0.2, and this generally indicated that the item discrimination is very good.The data shows that the mean for person is 1.38 logits with a standard deviation of 1.67 logits, while the item mean is zero with a standard deviation of 1.02. This indicates that the position of item and person do not fully match, and thus shows a medium difficulty. The overall item quality is good and all 32 items of K-RPH are retained.

Using the Bootstrap Method to Evaluate the Critical Range of Misfit for Polytomous Rasch Fit Statistics.

The purpose of this study was to apply the bootstrap procedure to evaluate how the bootstrapped confidence intervals (CIs) for polytomous Rasch fit statistics might differ according to sample sizes and test lengths in comparison with the rule-of-thumb critical value of misfit. A total of 25 simulated data sets were generated to fit the Rasch measurement and then a total of 1,000 replications were conducted to compute the bootstrapped CIs under each of 25 testing conditions. The results showed that rule-of-thumb critical values for assessing the magnitude of misfit were not applicable because the infit and outfit mean square error statistics showed different magnitudes of variability over testing conditions and the standardized fit statistics did not exactly follow the standard normal distribution. Further, they also do not share the same critical range for the item and person misfit. Based on the results of the study, the bootstrapped CIs can be used to identify misfitting items or persons as they offer a reasonable alternative solution, especially when the distributions of the infit and outfit statistics are not well known and depend on sample size.

The Computer-Vision Symptom Scale (CVSS17): development and initial validation.

To develop a questionnaire (in Spanish) to measure computer-related visual and ocular symptoms (CRVOS). A pilot questionnaire was created by consulting the literature, clinicians, and video display terminal (VDT) workers. The replies of 636 subjects completing the questionnaire were assessed using the Rasch model and conventional statistics to generate a new scale, designated the Computer-Vision Symptom Scale (CVSS17). Validity and reliability were determined by Rasch fit statistics, principal components analysis (PCA), person separation, differential item functioning (DIF), and item-person targeting. To assess construct validity, the CVSS17 was correlated with a Rasch-based visual discomfort scale (VDS) in 163 VDT workers, this group completed the CVSS17 twice in order to assess test-retest reliability (two-way single-measure intraclass correlation coefficient [ICC] and their 95% confidence intervals, and the coefficient of repeatability [COR]). The CVSS17 contains 17 items exploring 15 different symptoms. These items showed good reliability and internal consistency (mean square infit and outfit 0.88-1.17, eigenvalue for the first residual PCA component 1.37, person separation 2.85, and no DIF). Pearson's correlation with VDS scores was 0.60 (P < 0.001). Intraclass correlation coefficient for test-retest reliability was 0.849 (95% confidence interval [CI], 0.800-0.887), and COR was 8.14. The Rasch-based linear-scale CVSS17 emerged as a useful tool to quantify CRVOS in computer workers. : Spanish Abstract.

Improving the psychometric utility of the hypomania checklist (HCL-32): A Rasch analysis approach

BackgroundThe HCL-32 is a widely-used screening questionnaire for hypomania. We aimed to use a Rasch analysis approach to (i) evaluate the measurement properties, principally unidimensionality, of the HCL-32, and (ii) generate a score table to allow researchers to convert raw HCL-32 scores into an interval-level measurement which will be more appropriate for statistical analyses. MethodsSubjects were part of the Bipolar Disorder Research Network (BDRN) study with DSM-IV bipolar disorder (n=389). Multidimensionality was assessed using the Rasch fit statistics and principle components analysis of the residuals (PCA). Item invariance (differential item functioning, DIF) was tested for gender, bipolar diagnosis and current mental state. Item estimates and reliabilities were calculated. ResultsThree items (29, 30, 32) had unacceptable fit to the Rasch unidimensional model. Item 14 displayed significant DIF for gender and items 8 and 17 for current mental state. Item estimates confirmed that not all items measure hypomania equally. LimitationsThis sample was recruited as part of a large ongoing genetic epidemiology study of bipolar disorder and may not be fully representative of the broader clinical population of individuals with bipolar disorder. ConclusionThe HCL-32 is unidimensional in practice, but measurements may be further strengthened by the removal of four items. Re-scored linear measurements may be more appropriate for clinical research.

The Development of an Instrument to Measure Quality of Vision: The Quality of Vision (QoV) Questionnaire

To develop an instrument to measure subjective quality of vision: the Quality of Vision (QoV) questionnaire. A 30-item instrument was designed with 10 symptoms rated in each of three scales (frequency, severity, and bothersome). The QoV was completed by 900 subjects in groups of spectacle wearers, contact lens wearers, and those having had laser refractive surgery, intraocular refractive surgery, or eye disease and investigated with Rasch analysis and traditional statistics. Validity and reliability were assessed by Rasch fit statistics, principal components analysis (PCA), person separation, differential item functioning (DIF), item targeting, construct validity (correlation with visual acuity, contrast sensitivity, total root mean square [RMS] higher order aberrations [HOA]), and test-retest reliability (two-way random intraclass correlation coefficients [ICC] and 95% repeatability coefficients [R(c)]). Rasch analysis demonstrated good precision, reliability, and internal consistency for all three scales (mean square infit and outfit within 0.81-1.27; PCA >60% variance explained by the principal component; person separation 2.08, 2.10, and 2.01 respectively; and minimal DIF). Construct validity was indicated by strong correlations with visual acuity, contrast sensitivity and RMS HOA. Test-retest reliability was evidenced by a minimum ICC of 0.867 and a minimum 95% R(c) of 1.55 units. The QoV Questionnaire consists of a Rasch-tested, linear-scaled, 30-item instrument on three scales providing a QoV score in terms of symptom frequency, severity, and bothersome. It is suitable for measuring QoV in patients with all types of refractive correction, eye surgery, and eye disease that cause QoV problems.

Rasch analysis informed the development of a classification system for a diabetes-specific preference-based measure of health

Objective To develop a classification system (CS) for a diabetes-specific preference-based measure of health titled the Diabetes Utility Index (DUI). Study Design and Setting Factor analysis of the Audit of Diabetes-Dependent Quality-of-Life (ADDQoL) items ( n = 385) identified plausible attributes. An expert panel provided qualitative input, including additional items. Data from three pilot rounds on patients with type 1 or type 2 diabetes were analyzed using Rasch analysis (RA). In a validation survey, the final version of the CS was mailed along with the SF-12v2, Well-Being Questionnaire, and Diabetes Empowerment Scale Short Form to a convenience sample (type 1 or type 2 diabetes). Results Factor analysis identified two plausible attributes. Experts rated the importance of ADDQoL and additional items, described attributes from item sets and suggested severity levels. Three pilot rounds ( n1 = 52, n2 = 65, n3 = 111) tested versions of a CS, containing five attributes with severity levels that were modified using RA and expert input. The final attributes were Physical Ability and Energy, Relationships, Mood and Feelings, Enjoyment of Diet, and Satisfaction with Management of Diabetes. The validation survey ( n = 396) results indicated satisfactory Rasch fit statistics, reliability, and severity scaling, whereas correspondence of responses to the CS with included measures suggested validity. Conclusion Results provide initial report of the validity and reliability of the CS of the DUI.

Rasch fit statistics and sample size considerations for polytomous data

BackgroundPrevious research on educational data has demonstrated that Rasch fit statistics (mean squares and t-statistics) are highly susceptible to sample size variation for dichotomously scored rating data, although little is known about this relationship for polytomous data. These statistics help inform researchers about how well items fit to a unidimensional latent trait, and are an important adjunct to modern psychometrics. Given the increasing use of Rasch models in health research the purpose of this study was therefore to explore the relationship between fit statistics and sample size for polytomous data.MethodsData were collated from a heterogeneous sample of cancer patients (n = 4072) who had completed both the Patient Health Questionnaire – 9 and the Hospital Anxiety and Depression Scale. Ten samples were drawn with replacement for each of eight sample sizes (n = 25 to n = 3200). The Rating and Partial Credit Models were applied and the mean square and t-fit statistics (infit/outfit) derived for each model.ResultsThe results demonstrated that t-statistics were highly sensitive to sample size, whereas mean square statistics remained relatively stable for polytomous data.ConclusionIt was concluded that mean square statistics were relatively independent of sample size for polytomous data and that misfit to the model could be identified using published recommended ranges.

Development and validation of the Brazilian version of the Attitudes to Aging Questionnaire (AAQ): an example of merging classical psychometric theory and the Rasch measurement model

BackgroundAging has determined a demographic shift in the world, which is considered a major societal achievement, and a challenge. Aging is primarily a subjective experience, shaped by factors such as gender and culture. There is a lack of instruments to assess attitudes to aging adequately. In addition, there is no instrument developed or validated in developing region contexts, so that the particularities of ageing in these areas are not included in the measures available. This paper aims to develop and validate a reliable attitude to aging instrument by combining classical psychometric approach and Rasch analysis.MethodsPilot study and field trial are described in details. Statistical analysis included classic psychometric theory (EFA and CFA) and Rasch measurement model. The latter was applied to examine unidimensionality, response scale and item fit.ResultsSample was composed of 424 Brazilian old adults, which was compared to an international sample (n = 5238). The final instrument shows excellent psychometric performance (discriminant validity, confirmatory factor analysis and Rasch fit statistics). Rasch analysis indicated that modifications in the response scale and item deletions improved the initial solution derived from the classic approach.ConclusionThe combination of classic and modern psychometric theories in a complementary way is fruitful for development and validation of instruments. The construction of a reliable Brazilian Attitudes to Aging Questionnaire is important for assessing cultural specificities of aging in a transcultural perspective and can be applied in international cross-cultural investigations running less risk of cultural bias.

The Quality of Life Impact of Refractive Correction (QIRC) Questionnaire: Development and Validation

The purpose of the study was to develop a questionnaire that could quantify the quality of life (QOL) of people with refractive correction by spectacles, contact lenses, and refractive surgery in the prepresbyopic age group. The questionnaire was developed and validated using traditional methods and Rasch analysis. A 90-item pilot questionnaire was developed through extensive literature search and use of professional and lay focus groups. Pilot study data were obtained from 306 subjects for item reduction to produce the 20-item Quality of Life Impact of Refractive Correction (QIRC) questionnaire. Validity and reliability studies (test-retest reliability with intraclass correlation coefficient and Bland-Altman limits of agreement, and internal consistency with Rasch fit statistics, factor analysis, and Cronbach's alpha) were performed from data of an additional 312 subjects. Rasch analysis demonstrated QIRC has good precision, reliability, and internal consistency (person separation, 2.03; reliability, 0.80; root-mean-square measurement error, 3.25; mean square +/- SD infit, 0.99 +/- 0.38; outfit, 1.00 +/- 0.39; item infit range, 0.70 to 1.24; and item outfit range, 0.78 to 1.32). The items (mean score, 50.3 +/- 7.3) were well targeted to the subjects (mean score, 47.8 +/- 5.5) with a mean difference of 2.45 (scale range, 0 to 100) units. Test-retest reliability (intraclass correlation coefficient, 0.88; coefficient of repeatability, +/-6.85 units), factor loading range (0.40 to 0.76), and Cronbach's alpha (0.78) also indicated the reliability and validity of QIRC. The 20-item QIRC questionnaire, which quantifies the QOL of people with refractive correction by spectacles, contact lenses, and refractive surgery in the prepresbyopic age group, was developed using Rasch analysis and shown to be valid and reliable. The use of Rasch scaling allows scores to be treated as a valid continuous variable. QIRC has broad applicability for cross-sectional and outcomes research.

A study of science teaching self‐efficacy and outcome expectancy beliefs of teachers in India

AbstractElementary and middle school teachers in urban schools in India provided responses to the science efficacy instrument (STEBI‐A). These responses were evaluated using Rasch analysis and parametric tests. Rasch fit statistics and person‐item maps were evaluated. It was found that the instrument worked well for the two groups of teachers, but the differential item functioning analysis found that the teachers utilized several items in the scale differently. Parametric tests suggested that self‐efficacy and outcome expectancy measures correlated highly for middle school teachers, for those that did not have a science degree and a written science curriculum. Significant predictors of self‐efficacy are—minutes per week science is taught, educational level, number of days in the school year, holding of a science degree, and the presence of a science curriculum. From all of the analyses we conclude that teaching experience is important, but not necessarily enough to increase teachers' outcome expectancy beliefs. The results of this study should benefit educators and policy makers with respect to teacher education in India and around the world. © 2004 Wiley Periodicals, Inc. Sci Ed 88:837–854, 2004