Unidimensionality Assumption Research Articles

An Assessment of Item Statistics Estimates of Basic Education Certificate Examination through Classical Test Theory and Item Response Theory approach

In this study, the researchers assessed comparability of
 item statistics of 2017 basic education certificate mathematics examination of
 National Examinations Council (NECO) through Classical Test Theory (CTT) and
 Item Response Theory (IRT) measurement frameworks. The study adopted
 instrumentation design. A 60-item NECO basic education certificate education
 mathematics objective test paper I was administered to 978 basic nine
 examinees, randomly selected from Osogbo and Olorunda Local Government Area,
 Osun State, Nigeria. The responses of the examinees to the test data were
 analysed using Marginal Maximum Likelihood Estimation of JMETRIK software.  The result showed that the test data obey the
 assumption of unidimensionality of 3-parameter logistic model and Classical
 Test Theory measurement framework deleted more items 33 (55%) compare to IRT
 measurement framework 12 (20%). Also, it was observed that item statistics from
 the two contrasting frameworks (CTT and IRT) were not comparable. Moreover,
 further analysis showed that there was low correlation among the item
 statistics index. The implication of this is that NECO should jettison the use
 of Classical Test Theory and embrace utilization of Item Response Theory
 framework during their test development and item analysis. 

Testing the Assumption of Population Homogeneity in the Measurement of Dispositional Optimism: Factor Mixture Modeling Analysis

The assumption of population homogeneity in the measurement of dispositional optimism was examined. U.S. and Canadian respondents (N = 591) completed an optimism scale. Possible population heterogeneity was analyzed using factor mixture modeling. Two major results emerged. First, population homogeneity was not supported: A large class of participants had trouble giving consistent answers to optimism items, optimism and pessimism items, or, especially, pessimism items. Second, after the removal of this problematic class, the correlation between optimism and pessimism was found to be .94, a magnitude large enough to support the assumption of unidimensionality. Although psychometric problems with the measurement of optimism have not been identified previously, this study suggests that the typical measurement of dispositional optimism requires substantial revision. The findings showcase the importance of factor mixture modeling in evaluating the psychometric properties of a measurement scale.

Mental health in primary care: an evaluation using the Item Response Theory.

ABSTRACTOBJECTIVE To determine the items of the Brazilian National Program for Improving Access and Quality of Primary Care that better evaluate the capacity to provide mental health care.METHODS This is a cross-sectional study carried out using the Graded Response Model of the Item Response Theory using secondary data from the second cycle of the National Program for Improving Access and Quality of Primary Care, which evaluates 30,523 primary care teams in the period from 2013 to 2014 in Brazil. The internal consistency, correlation between items, and correlation between items and the total score were tested using the Cronbach’s alpha, Spearman’s correlation, and point biserial coefficients, respectively. The assumptions of unidimensionality and local independence of the items were tested. Word clouds were used as one way to present the results.RESULTS The items with the greatest ability to discriminate were scheduling of the agenda according to risk stratification, keeping of records of the most serious cases of users in psychological distress, and provision of group care. The items that required a higher level of mental health care in the parameter of location were the provision of any type of group care and the provision of educational and mental health promotion activities. Total Cronbach’s alpha coefficient was 0.87. The items that obtained the highest correlation with total score were the recording of the most serious cases of users in psychological distress and scheduling of the agenda according to risk stratification. The final scores obtained oscillated between -2.07 (minimum) and 1.95 (maximum).CONCLUSIONS There are important aspects in the discrimination of the capacity to provide mental health care by primary health care teams: risk stratification for care management, follow-up of the most serious cases, group care, and preventive and health promotion actions.

Creation and validation of the barriers to alcohol reduction (BAR) scale using classical test theory and item response theory

Those who binge drink are at increased risk for alcohol-related consequences when compared to non-binge drinkers. Research shows individuals may face barriers to reducing their drinking behavior, but few measures exist to assess these barriers. This study created and validated the Barriers to Alcohol Reduction (BAR) scale. Participants were college students (n = 230) who endorsed at least one instance of past-month binge drinking (4+ drinks for women or 5+ drinks for men). Using classical test theory, exploratory structural equation modeling found a two-factor structure of personal/psychosocial barriers and perceived program barriers. The sub-factors, and full scale had reasonable internal consistency (i.e., coefficient omega = 0.78 (personal/psychosocial), 0.82 (program barriers), and 0.83 (full measure)). The BAR also showed evidence for convergent validity with the Brief Young Adult Alcohol Consequences Questionnaire (r = 0.39, p < .001) and discriminant validity with Barriers to Physical Activity (r = −0.02, p = .81). Item Response Theory (IRT) analysis showed the two factors separately met the unidimensionality assumption, and provided further evidence for severity of the items on the two factors. Results suggest that the BAR measure appears reliable and valid for use in an undergraduate student population of binge drinkers. Future studies may want to re-examine this measure in a more diverse sample.

Modelación con teoría de grafos para la unidimensionalidad de un instrumento de evaluación

Modelación con teoría de grafos para la unidimensionalidad de un instrumento de evaluación

Kayıp Veri ile Baş Etme Yöntemlerinin Madde Parametrelerine Etkisinin İncelenmesi

In this study, the aim is to determine how the item and test parameters affect the missing data techniques for different sample sizes and different items with different missing data rates. 500, 100 and 2500 students randomly selected from the 5073 students who participated in the PISA 2015 study and responded to the "ambition perception" scale included in the study constitute the study group of the research. First of all, the assumptions of normality, unidimensionality, local independence and model-data fit were examined for each data set. Afterwards, 5%, 10%, 15%, and 20% missing data were formed for four out of five items and there was no missing data in one item, then analyses were carried out. Once it is determined that the missing data are missing completely random, first with complete and incomplete data, then with serial mean, median of nearby points, mean of nearby points, linear interpolation, linear trend at point, regression, expectation maximization algorithm data item and test parameters were estimated. In the estimated process, descriptive statistics and cronbach alpha reliability coefficient and marginal reliability coefficient; the threshold parameters and the difficulty indices were estimated according to the graded response theory, which is one of the IRT models. The results of the study showed that the item and test parameters were influenced by incomplete and missing data techniques; it was determined that the best estimation results were obtained by linear interpolation method with different data.

Calibration and validation of an item bank for measuring general physical function of patients in medical rehabilitation settings.

ObjectiveThe objective of this study was to report the item response theory (IRT) calibration of an 18-item bank to measure general physical function (GPF) in a wide range of conditions and evaluate the validity of the derived scores.MethodsAll 18 items were administered to a large sample of patients (n=2337) who responded to the items in the context of their outpatient rehabilitation care. The responses, collected 1997– 2000, were modeled using the graded response model, an IRT model appropriate for items with two or more response options. Inter-item consistency was evaluated based on Cronbach’s alpha and item to total correlations. Validity of scores was evaluated based on known-groups comparisons (age, number of health problems, symptom severity). The strength of a single, general factor was evaluated using a bi-factor model. Results were used to evaluate IRT assumption and as an indicator of construct validity. Local independence of item responses was also evaluated.ResultsResponse data met the assumptions of unidimensionality and local independence. Explained common variance of a single general factor was 0.88 (omega hierarchical =0.86). Only two of the 153 pairs of item residuals were flagged for local dependence. Inter-item consistency was high (0.93) as were item to total correlations (mean =0.61). Substantial variation was found in both IRT location (difficulty) and discrimination parameters. All omnibus known-groups comparisons were statistically significant (p<0.001).ConclusionItem responses fit the IRT unidimensionality assumptions and were internally consistent. The usefulness of GPF scores in discriminating among patients with different levels of physical function was confirmed. Future studies should evaluate the validity of GPF scores based on an adaptive administration of items.

Propensity Score Matching Methods For Rasch Common-Person Equating Requirement: A Feasibility Study

To test if the Propensity Score (PS) matching method supports the unidimensionality assumption of the Rasch model, which would support use of PS models when the common-person approach is not plausible.

A neglected dimension of good forecasting judgment: The questions we choose also matter

Forecasters are typically evaluated via proper scoring rules such as the Brier score. These scoring rules use only the reported forecasts for assessment, neglecting related variables such as the specific questions that a person chose to forecast. In this paper, we study whether information related to question selection influences our estimates of forecaster ability. In other words, do good and bad forecasters tend to select questions in different ways? If so, can we capitalize on these selections when estimating forecaster ability? We address these questions by extending a recently-developed psychometric model of forecasts to include question selection data. We compare the extended psychometric model to a simpler model, studying its unidimensionality assumption and highlighting the unique information that it can provide. We find that the model can make use of the fact that good forecasters tend to select more questions than bad forecasters, and we conclude that question selection data can be beneficial above and beyond reported forecasts. As a side benefit, the resulting model can potentially provide unique incentives for forecaster participation.

Multidimensional latent trait linear mixed model: an application in clinical studies with multivariate longitudinal outcomes.

Multilevel item response theory (MLIRT) models have been widely used to analyze the multivariate longitudinal data of mixed types (e.g., categorical and continuous) in clinical studies. The MLIRT models often have unidimensional assumption, that is, the multiple outcomes are clinical manifestations of a univariate latent variable. However, the unidimensional assumption may be unrealistic because some diseases may be heterogeneous and characterized by multiple impaired domains with variable clinical symptoms and disease progressions. We relax this assumption and propose a multidimensional latent trait linear mixed model (MLTLMM) to allow multiple latent variables and within-item multidimensionality (one outcome can be a manifestation of more than one latent variable). We conduct extensive simulation studies to assess the unidimensional MLIRT model and the proposed MLTLMM model. The simulation studies suggest that the MLTLMM model outperforms unidimensional model when the multivariate longitudinal outcomes are manifested by multiple latent variables. The proposed model is applied to two motivating studies of amyotrophic lateral sclerosis: a clinical trial of ceftriaxone and the Pooled Resource Open-Access ALS Clinical Trials database. Copyright © 2017 John Wiley & Sons, Ltd.

Application of the Korean version of the Modified Barthel Index: Development of a keyform for use in clinical practice

ObjectiveTo demonstrate the clinical application of the Korean version of the Modified Barthel Index (K-MBI) using Rasch analysis.MethodsA total of 276 patients with neurological disorders were assessed with the K-MBI in outpatient clinics. The Rasch partial-credit model was used to generate a keyform based on investigating the psychometric properties of the K-MBI, including dimensionality, precision (person strata and reliability), and hierarchical item difficulty. The Minimal Detectable Change (MDC) in item difficulty was used to establish right-challenging treatment goals and long-term treatment plans.ResultsThe findings demonstrated that the Korean version of the MBI satisfied the assumption of unidimensionality. It also showed a hierarchical structure in terms of item difficulty, good reliability (Cronbach alpha, 0.92), and approximately five distinct person strata (4.6). The MDC (raw score, 20.1) of the item difficulty of the test items demonstrated equivalent cutoff scores for targeted short-term treatment goals on the keyform, a Rasch-derived display of patient responses. Long-term treatment goals were identified based on the test items of the keyform.ConclusionsThe findings suggest that a Rasch keyform can be applied to clinical practice in Korean settings by identifying clinically and statistically meaningful test items and their step thresholds as short- and long-term goals.

Validation of the Mini-OAKHQOL for use in patients with osteoarthritis in Spain.

The Mini-Osteoarthritis Knee and Hip Quality of Life (Mini-OAKHQOL) questionnaire osteoarthritis is specific to individuals with knee or hip osteoarthritis. The objective of this study was to perform a validation of the Mini-OAKHQOL for use in Spain in terms of its psychometric properties of reliability, validity and responsiveness. Patients with osteoarthritis from the waiting list for a joint replacement completed the OAKHQOL, Short Form 36 Health Survey and Western Ontario and McMaster Universities Osteoarthritis Index. Reliability was assessed in terms of internal consistency and test-retest data, and convergent validity using Spearman's correlation coefficient. Structural validity was investigated by confirmatory factor analysis, and Rasch analysis was used to examine the unidimensionality of the scales. Responsiveness was assessed by calculating effect sizes. Confirmatory factor analysis confirmed the five-factor model, and the results of the Rasch analyses supported the unidimensionality assumption, with infit and outfit statistics. Cronbach's alpha ranged from 0.76 to 0.89 for all except the social dimensions. Statistically significant differences were observed between patients with different degrees of disease severity on all dimensions. There was convergent validity among dimensions expected to be correlated. The OAKHQOL questionnaire showed good responsiveness, with large changes for all dimensions apart from the two social dimensions, which had small effect sizes. Results of the study support the view that the Spanish version of the Mini-OAKHQOL questionnaire is a valid instrument to measure health-related quality of life in patients with osteoarthritis of the lower limb.

Demonstrating Construct Validity of the American Board of Physical Medicine and Rehabilitation Part I Examination: An Analysis of Dimensionality

BackgroundIdeally, high-stakes examinations assess 1 dimension of medical knowledge to produce precise estimates of a candidate’s performance. It has not been reported whether the American Board of Physical Medicine and Rehabilitation Part 1 Certification Examination (ABPMR-CE-1) is unidimensional or not. ObjectiveTo examine the ABPMR-CE-1 to measure how many dimensions it assesses. DesignRetrospective observational study. SettingWe assessed examination results from the 2015 ABPMR-CE-1. ParticipantsA total of 489 deidentified candidates taking the 2015 ABPMR-CE-1. MethodsA 1-parameter Item Response Theory (IRT) measurement model was utilized. A Principal Components Analysis (PCA) of standardized residual correlations was used to detect multidimensionality. Main Outcome MeasureNumber of primary dimensions reflected in the 325 test questions. ResultsThe results of the dimensionality analysis indicated that the ABPMR-CE-1 examination is highly unidimensional from a psychometric perspective. Expert content review of the substantive content of small contrasting clusters of questions provided additional assurance of the unidimensional nature of the examination. ConclusionsThe ABPMR-CE-1 appears indeed to measure a single construct, which suggests a sound structure of the examination. It closely approximates the assumption of statistical unidimensionality. Level of EvidenceNot applicable.

Comparing five depression measures in depressed Chinese patients using item response theory: an examination of item properties, measurement precision and score comparability

BackgroundItem response theory (IRT) has been increasingly applied to patient-reported outcome (PRO) measures. The purpose of this study is to apply IRT to examine item properties (discrimination and severity of depressive symptoms), measurement precision and score comparability across five depression measures, which is the first study of its kind in the Chinese context.MethodsA clinical sample of 207 Hong Kong Chinese outpatients was recruited. Data analyses were performed including classical item analysis, IRT concurrent calibration and IRT true score equating. The IRT assumptions of unidimensionality and local independence were tested respectively using confirmatory factor analysis and chi-square statistics. The IRT linking assumptions of construct similarity, equity and subgroup invariance were also tested. The graded response model was applied to concurrently calibrate all five depression measures in a single IRT run, resulting in the item parameter estimates of these measures being placed onto a single common metric. IRT true score equating was implemented to perform the outcome score linking and construct score concordances so as to link scores from one measure to corresponding scores on another measure for direct comparability.ResultsFindings suggested that (a) symptoms on depressed mood, suicidality and feeling of worthlessness served as the strongest discriminating indicators, and symptoms concerning suicidality, changes in appetite, depressed mood, feeling of worthlessness and psychomotor agitation or retardation reflected high levels of severity in the clinical sample. (b) The five depression measures contributed to various degrees of measurement precision at varied levels of depression. (c) After outcome score linking was performed across the five measures, the cut-off scores led to either consistent or discrepant diagnoses for depression.ConclusionsThe study provides additional evidence regarding the psychometric properties and clinical utility of the five depression measures, offers methodological contributions to the appropriate use of IRT in PRO measures, and helps elucidate cultural variation in depressive symptomatology. The approach of concurrently calibrating and linking multiple PRO measures can be applied to the assessment of PROs other than the depression context.

Care coordination in PMAQ-AB: an Item Response Theory-based analysis.

ABSTRACTOBJECTIVE Analyze the quality of the National Program for Primary Care Access and Quality Improvement variables to evaluate the coordination of primary care.METHODS A cross-sectional study based on data from 17,202 primary care teams that participated in the National Program for Primary Care Access and Quality Improvement in 2012. Based on the Item Response Theory, Samejima’s Gradual Response Model was used to estimate the score related to the level of coordination. The Cronbach’s alpha and Spearman’ coefficients and the point-biserial correlation were used to analyze the internal consistency and the correlation between the items and between the items and the total score. We evaluated the assumptions of unidimensionality and local independence of the items. Cloud-type word charts aided in the interpretation of coordination levels.RESULTS The Program items with the greatest discrimination in coordination level were: telephone/Internet existence, institutional communication flows, and matrix support actions. The specialists’ contact frequency with the primary care and integrated electronic medical record required a greater level of coordination among the teams. The Cronbach’ alpha was 0.8018. The institutional communication flows and telephone/Internet items had a higher correlation with the total score. Coordination scores ranged from -2.67 (minimum) to 2.83 (maximum). More communication, information exchange, matrix support, health care in the territory and the domicile had a significant influence on the levels of coordination.CONCLUSIONS The ability to provide information and the frequency of contact among professionals are important elements for a comprehensive, continuous and high-quality care.

A computerized adaptive test for enterprising personality assessment in youth.

A pool of 161 items was developed and applied to two sets of participants (n 1 = 357 students, M age = 17.89; SD age = 3.26; n 2 = 2,693 students; M age = 16.52, SD age = 1.38) using a stratified sampling method. 107 items that assess achievement motivation, risk-taking, innovativeness, autonomy, self-efficacy, stress tolerance, internal locus of control, and optimism were selected. The assumption of unidimensionality was tested. The CAT demonstrated high precision for a wide range of q, using a mean of 10 items and demonstrating a relatively low Standard Error (0.378). A brief, valid, and precise instrument was obtained with relevant implications for educational and entrepreneurial contexts.

PHP85 - Integration Of Basic Attention To The Specialized Attention Network In Brazil’s Health System

In universal health systems, integral care management is considered the structural axis in overcoming health care fragmentation of services. This research intends to propose a classification for the integration level between basic and specialized attention and to apply this classification in the analysis of the integration between the care complexity levels on Brazil's health system. Transversal study with data from 17.202 family health teams that answered the external evaluation questionnaire of the National Program to Access and Quality Improvement of Basic Attention (PMAQ), 2012. In order to estimate the team’s integration level, the answers of 19 selected questions from PMAQ were fitted with the Gradual Response Model of the Item Response Theory (IRT). Based on the results, the usual IRT scale was categorized in three levels of integration: low, medium and high. The assumption of unidimensionality was considered valid. Cronbach’s alpha indicated high internal consistency (0.85). Southeast region has got the lowest percentages of teams with low integration and the highest percentages in the high integration, similarly to the South region. From the selected questions, twelve carried a greater amount of information about team’s integration to the network. Of these, we highlight matricial support actions (medical appointment; case discussions; shared clinical actions; joined construction of therapeutic projects; permanent education activities; discussion of work processes; interventions on territory and visits with a basic attention professional). In high integration level was recorded frequency of positive answers referred to communication (institutional flow, communication channel and contact list). Nine questions presented negative answers frequency around 50%. According to the proposed integration levels, matricial support actions improved the team’s performance. Communication devices between them reinforce this understanding. However, in order to consolidate integral care, a need to qualify the integration between the actions of basic and specialized attention was observed.

The Utility of the Bifactor Method for Unidimensionality Assessment When Other Methods Disagree

This article provides an empirical illustration of the utility of the bifactor method for unidimensionality assessment when other methods disagree. Specifically, we used two popular methods for unidimensionality assessment: (a) evaluating the model fit of a one-factor model using Mplus, and (b) DIMTEST to show that different unidimensionality methods may lead to different results, and argued that in such cases the bifactor method can be particularly useful. Those procedures were applied to English Placement Test (EPT), a high-stakes English proficiency test in Saudi Arabia, to determine whether EPT is unidimensional so that a unidimensional item response theory (IRT) model can be used for calibration and scoring. We concluded that despite the inconsistency between the one-factor model approach and DIMTEST, the bifactor method indicates that, for practical purposes, unidimensionality assumption holds for EPT.

Toward a Standardized Reporting of Outcomes in Hand Osteoarthritis: Developing a Common Metric of Outcome Measures Commonly Used to Assess Functioning.

Functioning is an important outcome in hand osteoarthritis (OA). The heterogeneity of patient-reported outcome measures (PROMs) available challenges the direct comparability of information collected by these instruments. This study aimed to examine whether it is possible to achieve metric equivalence of PROMs commonly used to measure functioning in people with hand OA. A secondary analysis of data from 253 persons who participated in the Vienna Hand Osteoarthritis Cohort Study was conducted applying the Rasch measurement model. Participants completed the Health Assessment Questionnaire, the Australian/Canadian Index for Hand Osteoarthritis, the Functional Index for Hand Osteoarthritis (FIHOA), and the Cochin Scale. The assumptions of stochastic ordering, local independence, unidimensionality, and invariance were tested for each scale independently before the scales were co-calibrated onto a common metric. Except for the FIHOA, all measures indicated issues of local dependency in the initial analyses. After accommodating those with testlets, all scales satisfied the assumptions of the Rasch model (χ(2) > 0.05). Marginal misfit in 2 items was found in the FIHOA, but this did not disturb person estimates. As the 4 scales form a common metric that satisfies the assumptions of the Rasch model, the scores can be equated. The scores of the 4 hand function measures can be transformed to a common 0-100 metric, such that scores can be interchanged. A user-friendly transformation table enables clinicians and researchers to have access to the common metric by simply adding up the total score for each instrument and identifying the corresponding transformed score on the common metric.

Structural Validity of the Mini-Balance Evaluation Systems Test (Mini-BESTest) in People With Mild to Moderate Parkinson Disease.

The Mini-Balance Evaluation Systems Test (Mini-BESTest) is a clinical balance test comprising 14 items assumed to reflect the unidimensional construct "dynamic balance." The study objective was to examine the dimensionality of the test and the properties of each item and their interrelationships in elderly people with mild to moderate Parkinson disease (PD). This was a cross-sectional study in a laboratory setting. A total of 112 participants (mean age=73 years) with idiopathic PD (Hoehn and Yahr stages 1-3) were assessed by physical therapists. Local independence among items was examined with Rasch modeling. Unidimensionality was tested by running a principal component analysis on the residuals. An exploratory factor analysis was used to examine the structure of the test, and a confirmatory factor analysis was used to evaluate the fit of the derived model. The first residual component of the principal component analysis, with an eigenvalue of greater than 2, superseded the assumption of unidimensionality. After the omission of item 7 because of convergence problems, the exploratory factor analysis suggested that a 3-factor solution best fit the data. A confirmatory factor analysis demonstrated acceptable fit of the final model, although item 14 loaded poorly on its factor. The sample size was on the lower end of what is generally recommended. This study could not confirm that the Mini-BESTest is unidimensional. Gait items were dispersed over all factors, indicating that they may reflect different constructs. Nonetheless, as there arguably is no clinical balance test superior to the Mini-BESTest today, we recommend using the total score for assessing gross balance in this population and individual items to identify specific weaknesses. Moreover, dual tasks should be assessed separately because they are an important aspect of balance control in people with PD, reflected in only one item of the test.