Number Of Response Categories Research Articles

Interchangeability between factor analysis, logistic IRT, and normal ogive IRT.

In existing studies, it has been argued that factor analysis (FA) is equivalent to item response theory (IRT) and that IRT models that use different functions (i.e., logistic and normal ogive) are also interchangeable. However, these arguments have weak links. The proof of equivalence between FA and normal ogive IRT assumes a normal distribution. The interchangeability between the logistic and normal ogive IRT models depends on a scaling constant, but few scholars have examined whether the usual values of 1.7 or 1.702 maximize interchangeability. This study addresses these issues through Monte Carlo simulations. First, the FA model produces almost identical results to those of the normal ogive model even under severe nonnormality. Second, no single scaling constant maximizes the interchangeability between logistic and normal ogive models. Instead, users should choose different scaling constants depending on their purpose in using a model and the number of response categories (i.e., dichotomous or polytomous). Third, the interchangeability between logistic and normal ogive models is determined by several conditions. The interchangeability is high if the data are dichotomous or if the latent variables follow a symmetric distribution, and vice versa. In summary, the interchangeability between FA and normal ogive models is greater than expected, but that between logistic and normal ogive models is not.

Performance of Estimation Methods in Bifactor Models with Ordered Categorical Data

Simulation studies are needed to investigate how many score categories are sufficient to treat ordered categorical data as continuous, particularly for bifactor models. The current simulation study aims to address such needs by investigating the performance of estimation methods in the bifactor models with ordered categorical data. Results support the application of categorical estimators to the ordered categorical data rather than the continuous estimators when sample size is large (750). Otherwise, an applied researcher may have to use the continuous estimators due to the model non-convergence. In this circumstance, the number of response categories needs to be at least 6 to avoid the rejection of correctly specified bifactor models by the chi-square test and estimate the model parameters accurately. The robust maximum likelihood (MLR) may be chosen among two continuous estimators due to its smaller type I error rate for the chi-square test than the ML. Practical implications of study findings are discussed.

Brief Research Report: Effects of Sampling Error and Categorization on Estimation of Measure of Sampling Adequacy

The second version of Kaiser’s Measure of Sampling Adequacy (MSA2) has been widely applied to assess the factorability of data in psychological research. The MSA2 is developed in the population and little is known about its behavior in finite samples. If estimated MSA2s are biased due to sampling errors, misleading inferences on the factorability of data are likely to occur. This study investigates the effect of sampling error on MSA2 estimation by systematically examining the accuracy and fluctuations of MSA2 estimates with simulated continuous and ordered categorical data. Features manipulated included the number of factors, number of variables per factor, factor loading, inter-factor correlations, sample size, number of response categories, skewness of variables, and types of correlation analyzed, Pearson and polychoric correlations. Results revealed that the MSA2s were underestimated due to the effect of sampling error. Severely biased MSA2 occurred when analyzing a large number of weakly correlated variables with insufficient participants. The underestimation of MSA2 became worse with categorized data. Polychoric correlations yielded slightly more accurate but relatively unstable MSA2 estimates compared to Pearson correlations. In practice, researchers need to bear in mind the downward bias of MSA2 estimates and interpret the value of sample MSA2s with caution.

Polytomous scoring correction and its effect on the model fit: A case of item response theory analysis utilizing R

In item response theory, the number of response categories used in polytomous scoring has an effect on the fit of the model used. When the initial scoring model yields unsatisfactory estimates, corrections to the initial scoring model need to be made. This exploratory descriptive study used response data from Take Home Exam (THE) participants in the Statistical Methods I course organized by the Open University, Indonesia, in 2022. The stages of data analysis include coding the rater’s score; analyzing frequency; analyze the fit of the model based on graded, partial, and generalized partial credit models; analyze the characteristic response function (CRF) curve; scoring correction (rescaling); and re-analyze the fit of the model. The fit of the model is based on the chi-square test and the root mean square error of approximation (RMSEA). All model fit analyzes were performed by using R. The results revealed that scoring corrections had an effect on model fit and that the partial credit model (PCM) produced the best item parameter estimates. All results and their implications for practice and future research are discussed.

Rasch rating scale analysis of the survey of attitudes toward statistics

Students in every discipline in higher education take at least one course in statistics. Therefore, it is necessary to enhance students’ understanding of statistics and their achievement in such courses by considering several factors that might contribute to this enhancement. Students’ attitudes toward statistics are a critical factor that influences their performance in statistics courses, and thus an accurate measurement of attitudes is needed. The survey of attitudes toward statistics (SATS-36) is widely used in measuring attitudes toward statistics; thus, it is important to ensure that its items accurately assess this construct. Therefore, the purpose of the current study was to validate this survey when administered to a convenience sample of 423 university students. Using the Rasch rating scale model, the current study examined the dimensionality, item fit to the Rasch model, item and person reliabilities, functionality of response categories, and distribution of the SATS-36 items along the attitudes toward statistics continuum.<br /> The findings revealed excellent item and person reliabilities (greater than 0.90) and the uni-dimensionality of the survey. Additionally, all items were closely aligned with the respondents, and the response categories were well-functioning as each category had more than 10 observations and outfit statistics were all low. However, some improvements were suggested. All items on the effect subscale and some others from different subscales need to be altered in content, deleting three items (two from the value subscale and one from the difficulty subscale) and adding more items to have a better distribution of items along the continuum. Finally, the number of response categories is recommended to be reduced to five instead of seven to have a more efficient rating scale. The findings of the current study imply that even though great care has been taken in the development of this survey, examining the quality of its items and the utility of its rating scale in new settings, and using different validation approaches is necessary.

Evaluations of the sum-score-based and item response theory-based tests of group mean differences under various simulation conditions.

The use of patient-reported outcomes measures is gaining popularity in clinical trials for comparing patient groups. Such comparisons typically focus on the differences in group means and are carried out using either a traditional sum-score-based approach or item response theory (IRT)-based approaches. Several simulation studies have evaluated different group mean comparison approaches in the past, but the performance of these approaches remained unknown under certain uninvestigated conditions (e.g. under the impact of differential item functioning (DIF)). By incorporating some of the uninvestigated simulation features, the current study examines Type I error, statistical power, and effect size estimation accuracy associated with group mean comparisons using simple sum scores, IRT model likelihood ratio tests, and IRT expected-a-posteriori scores. Manipulated features include sample size per group, number of items, number of response categories, strength of discrimination parameters, location of thresholds, impact of DIF, and presence of missing data. Results are summarized and visualized using decision trees.

The number of response categories in ordered response models.

The choice of the number m of response categories is a crucial issue in categorization of a continuous response. The paper exploits the Proportional Odds Models' property which allows to generate ordinal responses with a different number of categories from the same underlying variable. It investigates the asymptotic efficiency of the estimators of the regression coefficients and the accuracy of the derived inferential procedures when m varies. The analysis is based on models with closed-form information matrices so that the asymptotic efficiency can be analytically evaluated without need of simulations. The paper proves that a finer categorization augments the information content of the data and consequently shows that the asymptotic efficiency and the power of the tests on the regression coefficients increase with m. The impact of the loss of information produced by merging categories on the efficiency of the estimators is also considered, highlighting its risks especially when performed in its extreme form of dichotomization. Furthermore, the appropriate value of m for various sample sizes is explored, pointing out that a large number of categories can offset the limited amount of information of a small sample by a better quality of the data. Finally, two case studies on the quality of life of chemotherapy patients and on the perception of pain, based on discretized continuous scales, illustrate the main findings of the paper.

Impact of DIF on General Factor Mean Comparisons for Bifactor, Ordinal Data

A simulation study was conducted to explore the impact of differential item functioning (DIF) on general factor difference estimation for bifactor, ordinal data. Common analysis misspecifications in which the generated bifactor data with DIF were fitted using models with equality constraints on noninvariant item parameters were compared under data generation conditions varying by sample size, number of response categories, effect size of the general factor mean difference, and the position and the magnitude of DIF. Estimation bias in the general factor mean difference resulting from ignoring DIF mainly depended on the type of parameters with DIF. When ignoring DIF in threshold parameters, very substantial estimation bias was produced, and Type I error rates (powers) were inflated for many conditions. In contrast, when ignoring DIF in general factor loadings, a little estimation bias was produced; when ignoring DIF in specific factor loadings, almost no estimation bias was produced. All Type I error rates fell within required limits when DIF was present in factor loadings, and powers were not influenced by ignoring DIF in factor loadings.

Reliability and Validity of Compulsive Buying Scale Without Middle Point

Objective: The study aims at finding out the affect of reliability and validity for the compulsive buying behavior scale by Valence, d’ Astou’s and Fortier Scale without middle point.
 Methodology: Responses across 5 to 9 point scales are obtained to calculate the reliability and validity of compulsive buying behavior scale by Valence, d’ Astou’s and Fortier Scale. Cronbach’s alpha is used to measure the internal reliability of the scale. To compare the reliability coefficients among different scale points Feldt test and Hakstian-Whalen test are used. Convergent validity for measuring inter-correlations between scales with different numbers of response categories is used and Fisher’s –r to –z transformation is used to test population correlation coefficient.
 Conclusion: From study it is concluded that there is no change in reliability and validity when the middle point is dropped from the compulsive buying scale.

Comparison of Reliability of Psychological Tests for Elementary Students by the Number of Response Categories on the Likert Scale (3, 4, 5, 6)

Comparison of Reliability of Psychological Tests for Elementary Students by the Number of Response Categories on the Likert Scale (3, 4, 5, 6)

General Factor Mean Difference Estimation in Bifactor Models with Ordinal Data

ABSTRACT A simulation study was conducted to explore robustness of general factor mean difference estimation in bifactor measurement models with ordered-categorical measures. Common analysis misspecifications in which generated bifactor data were fitted using a unidimensional model and/or ordered-categorical data were treated as continuous were compared under generation conditions varying by sample size, number of response categories, effect size of the general factor mean difference, and loadings on the specific factors. Fitting bifactor data using unidimensional models resulted in estimation bias in the general factor mean difference, with magnitude largely determined by degree of unidimensionality and size of the general factor mean difference. Although bifactor models produced less estimation bias, estimates were less precise. Modeling the data as categorical and employing the WLSMV estimator provided somewhat more power, whereas modeling the data as continuous and applying MLR produced relatively less estimation bias when unidimensional models were specified for strongly bifactor data.

Variability of residents\u2019 ratings of faculty\u2019s teaching performance measured by five- and seven-point response scales

BackgroundMedical faculty’s teaching performance is often measured using residents’ feedback, collected by questionnaires. Researchers extensively studied the psychometric qualities of resulting ratings. However, these studies rarely consider the number of response categories and its consequences for residents’ ratings of faculty’s teaching performance. We compared the variability of residents’ ratings measured by five- and seven-point response scales.MethodsThis retrospective study used teaching performance data from Dutch anaesthesiology residency training programs. Questionnaires with five- and seven-point response scales from the extensively studied System for Evaluation of Teaching Qualities (SETQ) collected the ratings. We inspected ratings’ variability by comparing standard deviations, interquartile ranges, and frequency (percentage) distributions. Relevant statistical tests were used to test differences in frequency distributions and teaching performance scores.ResultsWe examined 3379 residents’ ratings and 480 aggregated faculty scores. Residents used the additional response categories provided by the seven-point scale – especially those differentiating between positive performances. Residents’ ratings and aggregated faculty scores showed a more even distribution on the seven-point scale compared to the five-point scale. Also, the seven-point scale showed a smaller ceiling effect. After rescaling, the mean scores and (most) standard deviations of ratings from both scales were comparable.ConclusionsRatings from the seven-point scale were more evenly distributed and could potentially yield more nuanced, specific and user-friendly feedback. Still, both scales measured (almost) similar teaching performance outcomes. In teaching performance practice, residents and faculty members should discuss whether response scales fit their preferences and goals.

Bayesian Inference for IRT Models with Non-Normal Latent Trait Distributions

Normality of latent traits is a common assumption made when estimating parameters for item response theory (IRT) models, but this assumption may be violated. The purpose of this research was to present a new Markov chain Monte Carlo (MCMC) method for ordinal items with flexible latent trait distributions (i.e., skewed and bimodal). Specifically, the Davidian curve (DC) was used to approximate the distribution of latent traits. The performance of the proposed MCMC algorithm with DCs was evaluated via a simulation study and compared with an EM method using DCs that is available in the “mirt” package (Chalmers, 2012). The manipulated factors included the number of response categories, sample size, and the shape of the latent trait distribution. The Hanna-Quinn (HQ) criterion was used to choose the best DC order. Results indicated that when informative priors were used, the MCMC algorithm with DCs could fit a flexible distribution well and the method provided good parameter estimates which, under some circumstances, had lower bias and RMSE than the EM method.

Performance simulations for categorical mediation: Analyzing khb estimates of mediation in ordinal regression models

Standard mediation techniques for fitting mediation models cannot readily be translated to nonlinear regression models because of scaling issues. Methods to assess mediation in regression models with categorical and limited response variables have expanded in recent years, and these techniques vary in their approach and versatility. The recently developed khb technique purports to solve the scaling problem and produce valid estimates across a range of nonlinear regression models. Prior studies demonstrate that khb performs well in binary logistic regression models, but performance in other models has yet to be investigated. In this article, we evaluate khb‘s performance in fitting ordinal logistic regression models as an exemplar of the wider set of models to which it applies. We examined performance across 38,400 experimental conditions involving sample size, number of response categories, distribution of variables, and amount of mediation. Results indicate that under all experimental conditions, khb estimates the difference (mediation) coefficient and its associated standard error with little bias and that the nominal confidence interval coverage closely matches the actual. Our results suggest that researchers using khb can assume that the routine reasonably approximates population parameters.

Evaluating the Fit of Sequential G-DINA Model Using Limited-Information Measures.

Limited-information fit measures appear to be promising in assessing the goodness-of-fit of dichotomous response cognitive diagnosis models (CDMs), but their performance has not been examined for polytomous response CDMs. This study investigates the performance of the M ord statistic and standardized root mean square residual (SRMSR) for an ordinal response CDM-the sequential generalized deterministic inputs, noisy "and" gate model. Simulation studies showed that the M ord statistic had well-calibrated Type I error rates, but the correct detection rates were influenced by various factors such as item quality, sample size, and the number of response categories. In addition, the SRMSR was also influenced by many factors and the common practice of comparing the SRMSR against a prespecified cut-off (e.g., .05) may not be appropriate. A set of real data was analyzed as well to illustrate the use of M ord statistic and SRMSR in practice.

PNS233 A NEWLY DEVELOPED POWER PROCEDURE FOR SAMPLE SIZE PLANNING IN COA VALIDATION STUDIES

No procedure exists for computing statistical power and sample size requirements for content validity analyses. Such analyses constitute the first step in psychometric validation of clinical outcome assessments (COAs). Power computations for content validity are important for optimizing domain specification and scoring, which ultimately help produce treatment-responsive scores. In this work a new power procedure is developed and validated that can account for the number of items, number of response categories per item, the complexity of the conceptual framework, and a pre-specifiable effect size. Power is derived for model fit statistics commonly used in content validity analyses. This presentation will describe the theoretical power procedure, review results of a simulations study validating the power procedure, and discuss how manipulating items and response categories can allow control of power. The validity of this power procedure was assessed in a simulation study designed to test the agreement between theoretical and empirical power. Factors manipulated in the simulation included the number of items, the effect size, and sample size. For each condition the power to reject a unidimensional model fitted to data generated from a multidimensional structure was assessed. Choosing between such models is a common task in COA development as these models help ascertain whether unidimensional or domain scores are required. The correlation between theoretical and empirical power was greater than or equal to 0.998 in all simulation conditions. This agreement validated the power procedure. A method for determining power and sample size in COA validation studies has been developed. These power computations will help align study sample size with the statistical demands of planned content validity analyses.

Does a difference in the number of response categories change the results for ACSI in the mobile phone sector?

This study aims to contribute to making customer satisfaction measurement more efficient in the mobile phone sector. For this purpose, data were collected from 3,342 mobile phone users living in five different cities in Turkey. A self-administered questionnaire was used in this study. Five different surveys were used, with the same questions but with different categories of response (4-, 5-, 6-, 7-, and 10-point scales). The research instrument items were adapted from the American customer satisfaction index (ACSI). With the dataset of each response category, the models were installed and subsequently compared. The originality of the study is due to the comparison of the models created with different datasets. According to the findings obtained, different response categories did not make a considerable difference in the measurement of customer satisfaction.

Radio buttons in web surveys: Searching for alternatives

Web surveys are struggling to attract and retain respondents due to high burden and competition for the users’ attention. One possible solution to this issue is the improvement of the visual design of surveys. In this article, we evaluate the impact of visual aids such as smiley faces, stars, hearts, and thumbs as alternatives to traditional radio buttons. We use an experimental design in a nonprobability online survey to investigate how the new designs compare with radio buttons and how the results might interact with device used for completion (PC vs mobile), the use of labels, the type of response scale (bipolar vs unipolar), and the number of response categories (5 vs 7 point). While we do not find big differences in response, quality and experience, there seem to be some indication that the use of smiley faces leads to worse data quality.

Validation of the 2012 European Social Survey Measurement of Wellbeing in Seventeen European Countries

The measurement of wellbeing provides an important indicator of the welfare of nations and presents opportunities for policy making. Researchers generally share the view of wellbeing as a multidimensional concept. The 2012 European Social Survey (ESS) measurement of personal and social wellbeing, a combination of theoretical models and evidence from statistical analysis, is defined as a six-dimensional construct: evaluative wellbeing, emotional wellbeing, functioning, vitality, community wellbeing and supportive relationships. In this paper, the proposed theoretical structure is investigated and the psychometric properties of the measure are assessed for 17 European countries. This involved splitting each country’s sample randomly into halves and performing Exploratory Factor Analysis (EFA) on the first half-samples. EFA resulted in a four-factor solution for Germany, Netherlands, Portugal, Slovenia, Spain, Switzerland and the UK, a five-factor solution for Belgium, Finland, France, Ireland, Norway, Poland, Russian Federation and Sweden, and a six-factor solution for Denmark and Hungary. These results were supported by Confirmatory Factor Analysis (CFA) performed on the second half-samples. Subscales were constructed based on analysis of the total samples, applying a simple transformation in order to deal with the different number of response categories used for the wellbeing items. Reliabilities and internal consistencies were investigated. Although the definition of each subscale differs from the proposed structure and across countries, the analysis did produce reliable and valid summary measures (subscales) of wellbeing for informing social policy in each country.

Confirmatory Factor Analysis of Ordinal Variables: A Simulation Study Comparing the Main Estimation Methods

In order to obtain evidences about construct validity through Confirmatory Factor Analysis, it has been usual treating Likert-type scales as if they were continuous variables measured on an interval scale. Therefore, Maximum Likelihood estimation method has been broadly applied, but in turn it implies problems concerning both Pearson correlations and skewness in the distribution of responses to items. In this simulation study we analyse —through χ2, Type I error, and power— correctly specified and misspecified models comparing five estimation methods (Maximum Likelihood —ml—, Robust Maximum Likelihood —rml—, Weighted Least Squares —wls—, nweighted Least Squares —uls— and Robust Unweighted Least Squares —ruls—) in relation to the models features: number of factors, number of response categories, items’ skewness, and sample size. We advise using ruls estimation method, in which polychoric correlations are implied.Keywords: Confirmatory Factor Analysis, Likert-type scales, estimation methods, Type I error, power.