Response Scale Format Research Articles

A Mixture Model for the Analysis of Categorical Variables Measured on Five-point Semantic Differential Scales

Ordered response scales are often used in questionnaires to measure individuals' attitudes or perceptions. Among different response scale formats, we focus on multi-point semantic differential scales, requiring the respondent to position himself/herself on a rating between two bipolar adjectives. The obtained rating data require appropriate statistical models. We resort to the CUM model (Combination of a discrete Uniform and a - linearly transformed - Multinomial random variable), recently proposed in the framework of the CUB (Combination of discrete Uniform and shifted Binomial random variables) class of models. CUM is also suited to all the ordinal response scales with a middle “indifference” option. In the seminal paper on CUM, the methodological approach was developed for an odd number m of response categories, while simulations, case studies and implementation in R were limited to m = 7. The objective of this paper is to extend the original proposal and investigate the model performance in the case of m = 5, which often arises in real situations. The R functions for fitting a CUM model with m = 5 are implemented and made available; simulation studies are developed and compared with results obtained for m = 7 and a case study concerned with the evaluation of museums' visitor experience is proposed.

Do simpler item wording and response scales reduce acquiescence in personality inventories? A survey experiment

Acquiescence (“yea-saying”) presents a major challenge in personality assessment via questionnaire. Cognitive load when responding to items seems to be related to a greater tendency to acquiesce. The aim of this study was to experimentally investigate whether reducing the complexity of personality items through simplified item wording and response scales with fewer options reduces acquiesce. Based on a heterogeneous sample of almost 6000 respondents, we probed acquiescent responding on the 60-item Big Five Inventory-2 (BFI-2) in a 2 × 2 design, comparing (a) the original items and simplified versions and (b) the original 5-point and a 4-point response scale without a midpoint. Results show that the response scale format did not affect acquiescent responding. Contrariwise, item simplification reduced the variance attributable to acquiescence in the total sample. Moderation analysis revealed stronger effects for lower-educated respondents.

Response scale overstretch: linear stretching of response scales does not ensure cross-project equivalence in harmonised data

Differences in the response-scale formats constitute a major challenge for ex-post harmonisation of survey data. Linear stretching of original response options onto a common range of values remains a popular response to format differences. Unlike its more sophisticated alternative, simple stretching proves readily applicable without requiring assumptions regarding scale length or access to auxiliary information. The transformation only accounts for response scale length, ignoring all other aspects of measurement quality, which makes the equivalence of harmonised survey variables questionable. This paper focuses on the inherent limitations of linear stretching based on a case study focusing on the measurements of trust in the European Parliament by the Eurobarometer and the European Social Survey—8 timewise corresponding survey waves in 14 European countries (2004–2018). Our analysis demonstrates that the linear stretch approach to harmonising question items with different underlying response scale formats does not make the results of the two surveys equivalent. Despite harmonisation, response scale effects are retained in the distributions of output variables.

Enhancing validity, reliability and participation in self-reported health outcome measurement for children and young people: a systematic review of recall period, response scale format, and administration modality

IntroductionSelf-report is the gold standard for measuring children’s health-related outcomes. Design of such measures is complex and challenging. This review aims to systematically appraise the evidence on recall period, response scale format, mode of administration and approaches needed to enable children and young people < 19 years to participate in valid and reliable self-reporting of their health outcomes.MethodPsycInfo, Medline, CINAHL and Embase were searched from 1 January 1990 to 15 March 2020, and citation searching undertaken in Scopus. Articles were included if they were primary research or case reports of ≥ 3 participants reporting the following: recall period, response scale selection, administration modality. Quality was assessed using QualSyst, and results synthesised narratively. This review was conducted and reported according to PRISMA guidelines.Results81 of 13,215 retrieved articles met the inclusion criteria. Children < 5 years old cannot validly and reliably self-report health outcomes. Face scales demonstrate better psychometric properties than visual analogue or Likert scales. Computerised and paper scales generally show equivalent construct validity. Children prefer computerised measures. Children ≤ 7 years old think dichotomously so need two response options. Those > 8 years old can reliably use a 3-point scale.ConclusionThe results of this review have both clinical and research implications. They can be used to inform appropriate choice of PROM for use with CYP in the clinical setting. We also give eight recommendations for future development of self-reported outcome measures for children and young people.

Extremity in horizontal and vertical Likert scale format responses. Some evidence on how visual distance between response categories influences extreme responding

In four survey experiments we show that people generally answer more extremely to survey items presented in vertical versus horizontal Likert formats. Our findings suggest that this effect may be at least partly driven by differences in the visual range spanned by the response scale (i.e. the visual distance between endpoint response categories is larger in horizontal than in a vertical format). In addition, compared to traditional horizontal Likert data, vertical Likert data contain more variance, which is mainly non-substantive. As a result, data obtained with scale formats that have different distances between response categories (as is typically the case for vertical vs. horizontal formats) may lead to differences in measurement model parameter estimates like residual terms, and in some cases factor loadings and construct correlations. Based on these results, we provide recommendations on the use of response scale formats in online surveys, bearing in mind that several online survey tool providers promote the use of vertical Likert formats and even automatically change traditional horizontal formats of Likert-type items to vertical Likert formats when viewed on small screens (e.g., on mobile phones).

Assessing Mathematics Self-Efficacy

The present study tested whether a reduced number of categories is optimal for assessing mathematics self-efficacy among middle school students using a 6-point Likert-type format or a 0- to 100-point format. Two independent samples of middle school adolescents ( N = 1,913) were administered a 24-item Middle School Mathematics Self-Efficacy Scale using either a 101-point or a 6-point response scale format. The findings suggest that the two different response scale formats were optimally represented by a 4-point scale and supported when samples were pooled. Results provide tentative evidence that middle school students make use of only 4 scale points and that the items on this scale are best matched with adolescents with average to below-average mathematics self-efficacy. Implications for the measurement of self-efficacy and related motivation constructs are discussed, and replications with a 4-point scale using category descriptors for each scale point are needed.

The calibrated sigma method: An efficient remedy for between-group differences in response category use on Likert scales

The authors propose a procedure, labeled the calibrated sigma method, which is designed to correct for between-group differences in endorsement likelihood of response categories that are unrelated to the content of the items. The method is especially useful in cross-cultural research where group differences may reflect variation in scale usage rather than substantive differences. However, the procedure is also relevant in other situations, for example, when different data collection modes or different experimental manipulations affect respondents' perception of the meaning of the scale labels. The calibrated sigma method uses information derived from heterogeneous control items (calibration items) to reweight the responses to substantive items in a group-specific way. The advantages of the calibrated sigma method are that it avoids the arbitrariness in the assignment of particular numerical values to response categories; that it is compatible with the linear model, which is used by most marketing researchers; and that it does not require the use of complex nonlinear models involving the estimation of many additional measurement model parameters. The authors validate the calibrated sigma method on a simulated cross-linguistic data set pertaining to 12 different languages; an empirical data set collected from respondents of the same nationality but from two different language groups; and an experimental data set consisting of responses to two different response scale formats. The findings demonstrate that the proposed procedure controls for artefactual scale use differences across groups but does not eliminate substantive differences. It is particularly efficient for marketing research agencies, panel providers and other marketing researchers who analyze surveys involving multiple language groups, different scale formats, multiple modes of data collection, or different manipulations affecting the meaning of the response category labels.

Application Driven Theory: Rigorously Combining Applied and Basic Research Relevant To Accounting and Marketing

Application Driven Theory: Rigorously Combining Applied and Basic Research Relevant To Accounting and Marketing

Development of a patient satisfaction questionnaire for HIV/AIDS patients in Nigeria

The aim of this study was to develop and validate a questionnaire for assessing HIV-infected patients' satisfaction with pharmaceutical care received in Nigerian HIV clinics. Questionnaire's items were selected based on similar published studies and designed on a 5- point Likert response scale. Face and content validity, feasibility, factorial validity, reliability, and construct validity were evaluated. The instrument's feasibility was assessed in a secondary health care facility (St. Charles Borromeo Hospital Onitsha) and validated in a tertiary health care facility (University of Nigeria Teaching Hospital Enugu). Factor analysis used principal components and varimax rotation. Reliability was established using internal consistency with Cronbach's alpha. Convergent and discriminant validity were determined using Spearman's rho correlation. A self-administered 16-item questionnaire in 5-point Likert response scale format was developed. Questionnaire evaluated cumulative experience of patients with comprehensive pharmaceutical care practice in pharmacies of HIV clinics. Eighty questionnaires were collected for pilot test while four hundred questionnaires were retrieved for the validity test. Factor analysis resulted in four factors: 'Interpersonal/Professional relationship with pharmacist', 'patient counseling', 'drug information' and 'managing therapy', with a cumulative variance of 56.7%. Cronbach's alpha for the whole questionnaire was 0.85, and 0.81, 0.66, 0.67 and 0.72 for the four factors, respectively. Four items used for convergent and discriminant validity showed convergence between the related items and variance between the unrelated items. The questionnaire developed is a reliable and valid instrument for assessing patient satisfaction with pharmaceutical care in HIV clinics in Nigeria. Further research is needed to expand the instruments' robustness.

Rasch analysis of the International Wellbeing Index in older adults

The International Wellbeing Index is a measure of general quality of life formed by two scales: the Personal Wellbeing Index (PWI) and the National Wellbeing Index (NWI). This paper studies the psychometric properties of the PWI and NWI, using Rasch analysis and classic psychometric methods. The PWI and NWI were applied to a representative sample of 1106 community-dwelling adults, aged 60 years and over, residing in Spain. Mean ± standard deviation age was 72.07 ± 7.83 years and 56.3% were women. Five PWI items (achieving in life, relationships, safety, community connectedness, and future security), and five NWI items (economic situation, state of environment, social conditions, business, and national security of the country) fitted the Rasch model. After adjusting the response scale format, satisfactory fit was obtained, with good reliability (person separation index of 0.91 for both the PSI and NWI), local independency of items, and strict unidimensionality. The measures showed adequate external construct validity with related measures. The PWI and the NWI, with fewer items and simpler response scale formats, provided valid and reliable linear measures in older adults, according to Rasch and classic psychometric analyses.

The Anchor Contraction Effect in International Marketing Research

In an increasingly globalized marketplace, it is common for marketing researchers to collect data from respondents who are not native speakers of the language in which the questions are formulated. Examples include online customer ratings and internal marketing initiatives in multinational corporations. This raises the issue of whether providing responses on rating scales in a person's native versus second language exerts a systematic influence on the responses obtained. This article documents the anchor contraction effect (ACE), the systematic tendency to report more intense emotions when answering questions using rating scales in a nonnative language than in the native language. Nine studies (1) establish ACE, test the underlying process, and rule out alternative explanations; (2) examine the generalizability of ACE across a range of situations, measures, and response scale formats; and (3) explore managerially relevant and easily implementable corrective techniques.

Variations in Response Style Behavior by Response Scale Format in Attitude Research

Studies concerning the impact of the length of response scales on the measurement of attitudes have primarily focused on the method bias associated with question format. At the same time another line of research has focused on the issue of response styles that affect how respondents answer to attitude questions. So far, research has paid less attention to the issue of whether the length of the response scales is related to response styles. In this study, we explore if differences in length of the response scale (i.e., method factor) have differential effects in evoking extreme and midpoint response style behavior (i.e., style factor). Our hypotheses read as follows. As the number of response categories increases, we expect subjects to be more likely to exert extreme response style. Furthermore, we expect subjects to be more likely to adopt a midpoint response style when they are offered a middle response category. To investigate these hypotheses we developed a split ballot experiment in which the number of response categories is manipulated from 5 to 11 categories. Data are collected by a random sample, large-scale web survey which allows for random assignment to the experimental conditions. The results show clear evidence of extreme response style and moderate evidence of midpoint response style. Extreme response style is not affected by the length of response scales, whereas the exertion of midpoint response style only popped up in the longer scale versions.

Subjective Response Differences between Visual Analogue, Ordinal and Hybrid Response Scales

The Human Factors and Ergonomics (HF/E) discipline employs different subjective response scale formats to measure subjective phenomena (e.g., hazard perception). Per the psychometrics literature, different scale formats can yield different participant responses, which is a potential threat to validity in replication studies if response scales are not consistent across study iterations. If ordinal response scales (e.g., Likert scales) yield ordinal data and continuous response scales (e.g., Visual Analogue Scales) yield continuous data then it is inappropriate to substitute one response scale for the other if they potentially yield different responses for the same question. The current research compared mean participant ratings for the same question on VAS, Likert and Hybrid response scales and found that VAS scales had significantly lower mean ratings than Likert and Hybrid response scales. Two scale features, the number of anchors (5 or 9) and the scale length (10.0cm and 19.2cm), were varied and no significant main effects or interactions resulted. In conclusion, scale types, not scale features, produced significant mean participant rating differences. To support the validity of replication research, this paper also provided a response scale taxonomy based on the scale features studied herein, that can be used to classify and report different response scales. Implications for these results and future research directions are discussed.

Subjective Response Differences between Visual Analogue, Ordinal and Hybrid Response Scales

The Human Factors and Ergonomics (HF/E) discipline employs different subjective response scale formats to measure subjective phenomena (e.g., hazard perception). Per the psychometrics literature, different scale formats can yield different participant responses, which is a potential threat to validity in replication studies if response scales are not consistent across study iterations. If ordinal response scales (e.g., Likert scales) yield ordinal data and continuous response scales (e.g., Visual Analogue Scales) yield continuous data then it is inappropriate to substitute one response scale for the other if they potentially yield different responses for the same question. The current research compared mean participant ratings for the same question on VAS, Likert and Hybrid response scales and found that VAS scales had significantly lower mean ratings than Likert and Hybrid response scales. Two scale features, the number of anchors (5 or 9) and the scale length (10.0cm and 19.2cm), were varied and no significant main effects or interactions resulted. In conclusion, scale types, not scale features, produced significant mean participant rating differences. To support the validity of replication research, this paper also provided a response scale taxonomy based on the scale features studied herein, that can be used to classify and report different response scales. Implications for these results and future research directions are discussed.

Assessment of Physical Self-Concept in Adolescents with Intellectual Disability: Content and Factor Validity of the Very Short Form of the Physical Self-Inventory

The purpose of this study was to test the factor validity and reliability of the Very Short Form of the Physical Self-Inventory- (PSI-VSF) within a sample of adolescents with mild to moderate Intellectual Disability (ID). A total of 362 ID adolescents were involved in two studies. In Study 1, the content and format scale response of the PSI-VSF were adapted for adolescents with ID. This instrument was thus renamed PSI-VSF-ID and two versions with two alternative responses scales format, were developed: Likert and graphical. In Study 2, results provided support for: (1) the factorial validity and reliability; and (2) factorial invariance across gender, age, type of school placement and ID level of the PSI-VSF-ID associated with a graphical response scale format.

A Comparison of Physical Self-Perception Profile Questionnaire Formats: Structured Alternative and Ordered Response Scale Formats

This study assessed the performance of the original version of the Physical Self-Perception Profile (PSPP; Fox & Corbin, 1989) as compared to a revised version. University students (N = 259) participated in the study, each answering and comparing the clarity of the original and a revised PSPP. While participants found the revised version to be significantly clearer, it performed similarly to the original in select analyses of internal consistency and construct validity. The original structured alternative format did not have lower levels of socially desirable responding, as compared to the Likert-type ordered response scale format of the revised PSPP. Because the revised version was deemed clearer, performed similarly to the original on select psychometric analyses, and did not have any data loss, there is merit in further examining additional validity and reliability evidence for the revised PSPP.

Does Response Scale Format Affect the Answering of Personality Scales?

When developing a questionnaire, one puts much effort into item formulation. Whether the format of the corresponding response scales affects response behavior, however, has rarely been studied, to date. The present study investigates (1) the effects of the response scale direction (ranging from positive to negative vs. negative to positive) and (2) the match between numerical labeling and scale direction, i.e., assigning high numbers to the positive pole and low numbers to the negative pole or vice versa. These response scale effects were studied based on responses in the BFI-10, the short-scale version of the widely-used Big Five Inventory (BFI), assessing the Big Five dimensions of personality by two items each. Using a dependent sample design, subjects answered the 10 items using end-point labeled response scales ranging from the negative (labeled “1”) to the positive pole (labeled “8”) at Time 1. At Time 2 (approximately 3 weeks later), respondents were split into two conditions: The BFI-10 was administered again with an 8-point scale ranging from the positive to the negative pole. In the first condition, the positive pole was labeled “8” and the negative pole was labeled “1”; in the second condition, the positive pole was labeled “1” and the negative pole was labeled “8.” Results clearly support the notion that the direction of the response scale (Condition 1) does not affect response behavior. There were no differences in means, standard deviations, or in the intercorrelations patterns. However, there seems to be an intuitive match between the positive pole and high numerical labeling. When poles were counterintuitively labeled (Condition 2), significant differences could be identified for all analyses conducted.

A menu of potential reinforcers in a methadone maintenance program

This study demonstrates the use of paired comparisons and interval scaling techniques for measuring the relative priority of program privileges available at a methadone maintenance clinic. Fifteen methadone program privileges were combined in all possible pairs ( N = 105) on a reinforcer menu and administered to a group of 12 methadone patients and a second group of counselors ( N = 4). Data were converted to interval scales using the law of comparative judgment to form a quantitative continuum from least to most preferable. Free methadone, free dental service, and more take-homes were ranked highest in both groups; however, patients showed less differentiation in their preference for these privileges. Dose decreases were least preferred. Results are discussed in terms of their clinical applicability in identifying privileges for potential use in modifying the behavior of drug abusers. The method of paired comparisons has excellent psychometric properties and may offer some advantages over other response scale formats.

Skin cancer: Assessing perceived risk and behavioural attitudes

Abstract Measures of attitude concerning exposure to the sun, and judgements of risk and other beliefs concerning skin cancer and four other problems (stomach cancer, deafness, AIDS and sunstroke) were obtained from a questionnaire completed by 176 university students. Subjects also estimated the incidence of each problem using one of three response formats. The highest incidence estimates were obtained when subjects were asked to guess at an absolute number, and the lowest when they used a scale which differentiated between lower frequencies, while grouping higher frequencies into a single response category. The effect of the response scale format, however, did not generalize to other ratings of personal risk. Subjects' ratings of their personal risk, compared with their peers, showed an optimistic bias over the five problems as a whole, particularly for AIDS, but not reliably so in the case of skin cancer. Optimism was inversely related to the amount of thought given to each problem. Men and women did not differ overall in their optimism regarding their own risk of skin cancer. However, differences as a function of gender and optimism were found on various behavioural attitudes. The results are discussed in relation to a tendency of disregard base-rates in subjective risk judgements, unrealistic optimism, and implications for health education.