What's wrong with rating scales? Psychology's replication and confidence crisis cannot be solved without transparency in data generation

Abstract

AbstractQuantitative explorations of behaviour, psyche and society are common in psychology. This requires methods that justify the attribution of results to the measurands (the entities to be measured, e.g., in individuals) and that make the results' quantitative meaning publicly interpretable (e.g., for decision making). Do rating scales—psychology's primary methods to generate numerical data—meet these criteria? This article summarises selected epistemological and methodological problems of rating scales that arise, amongst others, from the intricacies of language‐based methods and from psychologists' challenges to distinguish their study phenomena from their means of exploring these phenomena. Failure to make this logical distinction entails that disparate scientific activities are conflated, thereby distorting scientific concepts and procedures. Rating scales promote such conflations because they serve both as description of the empirical study system (e.g., behaviours) and as symbolic study system (e.g., data variables), leaving the interpretation of each system and the mapping relations between them to raters' intuitive decisions. Verbal scales, however, have broad semantic fields of meanings, which are context‐sensitive and therefore interpreted differently, and which cannot logically match the quantitative meaning commonly ascribed to the numerical scores derived from them. The ease of using verbal descriptions as means of exploration drew psychologists' attention to the conceptual‐interpretive level, away from their actual study phenomena. This also led them to overlook key elements of data generation and measurement. The pragmatic necessity to analyse rating scores through between‐individual comparisons entailed the erroneous assumption that psychometrics and sample‐level statistics could enable measurement. Improving data analyses, as currently discussed, is therefore insufficient for overcoming psychology's crises of replication, confidence, validity and generalizability. Data generation methods are necessary that make the entire process—from the empirical study phenomena up to the results—fully transparent and traceable. This rigorous analysis of rating scales highlights important steps for future directions.