The Measurement of Values in Surveys: A Comparison of Ratings and Rankings

Abstract

Social values are most commonly measured using ranking techniques, but there is scarcity of systematic comparisons between rankings and other approaches to measuring values in survey research. On the basis of data from the 1980 General Social Survey, this article evaluates the comparability of results obtained using rankings and ratings of valued qualities. The comparison focuses on (1) the ordering of aggregate value preferences and (2) the measurement of individual differences in latent value preferences. The two methods are judged to be similar with-respect to ordering the aggregate preferences of the sample, but dissimilar with regard to the latent variable structure underlying the measures. Duane F. Alwin is Research Scientist and Professor of Sociology, Survey Research Center, Institute for Social Research, The University of Michigan. Jon A. Krosnick is Assistant Professor of Psychology, The Ohio State University. This article is revised version of paper presented at the annual meetings of the American Association for Public Opinion Research, May 17-20, 1984. An earlier version of the paper was reprinted as Technical Report No. 40, General Social Survey, NORC, Chicago. The research was supported by grant from the National Institute of Mental Health (MH-37289). An important debt is owed NORC and the staff of the General Social Survey for cooperation in conducting this research. The authors wish to thank Frank Andrews, David Jackson, Melvin Kohn, and Tom Smith for helpful comments on earlier drafts and Lynn Dielman and Marion Wirick for help in manuscript preparation. Address all correspondence to Duane F. Alwin, Survey Research Center, Institute for Social Research, P.O. Box 1248, Ann Arbor, MI 48106. Public Opinion Quarterly Vol. 49:535-552 C) by the American Association for Public Opinion Research Published by Elsevier Science Publishing Co, Inc. 0033-362X/85/0049-535/$2.50 This content downloaded from 157.55.39.116 on Sat, 11 Jun 2016 05:15:19 UTC All use subject to http://about.jstor.org/terms 536 DUANE F. ALWIN AND JON A. KROSNICK have number of problems tied to their degree of difficulty, and some have suggested that other techniques, such as ratings, could serve as effective alternatives. In this article, we compare the rating and ranking approaches to measuring values using data on parental orientations toward children from randomized split-ballot experiment carried out in the 1980 General Social Survey. Before reporting our findings, we place the work in context by reviewing the literature that has discussed the relative advantages and disadvantages of rating and ranking techniques for measuring values. The Measurement of Values: Ratings vs. Rankings Value researchers have consistently argued on theoretical grounds that ranking techniques provide the most appropriate conceptual mapping to conceptions of values. For example, Rokeach (1973:6) points out that values are often thought to be inherently comparative and competitive, and thus the choice nature of the ranking task fits nicely with this conceptualization. Also, Kohn (1977:19) observes that the ranking approach to measuring values is demanded by their very nature, in that a central manifestation of value is to be found in choice. This point of view is validated to some extent by the sheer prevalence of the use of rankings to measure values (e.g., Allport et al., 1960; Kluckholn and Strodtbeck, 1961; Lenski, 1961; Bengston, 1975). Despite these arguments regarding the conceptual mapping of ranking techniques to the concept of values, this approach has number of significant practical drawbacks when used in survey research. First, rankings are often difficult and taxing for respondents, demanding considerable cognitive sophistication and concentration. This is particularly problematic when the list of concepts to be ranked is lengthy (Rokeach, 1973:28; Feather, 1973:228). Second, the use of ranking techniques is time-consuming and may therefore be more expensive to administer (Munson and McIntyre, 1979:49). Third, since rankings often require the use of visual aids, or show-cards, it is difficult to gather such information using telephone methods of data collection (Groves and Kahn, 1979:122-33). And finally, the sum of the ranks for any individual respondent equals constant, so there is linear dependency among the set of ranked items (Clemans, 1966; Jackson and Alwin, 1980). Consequently, it may not always be possible to employ conventional statistical techniques in the analysis of the latent content of ranked preference data. In contrast to rankings, rating scales are easy to present to respondents. Munson and McIntyre (1979:49) estimate that ranking tasks take three times longer than similar rating tasks and involve considerable decrease in respondent burden. Also, ratings can readily be administered over the This content downloaded from 157.55.39.116 on Sat, 11 Jun 2016 05:15:19 UTC All use subject to http://about.jstor.org/terms RATINGS AND RANKINGS 537 telephone and do not involve the linear dependency problem inherent in rankings. So ratings have none of the major disadvantages of rankings and might therefore be good substitute for survey measurements of