Measuring and Explaining Interviewer Effects in Centralized Telephone Surveys

Abstract

Estimates of interviewer effects on survey statistics are examined from nine surveys conducted over a six-year period at the Survey Research Center. Estimates of intraclass correlations associated with interviewers are found to be unstable, given the number of interviewers (30-40) used on most surveys. This finding calls into question inference from earlier studies of interviewer effects. To obtain more reliable information about magnitudes of interviewer effects, generalized effects are constructed by cumulating estimates over statistics and surveys. These generalized correlations are found to be somewhat smaller than those reported in the past literature. Few differences in generalized interviewer effect measures are found between open and closed questions or between factual and attitudinal questions. Small reductions in effects were obtained when a Computer Assisted Telephone Interviewing (CATI) system was used; there was some evidence of elderly respondents being more susceptible to interviewer effects; the number and type of second responses to open questions were affected by interviewer behavior; and changes in interviewing techniques reduced interviewer effects. Robert M. Groves is an Associate Research Scientist in the Survey Research Center and an Associate Professor in the Department of Sociology at The University of Michigan. Lou J. Magilavy is a Ph.D. candidate in the Department of Biostatistics at The University of Michigan. Data in this article were collected in projects supported by the National Science Foundation and the National Center for Health Statistics. The views in this article do not necessarily represent those of either organization. An earlier version of this article was presented at the meetings of the International Statistical Institute, 1983. The authors appreciate the comments of Leslie Kish, Stanley Presser, and Howard Schuman on earlier versions, and extend their gratitude to Richard Curtin for access to the data from the Surveys of Consumer Attitudes. Public Opinion Quarterly Vol. 50:251-266 ? 1986 by the American Association for Public Opinion Research Published by The University of Chicago Press 0033-362X/86/0050-251/$2.50 This content downloaded from 207.46.13.33 on Sun, 20 Nov 2016 04:16:07 UTC All use subject to http://about.jstor.org/terms 252 ROBERT M. GROVES AND LOU J. MAGILAVY marizes six years of research in the measurement of interviewer effects in centralized telephone surveys, on topics ranging from health experiences to political attitudes. We attempt to address two weaknesses in past research-problems of estimation (obtaining precise and accurate estimates of interviewer effects) and problems of explanation (learning what design features affect the magnitude of interviewer effects).