Survey Context Effects and Implications for Validity: Measuring Political Discussion Frequency in Survey Research

Abstract

Although political discussion behavior is an important area of political communication research, analysis of the reliability and validity of political discussion survey measures has only recently begun. This study uses panel survey data to examine the effects of survey context on self-reported political discussion frequency measures (e.g., general political discussion, discussion about the economy), the moderating influence of individual difference variables, and implications for measurement validity. A quasi-experiment demonstrates that including a large (versus small) quantity of preceding issue-relevant questions leads to higher reports of political and economic discussion frequency, and this effect is greater for individuals with higher levels of political and economic interest, respectively. Results of a survey experiment reveal that inclusion of a preceding thought-listing question about the economy produces higher reports of economic talk frequency, but only among those who possess relatively expansive conceptualizations of the economy. Such survey context effects suggest problems with the construct validity of self-reported political discussion frequency measurements. Potential consequences of survey context for concurrent and predictive validity are assessed by examining relationships between discussion frequency and known correlates (e.g., education, interest) and outcomes (e.g., current affairs knowledge) of political talk. Results provide tentative evidence that discussion measures placed after a large (versus small) quantity of issue-relevant questions may exhibit better criterion validity (quasi-experiment), whereas the order of a thought-listing question does not appear to impact criterion validity (experiment). Results of this study clearly underscore the need for additional research on self-report measures of political behavior, including political discussion frequency.