Survey Experiments in Sociology

Since its inception in February 2005, Methodology, the official journal of the European Association of Methodology, has been an online journal with a strong European vocation and dedication ‘‘to promote research and the development of empirical research methods in the fields of behavioral, social, educational, health and economic sciences, as well as in the field of evaluation research’’ (Ato & Eid, 2005; Ato & Hox, 2009). In 2009, the then editors of Methodology, Manuel Ato and Joop Hox, wrote an editorial in which they looked back at the first 4 years of Methodology (Ato & Hox, 2009). Now, 6 years later, two different editors, Peter Lugtig of Utrecht University and Nekane Balluerka of the University of the Basque Country, have taken over the editorship of Methodology. As Methodology celebrates its 10th birthday, it is time to look back and forward. Just like the editors before us, we firmly believe that the methodology underpinning the social sciences has many similarities across disciplines. Any differences between the sciences mostly reflect differences in the type of research questions asked, in research designs and data analysis practices, and in historical developments. Many of the articles published in Methodology have taken a cross-disciplinary perspective and many of the developments in social science methodology are similar across disciplines. A large number of articles in Methodology have explored the methodology of the Generalized Linear Model (Schweizer, 2010; Voelkle & McKnight, 2012). Simulation studies have, for example, focused on the properties of multilevel models (Bell, Morgan, Schoeneberger, Kromrey, & Ferron, 2014; Pacagnella, 2011). Latent variables in general, and how to model the measurements, is probably the topic that has been written about most, whether in the context of psychometrics (Balluerka, Plewis, Gorostiaga, & Padilla, 2014; Botella & Suero, 2012; Gonzalez-Betanzos & Abad, 2012), or of factor analysis to estimate equivalence across groups (Kankaras & Moors, 2011; Lugtig, Boeije, & Lensvelt-Mulders, 2012; Steinmetz, 2013). Other articles dealing with data collection methods have often focused on comparing different approaches to deal with violations of model assumptions (Blanca, Arnau, Lopez-Montiel, Bono, & Bendayan, 2013; Haupt, Losel, & Stemmler, 2014; Schmider, Ziegler, Danay, Beyer, & Buhner, 2010; Wolff Smith, & Beretvas, 2014). Apart from methodological advances in the field of data analysis, another development in social science research methodology is the availability of software tools. Here, a number of articles have discussed methods to implement new statistical modeling methodologies in software packages (e.g., Flora, 2011; Grilli & Variale, 2014). We envisage that in the coming years developments in this field of research will continue. The growing possibilities in terms of statistical modeling will increase demand for articles explaining which model should be used when, and how. For this reason, future issues of Methodology will feature a section with tutorial articles which show how to implement statistical modeling techniques in widely used software packages. To ensure such articles can be used by applied social science researchers, we will publish the data and code accompanying these articles. It is our continuing goal to feature special thematic issues edited by prominent researchers in their field. Since 2009, four special issues have been published. In 2009, Michael Eid and Fridtjof Nussbeck edited an issue to celebrate the 50th anniversary of the multitrait-multimethod matrix (Eid & Nussbeck, 2009). In 2010, Elmar Schlueter and Peter Schmidt provided an issue on ‘‘survey experiments’’ (Schlueter & Schmidt, 2010), while in the same year Andries Van der Ark and Jeroen Vermunt edited an issue on ‘‘new developments in missing data analysis’’ (Van der Ark & Vermunt, 2010). In 2013, Gordon Willis and Hennie Boeije contributed a special issue on the ‘‘systematic reporting of questionnaire development and pretesting’’ (Willis & Boeije, 2013). Readers of Methodology

Survey experiments are burgeoning in the study of religion/s. A survey experiment is an experiment conducted within a survey. Survey experiments address key limitations of nonexperimental surveys and experiments on convenience samples. Surveys are useful for speaking to patterns in a broader population but are limited in their ability to examine causal mechanisms (i.e., what exactly causes what). Lab experiments speak to causality, but using convenience samples limits generalizability (i.e., the extent to which the patterns can be expected to apply to a wider population). Survey experiments combine strengths from survey methods and experimental methods, providing leverage on both generalizability and causality. By using population-based sampling frames from survey research, survey experiments can generalize to a broader population. And by using experimental methods, specifically randomly assigning people to experimental conditions, survey experiments can establish whether one thing causes another. Although survey experiments are becoming more common in the social scientific study of religion, we are only beginning to scratch the surface of the range of questions and contexts that could be explored with survey experiments.

As survey experiments have become increasingly common in political science, some scholars have questioned whether inferences about the real world can be drawn from experiments involving hypothetical, text-based scenarios. In response to this criticism, some researchers recommended using realistic, context-heavy vignettes while others argue that abstract vignettes do not generate substantially different results. We contribute to this debate by evaluating whether incorporating contextually realistic graphics into survey experiment vignettes affects experimental outcomes. We field three original experiments that vary whether respondents are shown a realistic graphic or a plain text description during an international crisis. In our experiments, varying whether respondents are shown realistic graphics or plain text descriptions generally yields little difference in outcomes. Our findings have implications for survey methodology and experiments in political science – researchers may not need to invest the time to develop contextually realistic graphics when designing experiments.

IntroductionAs a research paradigm integrating internal and external validity, the survey experiment method has been widely used in the humanities and social sciences. However, there is currently no research that provides a comprehensive summary of the application of survey experiment in sports psychology.MethodsIn present study, we conducted a literature search of PubMed, Cochrane Library, Web of Science, ELSEVIER and EBSCO from January 1st, 2001 to April 30th, 2025, using the following search terms of titles and abstracts: “Survey Experiment.”ResultsIn total, 59 studies were included in the final review, after thoroughly reviewing all the studies, the current review addressed the following aspects using narrative method: the connotations and characteristics of survey experiment; the design of survey experiment; a case study on the design of survey experiment in sport psychology research; and the advantages of the Internet and big data in advancing survey experiment.

Survey experiments are now quite common in political science. A recent analysis of the number of mentions of this term in political science journal articles demonstrates a dramatic increase from 2000 to 2013. In addition, the term survey experiment has been picked up by many other disciplines, by researchers in a variety of different countries. Given the large number of survey experiments already published, the goal here is not to review the numerous excellent studies using this methodology, because there are far too many, spanning too many different topics. Instead, this juncture—marked by both progress and the proliferation of this method—is used to highlight some of the issues that have arisen as this methodological approach has come of age. How might research using this methodology improve in political science? What are the greatest weaknesses of survey experimental studies in this discipline to date? The explosive growth of survey experiments in political science speaks to their popularity as a means of establishing causal inference. In his reflection on the origins of survey experiments, Paul Sniderman has suggested that their quick rise in popularity was due to two factors: a) their ability to meet expected standards of external validity within the discipline without sacrificing internal validity, and b) the lower marginal cost per study relative to studies that were representative national surveys. Collaborative data collection efforts such as the Multi-Investigator Project and Time-sharing Experiments for the Social Sciences (TESS) made it possible for more scholars to execute population-based survey experiments at a lower cost per study than traditional surveys. Using shared platforms, researchers can execute many experiments for the price of one representative survey. These explanations make perfect sense in the context of a field such as political science, where external validity traditionally has been valued more highly than internal validity. It may be surprising to younger colleagues to learn that, not all that long ago, experiments were deemed completely inappropriate within the discipline of political science, unless they were field experiments executed in the real world. Experiments involving interventions in naturally occurring political environments were deemed tolerable, but only political psychologists were likely to find experimentation more broadly acceptable due to their strong ties to psychology. In political science, survey experiments were a means of promoting experimental methods in an external-validity-oriented discipline. Survey experiments freed political scientists from college sophomores as subjects and promised that external validity need not be sacrificed for strong causal inference. Times have obviously changed, and political scientists now embrace a much broader array of methodologies including both observational and experimental methods. This occasion provides an opportunity to re-evaluate the strengths and weaknesses of this innovative method, in theory and in practice.

Survey experiments on probability samples are a popular method for investigating population-level causal questions due to their strong internal validity. However, lower survey response rates and an increased reliance on online convenience samples raise questions about the generalizability of survey experiments. We examine this concern using data from a collection of 50 survey experiments which represent a wide range of social science studies. Recruitment for these studies employed a unique double sampling strategy that first obtains a sample of “eager” respondents and then employs much more aggressive recruitment methods with the goal of adding “reluctant” respondents to the sample in a second sampling wave. This approach substantially increases the number of reluctant respondents who participate and also allows for straightforward categorization of eager and reluctant survey respondents within each sample. We find no evidence that treatment effects for eager and reluctant respondents differ substantially. Within demographic categories often used for weighting surveys, there is also little evidence of response heterogeneity between eager and reluctant respondents. Our results suggest that social science findings based on survey experiments, even in the modern era of very low response rates, provide reasonable estimates of population average treatment effects among a deeper pool of survey respondents in a wide range of settings.

The article offers an overview of the use of survey experiments in political research by relying on available examples, bibliographic data and a content analysis of experimental manuscripts published in leading academic journals over the last two decades. After a short primer to the experimental approach, we discuss the development, applications and potential problems to internal and external validity in survey experimentation. The article also provides original examples, contrasting a traditional factorial and a more innovative conjoint design, to show how survey experiments can be used to test theory on relevant political topics. The main challenges and possibilities encountered in envisaging, planning and implementing survey experiments are examined. The article outlines the merits, limits and implications of the use of the experimental method in political research.

Deliberative democrats aim to reform real-world politics. This goal, however, poses a dilemma for researchers assessing deliberation. Researchers can study deliberation through controlled but artificial laboratory and survey experiments. They can confidently infer causality (high internal validity) but cannot be sure that deliberation will operate the same way in applied politics (low external validity). Meanwhile, researchers can observe naturally occurring deliberation in realistic settings. In this case, they can be confident that the patterns they find are politically relevant (high external validity) but cannot be sure that they represent causal relationships (low internal validity). Without causality, however, scholars of deliberation cannot know whether reforms—changes to the system—will work as intended. Deliberative field experiments can help resolve this dilemma by conducting real experiments on real politics (achieving high internal and external validity). Like any research method, though, field experiments have their pitfalls, and so they are no panacea. This chapter explains the concept and mechanics of deliberative field experiments, links them to broader theories of deliberative democracy, and characterizes the kinds of questions that they are well suited to answering. The chapter concludes by discussing some of the limitations of deliberative field experiments and how they can be combined with other research methods to assess the promise and perils of deliberative reform more thoroughly.

Weber, who systematized the science of sociology, brought the understanding method that can be used in social action research to sociology. Influenced by German idealism during the period when the methodologies of positive sciences dominated social sciences, Weber made the method of understanding suitable for sociological research based on the ideas of William Dilthey. This research is essential in providing a new perspective to the literature by analyzing the method of understanding, which has many disagreements based on the debates. The main purpose of this study is to try to make the understanding method more comprehensible sociologically by referring to the analyzes in the literature. The research was prepared following the literature review method and expanded by scanning the Weberian literature. Due to the richness of the literature, this study is limited to several works in which Weber uses the method of understanding. The works selected for the research were determined as "Protestant ethic and the spirit of capitalism", "Methodology of social sciences", and "Sociology of religion". The reason for choosing these works is that they are accepted in the sociology literature as the main sources for which Weber's understanding method is used. As a result of the research, it was found that Weber's understanding method was divided into two types such as direct observational and explanatory understanding. In addition, two basic definitions are used for the understanding method. These definitions have been interpreted as an empathic analysis of human behavior and directing the researcher's work by putting himself in someone else's shoes. The main reasons for Weber's choosing to use the understanding method are to criticize the scientific methods dominated by the positive sciences, to create his research path of the social sciences, to bring the interpretive and observational research method to sociology, and to put Dilthey's philosophical approach on a scientific plane. In addition, it is the search for a method that can be used in analyzing historical studies in terms of social action. Weber used the understanding method over certain ideal types and chose the understanding method as the main way to create ideal types. As a result, in addition to the fact that the method of understanding should set a path for sociological studies, it has also been reached that the method of understanding does not only express the understanding of the subject through the mind. While the method of understanding offers a wide range of methods to the researchers in this respect, it has also been effective in breaking the effect of positive sciences on social sciences.

This article presents data from a study in which national bureaucrats working in the fields of taxation and food law in Sweden and Denmark are asked which legal sources and methods of interpretation they use when implementing EU law. The purpose is to contribute to the discussion about European legal method by using social science methodology. National agencies and authorities in the fields of taxation and food law face a ‘multilayered’ or ‘multiprincipal’ reality in which there is room for policy choices. The answers given by the interviewees speak of a plurality of legal sources, a situation where bureaucrats are becoming reluctant lawmakers instructing others on how EU law is to be applied and where bureaucrats find it necessary to found their decisions on what colleagues within the authority or from other Member States have said about how EU law should be applied.

Because health systems are conceptualized as social systems, embedded in social contexts and shaped by human agency, values are a key factor in health system change. As such, health systems software—including values, norms, ideas and relationships—is considered a foundational focus of the field of health policy and systems research (HPSR). A substantive evidence-base exploring the influence of software factors on system functioning has developed but remains fragmented, with a lack of conceptual clarity and theoretical coherence. This is especially true for work on ‘social values’ within health systems—for which there is currently no substantive review available. This study reports on a systematic mixed-methods evidence mapping review on social values within HPSR. The study reaffirms the centrality of social values within HPSR and highlights significant evidence gaps. Research on social values in low- and middle-income country contexts is exceedingly rare (and mostly produced by authors in high-income countries), particularly within the limited body of empirical studies on the subject. In addition, few HPS researchers are drawing on available social science methodologies that would enable more in-depth empirical work on social values. This combination (over-representation of high-income country perspectives and little empirical work) suggests that the field of HPSR is at risk of developing theoretical foundations that are not supported by empirical evidence nor broadly generalizable. Strategies for future work on social values in HPSR are suggested, including: countering pervasive ideas about research hierarchies that prize positivist paradigms and systems hardware-focused studies as more rigorous and relevant to policy-makers; utilizing available social science theories and methodologies; conceptual development to build common framings of key concepts to guide future research, founded on quality empirical research from diverse contexts; and using empirical evidence to inform the development of operationalizable frameworks that will support rigorous future research on social values in health systems.

Weighting techniques are employed to generalize results from survey experiments to populations of theoretical and substantive interest. Although weighting is often viewed as a second-order methodological issue, these adjustment methods invoke untestable assumptions about the nature of sample selection and potential heterogeneity in the treatment effect. Therefore, although weighting is a useful technique in estimating population quantities, it can introduce bias and also be used as a researcher degree of freedom. We review survey experiments published in three major journals from 2000–2015 and find that there are no standard operating procedures for weighting survey experiments. We argue that all survey experiments should report the sample average treatment effect (SATE). Researchers seeking to generalize to a broader population can weight to estimate the population average treatment effect (PATE), but should discuss the construction and application of weights in a detailed and transparent manner given the possibility that weighting can introduce bias.

Researchers use survey experiments to establish causal effects in descriptively representative samples, but concerns remain regarding the strength of the stimuli and the lack of realism in experimental settings. We explore these issues by comparing three national survey experiments on Medicare and immigration with contemporaneous natural experiments on the same topics. The survey experiments reveal that providing information increases political knowledge and alters attitudes. In contrast, two real-world government announcements had no discernable effects, except among people who were exposed to the same facts publicized in the mass media. Even among this exposed subsample, treatment effects were smaller and sometimes pointed in the opposite direction. Methodologically, our results suggest the need for caution when extrapolating from survey experiments. Substantively, we find that many citizens are able to recall factual information appearing in the news but may not adjust their beliefs and opinions in response to this information.

Methodological debates in the comparative studies field have become polarized. On the one hand, philosophers such as Winch, Quine and Louch have argued for different reasons that scientific method is inappropriate for social science research. Winch (1958) rules out the possibility of cross-cultural comparison, claiming that outside observers are unable to understand the meaning of rule-governed behaviour, which is dependent on the motives, reasons and decisions of participants in a particular society or culture. Quine's 'translational indeterminacy' thesis implies that equivalence of meaning across cultures is impossible. Louch (1969) argues that observer values will inevitably preclude objective scientific investigation, since social science concepts are moral, not scientific, and that generalization is precluded in any case because activity occurs in such varied contexts. On the other hand, it has been maintained that scientific method in comparative research is conditionally possible, given improved accuracy of measurement (Deutsch 1966), testing of the equivalence of meaning (Levison 1974), or moving to an appropriate and lower level of generalization than scientific laws (MacIntyre 1971). Macintyre argues that the research questions should ask what functions institutions serve in the culture, rather than, as Winch suggests, what institutions mean for participants. With such a move from a subjective towards an objective frame of reference, theory-building becomes possible. The views which support scientific method in social science research are persuasive on both philosophical and practical grounds. Those philosophers who argue that comparative research is impossible, have not only produced incomplete evidence and flawed argument, they have failed to notice that the gap between natural and social science is in practice far narrower than in theory: natural science is subject to uncertainty, based on the difficulty of disentangling causal chains and on the variability of observer perceptions, which is quantitatively, but not qualitatively, different from the uncertainty of social science research. Polarization seems endemic in social science methodology: fact vs meaning; structuralism vs phenomenology; quantitative vs qualitative method. Such debates are likely to continue. Therefore,

The serious presence of cinematographic productions, particularly documentaries, in academic sociological studies is the reason for this comparative study between a documentary, #Female Pleasure, as a non-academic social study, and academic sociological research to show their methodological differences. Departing from Weber’s teachings, this paper examines the methodological aspects of #Female Pleasure as a quasi-research documentary, in four steps: presenting a brief description of the film, indicating reasons that justify this film as sociological research, outlining the principles of Weber’s methodology for Social Sciences, and then comparing Weber’s proposed criteria with the analytic method of this film.