Question Order Effects Research Articles

Quantum-like models cannot account for the conjunction fallacy

Human agents happen to judge that a conjunction of two terms is more probable than one of the terms, in contradiction with the rules of classical probabilities---this is the conjunction fallacy. One of the most discussed accounts of this fallacy is currently the quantum-like explanation, which relies on models exploiting the mathematics of quantum mechanics. The aim of this paper is to investigate the empirical adequacy of major quantum-like models which represent beliefs with quantum states. We first argue that they can be tested in three different ways, in a question order effect configuration which is different from the traditional conjunction fallacy experiment. We then carry out our proposed experiment, with varied methodologies from experimental economics. The experimental results we get are at odds with the predictions of the quantum-like models. This strongly suggests that this quantum-like account of the conjunction fallacy fails. Future possible research paths are discussed.

Testing quantum-like models of judgment for question order effect

Lately, so-called “quantum” models, based on parts of the mathematics of quantum mechanics, have been developed in decision theory and cognitive sciences to account for seemingly irrational or paradoxical human judgments. We consider here some such quantum-like models that address question order effects, i.e. cases in which given answers depend on the order of presentation of the questions. Models of various dimensionalities could be used; can the simplest ones be empirically adequate? From the quantum law of reciprocity, we derive new empirical predictions that we call the Grand Reciprocity equations, that must be satisfied by several existing quantum-like models, in their non-degenerate versions. Using substantial existing data sets, we show that these non-degenerate versions fail the GR test in most cases, which means that, if quantum-like models of the kind considered here are to work, it can only be in their degenerate versions. However, we suggest that the route of degenerate models is not necessarily an easy one, and we argue for more research on the empirical adequacy of degenerate quantum-like models in general.

Question Order Effects in Sustainable Behavior Norm Measurement: Gender Differences

Sustainable behavior is an important and increasingly popular field of research, but the question order effects that may occur in descriptive and injunctive sustainable behavior norm measurements remain largely unexplored. Even less attention is drawn to gender differences that may be present in question order effects. In the present study, we have investigated the question order effects and their gender differences in measurements of descriptive and injunctive social norms directed at sustainable behavior. For the purposes of the study, three questionnaires were constructed, each questionnaire consisting of 16 items half of which were directed at injunctive and half at descriptive social norms. The questionnaires differed only in the order in which the items were presented. Engineering, art, and social science students from various Lithuanian universities participated in the study (N = 296). The participants were equally distributed by gender, their mean age was 20 years (SD = 1.5). Participants were asked to fill in the questionnaires during lectures, and each participant was randomly assigned to fill in one of three types of questionnaires. The results have shown that the evaluations of injunctive and descriptive social norms were sensitive to the question order. When priming participants with one type of social norm, the evaluation of the other type of social norm was affected. Presenting questionnaire items in random order did not cancel priming; rather, it made both types of social norms salient at the same time, and therefore norm measurements of this group did not differ from those that were obtained under priming. The further analysis indicated that the question order effects were present only in the female sample. We recommend presenting questionnaire items directed at descriptive and injunctive social norms in random order. When measuring just one type of social norm, we would recommend adding items directed at the other type of social norm as well, in order to obtain data that would be more comparable with other studies.

Questionnaire Design Effects in Climate Change Surveys

Despite strong agreement among scientists, public opinion surveys reveal wide partisan disagreement on climate issues in the United States. We suggest that this divide may be exaggerated by questionnaire design variables. Following a brief literature review, we report on a national survey experiment involving U.S. Democrats and Republicans ( n = 2,041) (fielded August 25–September 5, 2012) that examined the effects of question wording and order on the belief that climate change exists, perceptions of scientific consensus, and support for limiting greenhouse gas emissions. Wording a questionnaire in terms of “global warming” (versus “climate change”) reduced Republicans’ (but not Democrats’) existence beliefs and weakened perceptions of the scientific consensus for both groups. Moreover, “global warming” reduced Republicans’ support for limiting greenhouse gases when this question immediately followed personal existence beliefs but not when the scientific consensus question intervened. We highlight the importance of attending to questionnaire design in the analysis of partisan differences.

Order effect in a study on US voters’ preferences: quantum framework representation of the observables

The US political system in the recent years has been mainly formed by a Divided Government, which is regarded as a consequence of ‘non-separability’ of voters’ preferences. The non- separability phenomenon emerges as a result of strong correlations that the voters establish between their preferences for the Congress and the White House contests. We investigate with help of the empirical data from the Smith et al (1999 J. Polit. Sci. 43 737–764) study what implications the upcoming information (encoded in the observables- questions) has on the non- separability emergence. We show that the informational context of the questions alters the preference frequencies that cannot be captured in a classical probabilistic framework. We attribute the changes to the incompatibility of observables C and P, which correspond to questions being asked. We embed our data in a quantum framework and model the voters’ mental state evolution as it is impacted by the operators, to show the non-commutativity of the transition probabilities as a result of the question order effect.

The effects of response option order and question order on self-rated health.

This study aims to assess the impact of response option order and question order on the distribution of responses to the self-rated health (SRH) question and the relationship between SRH and other health-related measures. In an online panel survey, we implement a 2-by-2 between-subjects factorial experiment, manipulating the following levels of each factor: (1) order of response options ("excellent" to "poor" versus "poor" to "excellent") and (2) order of SRH item (either preceding or following the administration of domain-specific health items). We use Chi-square difference tests, polychoric correlations, and differences in means and proportions to evaluate the effect of the experimental treatments on SRH responses and the relationship between SRH and other health measures. Mean SRH is higher (better health) and proportion in "fair" or "poor" health lower when response options are ordered from "excellent" to "poor" and SRH is presented first compared to other experimental treatments. Presenting SRH after domain-specific health items increases its correlation with these items, particularly when response options are ordered "excellent" to "poor." Among participants with the highest level of current health risks, SRH is worse when it is presented last versus first. While more research on the presentation of SRH is needed across a range of surveys, we suggest that ordering response options from "poor" to "excellent" might reduce positive clustering. Given the question order effects found here, we suggest presenting SRH before domain-specific health items in order to increase inter-survey comparability, as domain-specific health items will vary across surveys.

Quantum models for psychological measurements: an unsolved problem.

There has been a strong recent interest in applying quantum theory (QT) outside physics, including in cognitive science. We analyze the applicability of QT to two basic properties in opinion polling. The first property (response replicability) is that, for a large class of questions, a response to a given question is expected to be repeated if the question is posed again, irrespective of whether another question is asked and answered in between. The second property (question order effect) is that the response probabilities frequently depend on the order in which the questions are asked. Whenever these two properties occur together, it poses a problem for QT. The conventional QT with Hermitian operators can handle response replicability, but only in the way incompatible with the question order effect. In the generalization of QT known as theory of positive-operator-valued measures (POVMs), in order to account for response replicability, the POVMs involved must be conventional operators. Although these problems are not unique to QT and also challenge conventional cognitive theories, they stand out as important unresolved problems for the application of QT to cognition. Either some new principles are needed to determine the bounds of applicability of QT to cognition, or quantum formalisms more general than POVMs are needed.

Culture-Sensitive Question Order Effects of Self-Rated Health Between Older Hispanic and Non-Hispanic Adults in the United States.

The aim of this study is to examine context effects created by the question order for self-rated health (SRH) by race/ethnicity and language. Differences in SRH estimates for non-Hispanic Whites and Hispanics were first examined with multiple observational data that asked SRH in different contexts. To examine context effects by socio-demographics and health-related characteristics, we conducted experiments on SRH question order. While Hispanics reported poorer health than non-Hispanic Whites, this difference, in part, depended on question contexts. With SRH asked after rather than before specific health questions, Hispanics, especially Spanish-speaking Hispanics, reported better health, while non-Hispanic Whites' reports remained consistent. Among Spanish-speaking Hispanics, the context effect was larger for unmarried and less educated persons and those with comorbidities. Question contexts influence SRH reports by Spanish-speaking older adults. Cross-cultural inquiries on the meaning of health and its dynamics with question contexts may explain what SRH measures for increasingly diverse populations.

Aircraft dose-response research in National Park backcountry areas

The Federal Aviation Administration and National Park Service conducted joint research to better understand visitor response to noise from commercial air tour operations over units of the National Park System. Dose-response research in National Parks was conducted during the 1990’s at heavily visited, frontcountry sites. To expand this work, aircraft dose-response measurements were recently performed at seven backcountry sites in four National Parks. These sites provided both day and overnight hiking and camping opportunities, with visits ranging from one hour to several days. The sampling approach included side-by-side use of three survey instruments, enabling the evaluation of multiple response variables (annoyance, interference with natural quiet, and acceptability) and comparison of audio clip and in situ dose-response, along with the effects of wording and question order. The evaluation of visits longer than previously studied necessitated methodological enhancements including the use of global positioning system based tracking of visitor locations. In total, over 4600 visitor surveys and 50 days of simultaneous acoustic data were collected. This paper, the first of two on this research, describes the research methods and visitor surveys and presents summary results of collected data.

Complementarity of Mental Observables

AbstractThe aim of this note is to complete the discussion on the possibility of creation of quantum‐like (QL) representation for the question order effect which was presented by Wang and Busemeyer (2013). We analyze the role of a fundamental feature of mental operators (given, e.g., by questions), namely, their complementarity.

Order Effects in Dynamic Semantics

In their target article, Wang and Busemeyer (2013) discuss question order effects in terms of incompatible projectors on a Hilbert space. In a similar vein, Blutner recently presented an orthoalgebraic query language essentially relying on dynamic update semantics. Here, I shall comment on some interesting analogies between the different variants of dynamic semantics and generalized quantum theory to illustrate other kinds of order effects in human cognition, such as belief revision, the resolution of anaphors, and default reasoning that result from the crucial non-commutativity of mental operations upon the belief state of a cognitive agent.

A Quantum Question Order Model Supported by Empirical Tests of anA Prioriand Precise Prediction

Question order effects are commonly observed in self-report measures of judgment and attitude. This article develops a quantum question order model (the QQ model) to account for four types of question order effects observed in literature. First, the postulates of the QQ model are presented. Second, an a priori, parameter-free, and precise prediction, called the QQ equality, is derived from these mathematical principles, and six empirical data sets are used to test the prediction. Third, a new index is derived from the model to measure similarity between questions. Fourth, we show that in contrast to the QQ model, Bayesian and Markov models do not generally satisfy the QQ equality and thus cannot account for the reported empirical data that support this equality. Finally, we describe the conditions under which order effects are predicted to occur, and we review a broader range of findings that are encompassed by these very same quantum principles. We conclude that quantum probability theory, initially invented to explain order effects on measurements in physics, appears to be a powerful natural explanation for order effects of self-report measures in social and behavioral sciences, too.

Measuring Attribute-Specific and Overall Satisfaction with Destination Experience

Researchers have measured tourist satisfaction at both overall (global) and attribute levels, and found that attribute-specific satisfaction significantly influences overall satisfaction. When satisfaction is operationalized at two levels, the measurement structure presents “specific-general” questions, which may yield psychological effects due to the ordering of questions. This study examined question order effects when measuring attribute and overall satisfaction with destination experience and investigated the role of information satisfaction. Results revealed that information satisfaction is a significant predictor of attribute and overall satisfaction, and overall satisfaction was consistently higher than the average of attribute-specific satisfaction which was asked prior to overall satisfaction. Therefore, it can be inferred that overall emotional satisfaction reflects something more than the aggregated satisfaction level with attributes of destinations.

The Effect of Changes to Question Order on the Prevalence of ‘Sufficient’ Physical Activity in an Australian Population Survey

Population surveys are frequently used to assess prevalence, correlates and health benefits of physical activity. However, nonsampling errors, such as question order effects, in surveys may lead to imprecision in self reported physical activity. This study examined the impact of modified question order in a commonly used physical activity questionnaire on the prevalence of sufficient physical activity. Data were obtained from a telephone survey of adults living in Queensland, Australia. A total of 1243 adults participated in the computer-assisted telephone interview (CATI) survey conducted in July 2008 which included the Active Australia Questionnaire (AAQ) presented in traditional or modified order. Binary logistic regression analyses was used to examine relationships between question order and physical activity outcomes. Significant relationships were found between question order and sufficient activity, recreational walking, moderate activity, vigorous activity, and total activity. Respondents who received the AAQ in modified order were more likely to be categorized as sufficiently active (OR = 1.28, 95% CI 1.01-1.60). This study highlights the importance of question order on estimates of self reported physical activity. This study has shown that changes in question order can lead to an increase in the proportion of participants classified as sufficiently active.

The financial crisis and the well-being of Americans.

The Great Recession was associated with large changes in income, wealth, and unemployment, changes that affected many lives. Since January 2008, the Gallup Organization has been collecting daily data on 1,000 Americans each day, with a range of self-reported well-being (SWB) questions. I use these data to examine how the recession affected the emotional and evaluative lives of the population, as well as of subgroups within it. In the fall of 2008, around the time of the collapse of Lehman Brothers, and lasting into the spring of 2009, at the bottom of the stock market, Americans reported sharp declines in their life evaluation, sharp increases in worry and stress, and declines in positive affect. By the end of 2010, in spite of continuing high unemployment, these measures had largely recovered, though worry remained higher and life evaluation lower than in January 2008. The SWB measures do a much better job of monitoring short-run levels of anxiety as the crisis unfolded than they do of reflecting the evolution of the economy over a year or two. Even large macroeconomic shocks to income and unemployment can be expected to produce only small and hard to detect effects on SWB measures. SWB, particularly evaluation of life as a whole, is sensitive to question order effects. Asking political questions before the life evaluation question reduces reported life evaluation by an amount that dwarfs the effects of even the worst of the crisis; these order effects persist deep into the interview, and condition the reporting of hedonic experience and of satisfaction with standard of living. Methods for controlling these effects need to be developed and tested if national measures are to be comparable over space and time.

Bipolarity and/or duality of social survey measurement scales and the question-order effect

When measuring (complex) attitudes within a social survey, researchers often use balanced lists of positive and negative items. The purpose of the present research is to investigate: (a) whether a specific order of measurement scale items can lead to the bipolar (single-dimensional) concept (attitude) being recognised as a dual (bi-dimensional) concept and vice-versa; and (b) whether item order can affect the consistency (metric characteristics) of a measurement scale. An experiment on a group of social science students was conducted: students were randomly split into three subgroups and three different version of a questionnaire (with three differing item orders) were applied. A multi-group confirmatory factor analysis (‘CFA’) and a single group CFA for each item order separately were applied. The final conclusion of the experiment is that there is no general rule about how and when respondents form separate (dual) or unidimensional (continuous) representations of measured concepts. Item-order effects are possible, but they are not as important as one would expect. The results of the experiment also suggest that other factors should be taken into account: the content of the measured concept and the cognitive sophistication of the respondents.

THE ORDER OF QUESTIONS IN A SURVEY ON CITIZEN SATISFACTION WITH PUBLIC SERVICES: LESSONS FROM A SPLIT‐BALLOT EXPERIMENT

Surveys of citizen satisfaction with local public services have become widespread, with the results increasingly used to reorganize services, to allocate budgets, and to hold managers accountable. But evidence from a split‐ballot experiment that we conducted suggests that the order of questions in a citizen survey has important effects on reported satisfaction with specific public services as well as overall citizen satisfaction. Moreover, the correlations of specific service ratings with overall satisfaction, and thus the identification of key drivers of overall satisfaction, also turn out to be highly sensitive to question order. These findings are in line with research on priming and question order effects in the survey methodology literature, but these effects have not been carefully examined before in the context of citizen surveys and public administration research. Policy and management implications of these finding are discussed.

Methodologic Artifacts in Adult Sun-Protection Trends

Skin cancer is the most common of all cancers and a public health concern of increasing magnitude in the U.S. A mid-year review of Healthy People 2010 found that the percentage of adults engaging in sun-safe practices increased from 59% in 2000 to 71% in 2005. This paper examines whether this increase in sun-safe practices in adults is an artifact caused by the change from 2000 to 2005 in the operational definition of "sun-safe practices" in the National Health Interview Survey (NHIS). This study analyzed the 2000 and 2005 NHIS data sets in 2009 to examine changes in sun-safe practice prevalence estimates and to estimate the relative odds of practicing sun-safe behaviors associated with gender, age, race/ethnicity, education, and geographic region. Contrary to the Healthy People 2010 mid-year review, it was found that engaging in sun-safe practices by adults decreased from 59% (2000) to 55% (2005). A question order effect and use of images for use of sun-safe hats likely caused an artificial change in "cover-up" behavior estimates from 31% (2000) to 18% (2005) and wide-brim hat usage from 27% (2000) to 12% (2005). When eliminating data from these two questions added in 2005, the relative odds of practicing sun-safe behaviors was significantly lower for men in both 2000 and 2005 but were not different for other demographic characteristics. The current analyses suggest that the increase in sun-safe behaviors in adults and a notable decrease in "hat use" may be due to methodologic artifacts. When operational definitions change, caution should be used in interpreting estimates over time.

Perceptions of safety and victimization: does survey construction affect perceptions?

Surveys are commonly used in the field of criminology and criminal justice; however, the effects of survey design on study findings are rarely examined. Using a randomized experiment, this study examined the effects of question order in a victimization survey on respondents’ perceptions of safety. The interactions between respondents’ characteristics and question-order effects were also explored. The simple question-order effect was not found for the overall sample, but further examination revealed that respondent characteristics such as sex and victim status interact with the arrangement of questions. The implications of the findings for victimization survey research, and more generally for studies using surveys as a primary tool, are discussed.

Question Order and Interviewer Effects in CATI Scale-up Surveys

The scale-up estimator is a network-based estimator for the size of hidden or hard to count subpopulations. Several issues arise in the public health context when the aim is the estimation of injuries occurring in a certain population, where two common problems are present: (a) Small injuries are usually difficult to observe and rarely reported in the official data and (b) people are not always compliant in giving information about some specific injuries, in particular when children are involved. This study checked the methodological issues arising from using a computer-assisted telephone interview (CATI) survey using the scale-up methodology for detecting the number of injuries due to choking in children ages 0 to 14 in Italy. For this purpose, 1,000 CATI interviews were conducted during a week using a questionnaire based on 33 questions about populations of known size according to census data. Then, each respondent was asked about other questions related to the main target population (e.g., number of children known to suffer from a choking accident). A sensitivity analysis was conducted for estimating the effect of varying subpopulations, order of the questions, and interviewer effects on the resulting estimates. For the interviewer effect, no particular differences were observed in the overall estimates of injuries. The conclusion is the scale-up estimator in association with CATI methodology shows a high potential in the field of injury prevention, being accurate and robust, but particular attention should be given to the training of the interviewers to improve stability of the estimates.