Question Order Effects Research Articles

Application of Theory of Quantum Instruments to Psychology: Combination of Question Order Effect with Response Replicability Effect.

Recently, quantum formalism started to be actively used outside of quantum physics: in psychology, decision-making, economics, finances, and social science. Human psychological behavior is characterized by a few basic effects; one of them is the question order effect (QOE). This effect was successfully modeled (Busemeyer–Wang) by representing questions A and B by Hermitian observables and mental-state transformations (back action of answering) by orthogonal projectors. However, then it was demonstrated that such representation cannot be combined with another psychological effect, known as the response replicability effect (RRE). Later, this no-go result was generalized to representation of questions and state transformations by quantum instruments of the atomic type. In light of these results, the possibility of using quantum formalism in psychology was questioned. In this paper, we show that, nevertheless, the combination of the QOE and RRE can be modeled within quantum formalism, in the framework of theory of non-atomic quantum instruments.

An Integrative Approach to Assess Subjective Well-Being. A Case Study on Romanian University Students

Subjective well-being (SWB) has presented long-lasting interest for researchers and the recent focus on the economic approach to SWB led to increased awareness of the topic. Despite the significant number of studies, conceptualizing and assessing SWB, along with finding predictors of SWB, need further empirical exploration. Following this rationale, using statistical and econometric methods (correlation analysis, Principal Component Analysis (PCA), Multinomial Logistic Regression (MLR)) applied on data collected via a survey on students from Bucharest University of Economic Studies (363 respondents), this study explores and provides insights that support a better understanding of defining and measuring SWB. Additionally, the study offers valuable information on the main determinants of SWB for a particular group, in this case, Romanian business students. According to findings, we argue that: (1) when assessing perception of life satisfaction and happiness, Romanian students tend to make slight distinctions between these two concepts; (2) question order effect is not significant, whereas negative sentiments (such as pessimism) impact self-assessment of happiness, but not of life satisfaction; (3) the main predictors for SWB are satisfaction with current activities, level of optimism/pessimism, health, and safety of the neighborhood. This paper proposes a new approach to modeling SWB by MLR, which features expressing the dependent variable with respect to the principal factors obtained by PCA.

The impacts of varying survey design on reported trip satisfaction

Research has shown that the way in which a question is asked or the order in which it is received can influence the answer of respondents. Surveys are frequently used by transport professionals to better understand the psychology behind travelers’ level of satisfaction and derive policies. An awareness of how survey design impacts individuals’ responses to satisfaction-related questions is vital for any future policy analysis and development. This study uses travel data collected through a university travel survey, which was carried out during the fall and winter of 2017/2018, to examine the effects of seasonality, question wording, and question order on travel satisfaction. To test for question wording and seasonality, respondents were asked to rate their satisfaction with their last trip and satisfaction with their typical commute to work or school under either “warm and sunny” or “cold and snowy” conditions, depending on season of survey completion. To test the effect of question order, respondents were randomly assigned one of two survey versions in which the order of the two aforementioned trip satisfaction questions were switched. Our results confirm the presence of question order and wording effects on reported satisfaction levels, and to a lesser degree, differences in trip satisfaction due to seasonality. Results of this study highlight the need to practice caution when designing travel surveys, particularly with respect to how transport survey respondents are impacted by seasonality, question wording and order.

A Closer Look at Attitude Scales with Positive and Negative Items. Response Latency Perspectives on Measurement Quality

Measurement quality in standardized surveys has been a core issue for decades in survey research. For questionnaire designers, it is common to use a mix of positive and negative worded items to measure multi-item-constructs in order to control for response effects like acquiescence bias. The paper shows that paradata such as response latency measurement can be used to identify specific subgroups of respondents with specific types of cognitive response modes. These response modes moderate the occurrence of response effects, systematic and random measurement errors, and thus the reliability and validity of attitudinal measurement models. Therefore, adapting paradata to detect low measurement quality can be used as a tool leading to a better understanding of respondents’ cognitive processes. Data of a German CATI-survey with experimental design and measurement of response latencies are used to analyze data quality of a measurement model of attitudes towards health nutrition with mixed items. Response effects are analyzed through the experimental variation of question order of negative and positive worded items. Structural equation models are estimated in a multiple-group moderator design to test validity and reliability of the latent attitude construct. As a result, the attitude scale shows acceptable values of validity and reliability only under the condition of spontaneous answers where question order effects appear.

On universality of classical probability with contextually labeled random variables

One can often encounter claims that classical (Kolmogorovian) probability theory cannot handle, or even is contradicted by, certain empirical findings or substantive theories. This note joins several previous attempts to explain that these claims are unjustified, illustrating this on the issues of (non)existence of joint distributions, probabilities of ordered events, and additivity of probabilities. The specific focus of this note is on showing that the mistakes underlying these claims can be precluded by labeling all random variables involved contextually. Moreover, contextual labeling also enables a valuable additional way of analyzing probabilistic aspects of empirical situations: determining whether the random variables involved form a contextual system, in the sense generalized from quantum mechanics. Thus, to the extent the Wang–Busemeyer QQ equality for the question order effect holds, the system describing them is noncontextual. The double-slit experiment and its behavioral analogues also turn out to form a noncontextual system, having the same probabilistic format (cyclic system of rank 4) as the one describing spins of two entangled electrons.

Question-Order Effects in the Evaluation of Political Institutions in Decentralized Polities

Abstract This research studies question-order effects in the evaluation of political institutions in socially heterogeneous regions within decentralized countries. Split-ballot experiments were embedded in 3 representative surveys fielded in the Spanish region of Catalonia. Significant consistency effects were spotted in all samples. Respondents who first evaluated the relatively less valued institutions of one governmental level assessed less favorably the relatively better rated institutions of the other level. Clear evidence of the reverse effect was present only in 1 of the experiments. In addition, heterogeneous question-order effects emerged among the 2 distinct national communities coexisting in Catalonia. The article suggests a mechanism beneath this sort of question-order effects and further proposes and empirically tests a remedial solution.

Impact of Question Type and Question Order on Tobacco Prevalence Estimates in US Young Adults: A Randomized Experiment.

This study identifies the effects of question type, order, and inclusion of product images on tobacco use estimates in a national sample of young adults. Ever use and past 30-day use prevalence estimates of specific products were affected by respondents answering breakout items or breakout items with images compared with respondents answering a checklist of items in the survey. Current surveys that include a "select all that apply" list format may be underestimating ever and past 30-day prevalence estimates of tobacco products.

Generalization of Classic Question Order Effects Across Cultures

Questionnaire design is routinely guided by classic experiments on question form, wording, and context conducted decades ago. This article explores whether two question order effects (one due to the norm of evenhandedness and the other due to subtraction or perceptual contrast) appear in surveys of probability samples in the United States and 11 other countries (Canada, Denmark, Germany, Iceland, Japan, the Netherlands, Norway, Portugal, Sweden, Taiwan, and the United Kingdom; N = 25,640). Advancing theory of question order effects, we propose necessary conditions for each effect to occur, and found that the effects occurred in the nations where these necessary conditions were met. Surprisingly, the abortion question order effect even appeared in some countries in which the necessary condition was not met, suggesting that the question order effect there (and perhaps elsewhere) was not due to subtraction or perceptual contrast. The question order effects were not moderated by education. The strength of the effect due to the norm of evenhandedness was correlated with various cultural characteristics of the nations. Strong support was observed for the form-resistant correlation hypothesis.

Testing Question Order Effects of Self-perception of Risk Propensity on Simple Lottery Choices as Measures of the Actual Risk Propensity

Testing Question Order Effects of Self-perception of Risk Propensity on Simple Lottery Choices as Measures of the Actual Risk Propensity

Question order effects in partial least squares path modelling: an empirical investigation

Effects of question order in surveys have been extensively researched. In the light of partial least squares path modeling, only few studies exist. In this study we analyze effects of question order with data from two waves of an annual satisfaction study. The positioning of area satisfaction questions and randomized attribute satisfaction questions was changed between the two waves. Results confirm previously described positive effects on explained variance if area satisfaction is asked after attribute satisfaction. This increase in explained variance in area performance does not lead to an improved model performance in explaining overall satisfaction.

Unintended Causal Pathways: Probing Experimental Mechanisms Through Mediation Analysis

We present the results of several experimental studies relevant to the general study of causal mechanisms, the specific study of audience costs, and the larger body of survey experiments. First, we show that in the context of audience cost experiments, a portion of the estimated treatment effect is related to the intended causal mechanism, national reputation, but this mechanism underlies a relatively small portion of the total causal effect, leaving much of the causal effect unexplained. Second, we show that the experimental treatment manipulates an unintended mechanism that also underlies part of the treatment effect. We conduct two tests to establish the relative independence of each mediator’s effect. Finally, we investigate the possibility that our findings represent question order effects rather than mediation effects, finding that priming does not substantially threaten our results. Our results and discussion advance our understanding of the challenge of studying causal mediation in survey experiments. We provide some recommendations for survey experimentation moving forward.

Beyond-quantum modeling of question order effects and response replicability in psychological measurements

A general tension-reduction (GTR) model was recently considered (Aerts and Sassoli de Bianchi 2015a, 2015b) to derive quantum probabilities as (universal) averages over all possible forms of non-uniform fluctuations and explain their considerable success in describing experimental situations also outside of the domain of physics, for instance in quantum models of cognition and decision. This result also highlighted the possibility of observing violations of the predictions of the Born rule in those situations where the averaging would not be large enough, or would be altered because of the combination of multiple measurements. We show that this is indeed the case in typical psychological measurements exhibiting question order effects, as the data require the larger (non-Hilbertian) framework of the GTR-model to receive an exact mathematical description. We also consider an unsolved problem of quantum cognition: response replicability. It has been noted that if question order effects and response replicability would occur together, the situation could not be handled anymore by quantum theory. Yet, we show that it can be easily and naturally described in the GTR-model. Based on these findings, we motivate the adoption in cognitive science of a hidden-measurements interpretation of the quantum formalism, and of its GTR-model generalization, as a natural interpretational framework explaining the data of psychological measurements on conceptual entities.

Testing Quantum Models of Conjunction Fallacy on the World Wide Web

The 'conjunction fallacy' has been extensively debated by scholars in cognitive science and, in recent times, the discussion has been enriched by the proposal of modeling the fallacy using the quantum formalism. Two major quantum approaches have been put forward: the first assumes that respondents use a two-step sequential reasoning and that the fallacy results from the presence of 'question order effects'; the second assumes that respondents evaluate the cognitive situation as a whole and that the fallacy results from the 'emergence of new meanings', as an 'effect of overextension' in the conceptual conjunction. Thus, the question arises as to determine whether and to what extent conjunction fallacies would result from 'order effects' or, instead, from 'emergence effects'. To help clarify this situation, we propose to use the World Wide Web as an 'information space' that can be interrogated both in a sequential and non-sequential way, to test these two quantum approaches. We find that 'emergence effects', and not 'order effects', should be considered the main cognitive mechanism producing the observed conjunction fallacies.

We Are Happier than We Realize: Underestimation and Conflation in Measuring Happiness

The study evaluates a very common question designed to measure happiness: “Taken all together, how would you say things are these days–would you say that you are very happy, pretty happy, or not too happy?” Through five representative survey experiments, we show that (1) this survey item underestimates the level of happiness with one’s life; (2) this is because the measure is more likely to reflect satisfaction with the state of the world rather than personal life; (3) this measures is more susceptible to priming; (4) the addition of three words “in your life” to the item greatly reduces priming and question order effects; and (5) the addition of these three words produces results that are very similar to life satisfaction measures that include “in your life” and are more positively associated with income. These results provide evidence that a simple correction better measures personal happiness. Furthermore, our findings reassess the foundation of a considerable volume of scholarship about how politics and income is associated happiness.

Probing the question order effect while developing a chemistry concept inventory

The design of assessment tools is critical to accurately evaluate students' understanding of chemistry. Although extensive research has been conducted on various aspects of assessment tool design, few studies in chemistry have focused on the impact of the order in which questions are presented to students on the measurement of students' understanding and students' performance. This potential impact has been labeled the question order effect in other literature and may be considered as a threat to the construct validity of the assessment tool. The set of studies described in this article tested whether question order effects were present within a concept inventory on acid-based chemistry. In particular, we tested whether the order of two conceptually isomorphic questions, one pictorial and one verbal, affected students' performance on the concept inventory. Two different versions of the inventory were developed and collected from students enrolled in the second semester of first-year university chemistry courses (general chemistry;N= 774) at two different institutions and to students enrolled in the first semester of organic chemistry (N= 163) at one of the two institutions. Students were further divided in two groups based on their self-reported level of effort in answering the concept inventory. Interviews were also conducted with a total of 19 students at various stages of the studies. Analyses of differences in students' responses to the two versions of the inventory revealed no question order effect in all settings. Implications for instructors and researchers are provided.

Question-order Effects in the Domains of Language and Marriage

The effect of question-order on respondents’ choices is well-studied and well established, but these effects are not always present and are sometimes contradictory. The question of whether prior involvement in an issue alters the question-order effect also exists. We examined the effect of question-order, or context, on the stated values of participants by exploiting the saliency effect through the use of framing questions. Although our results showed some influence of question order, we encountered a ceiling effect. Our ceiling may have been caused by the same factor observed by Rimal and Real (2005): high-involvement mitigates question-order effect. Nevertheless, for those interested in educational campaigns, studying how context influences an individual’s values and preferences is a subject worthy of further pursuit.

직무만족 조사에서의 질문 순서 및 질문 의도 효과에 관한 연구

직무만족 조사에서의 질문 순서 및 질문 의도 효과에 관한 연구

Question Order Affects the Measurement of Bullying Victimization Among Middle School Students.

Bullying among youth is recognized as a serious student problem, especially in middle school. The most common approach to measuring bullying is through student self-report surveys that ask questions about different types of bullying victimization. Although prior studies have shown that question-order effects may influence participant responses, no study has examined these effects with middle school students. A randomized experiment (n = 5,951 middle school students) testing the question-order effect found that changing the sequence of questions can result in 45% higher prevalence rates. These findings raise questions about the accuracy of several widely used bullying surveys.

On the Foundations of the Brussels Operational-Realistic Approach to Cognition

The scientific community is becoming more and more interested in the research that applies the mathematical formalism of quantum theory to model human decision-making. In this paper, we provide the theoretical foundations of the quantum approach to cognition that we developed in Brussels. These foundations rest on the results of two decade studies on the axiomatic and operational-realistic approaches to the foundations of quantum physics. The deep analogies between the foundations of physics and cognition lead us to investigate the validity of quantum theory as a general and unitary framework for cognitive processes, and the empirical success of the Hilbert space models derived by such investigation provides a strong theoretical confirmation of this validity. However, two situations in the cognitive realm, 'question order effects' and 'response replicability', indicate that even the Hilbert space framework could be insufficient to reproduce the collected data. This does not mean that the mentioned operational-realistic approach would be incorrect, but simply that a larger class of measurements would be in force in human cognition, so that an extended quantum formalism may be needed to deal with all of them. As we will explain, the recently derived 'extended Bloch representation' of quantum theory (and the associated 'general tension-reduction' model) precisely provides such extended formalism, while remaining within the same unitary interpretative framework.

Attitudes towards assisted dying are influenced by question wording and order: a survey experiment

BackgroundSurveys on attitudes towards assisted dying play an important role in informing public debate, policy and legislation. Unfortunately, surveys are often designed with insufficient attention to framing effects; that is, effects on the respondents’ stated attitudes caused by question wording and context. The purpose of this study was to demonstrate and measure such framing effects.MethodsSurvey experiment in which an eight-question survey on attitudes towards assisted dying was distributed to Norwegian citizens through a web-based panel. Two variations of question wording as well as two variations of question order were employed. Respondents were randomized to receive one of four questionnaire versions.ResultsThree thousand and fifty responses were received. There were moderate to large question wording and question order effects. A majority of Norwegian citizens favour the legalization of assisted dying for patients with terminal or chronic disease.ConclusionsStakeholders in the assisted dying debate need to acknowledge potential framing effects, and accordingly should interpret survey results with caution. The same holds for researchers who conduct attitude surveys in the field of bioethics.Electronic supplementary materialThe online version of this article (doi:10.1186/s12910-016-0107-3) contains supplementary material, which is available to authorized users.