Does Truth Pay? Investigating the Effectiveness of the Bayesian Truth Serum With an Interim Payment: A Registered Report

Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

An experiment of predicting the result of the National Basketball Association (NBA) is designed to compare the performance of Bayesian truth serum (BTS) and robust Bayesian truth serum (RBTS) in expert truth telling incentive. It shows that 1) the fundamental hypothesis of BTS, i.e. individuals treating personal opinions as an “impersonally informative” signal about the population distribution, is valid; 2) the relative position of the individual in a group significantly affects the individual's score of RBTS due to RBTS algorithm, which leads to a poor incentive to tell truth, while BTS does not have this problem; 3) BTS does not encourage extreme opinions, while RBTS cannot effectively reward the extreme opinions due to significant deviation of their RBTS scores. The analytic results prefer to BTS, but when the number of individuals is small, RBTS should be employed.

Over the past few decades, psychology and its cognate disciplines have undergone substantial scientific reform, ranging from advances in statistical methodology to significant changes in academic norms. One aspect of experimental design that has received comparatively little attention is incentivization, i.e., the way that participants are rewarded and incentivized monetarily for their participation in experiments and surveys. While incentive-compatible designs are the norm in disciplines like economics, the majority of studies in psychology and experimental philosophy are constructed such that individuals' incentives to maximize their payoffs in many cases stand opposed to their incentives to state their true preferences honestly. This is in part because the subject matter is often self-report data about subjective topics, and the sample is drawn from online platforms like Prolific or MTurk where many participants are out to make a quick buck. One mechanism that allows for the introduction of an incentive-compatible design in such circumstances is the Bayesian Truth Serum (BTS; Prelec, 2004), which rewards participants based on how surprisingly common their answers are. Recently, Schoenegger (2021) applied this mechanism in the context of Likert-scale self-reports, finding that the introduction of this mechanism significantly altered response behavior. In this registered report, we further investigate this mechanism by (1) attempting to directly replicate the previous result and (2) analyzing if the Bayesian Truth Serum's effect is distinct from the effects of its constituent parts (increase in expected earnings and addition of prediction tasks). We fail to find significant differences in response behavior between participants who were simply paid for completing the study and participants who were incentivized with the BTS. Per our pre-registration, we regard this as evidence in favor of a null effect of up to V = .1 and a failure to replicate but reserve judgment as to whether the BTS mechanism should be adopted in social science fields that rely heavily on Likert-scale items reporting subjective data, seeing that smaller effect sizes might still be of practical interest and results may differ for items different from the ones we studied. Further, we provide weak evidence that the prediction task itself influences response distributions and that this task's effect is distinct from an increase in expected earnings, suggesting a complex interaction between the BTS' constituent parts and its truth-telling instructions.

The Bayesian truth serum (BTS) is a survey scoring method that creates truth-telling incentives for respondents answering multiple-choice questions about intrinsically private matters, such as opinions, tastes, and behavior. The authors test BTS in several studies, primarily using recognition questionnaires that present items such as brand names and scientific terms. One-third of the items were nonexistent foils. The BTS mechanism, which mathematically rewards “surprisingly common” answers, both rewarded truth telling, by heavily penalizing foil recognition, and induced truth telling, in that participants who were paid according to their BTS scores claimed to recognize fewer foils than control groups, even when given competing incentives to exaggerate. Survey takers who received BTS-based payments without explanation became less likely to recognize foils as they progressed through the survey, suggesting that they learned to respond to BTS incentives despite the absence of guidance. The mechanism also outperformed the solemn oath, a competing truth-inducement mechanism. Finally, when applied to judgments about contributing to a public good, BTS eliminated the bias common in contingent valuation elicitations.

Bayesian truth serum (BTS) is an exciting new method for improving honesty and information quality in multiple-choice survey, but, despite the method’s mathematical reliance on large sample sizes, existing literature about BTS only focuses on small experiments. Combined with the prevalence of online survey platforms, such as Amazon’s Mechanical Turk, which facilitate surveys with hundreds or thousands of participants, BTS must be effective in large-scale experiments for BTS to become a readily accepted tool in real-world applications. We demonstrate that BTS quantifiably improves honesty in large-scale online surveys where the “honest” distribution of answers is known in expectation on aggregate. Furthermore, we explore a marketing application where “honest” answers cannot be known, but find that BTS treatment impacts the resulting distributions of answers.

Peer prediction mechanisms allow the truthful elicitation of private signals (e.g., experiences, or opinions) in regard to a true world state when this ground truth is unobservable. The original peer prediction method is incentive compatible for any number of agents n >= 2, but relies on a common prior, shared by all agents and the mechanism. The Bayesian Truth Serum (BTS) relaxes this assumption. While BTS still assumes that agents share a common prior, this prior need not be known to the mechanism. However, BTS is only incentive compatible for a large enough number of agents, and the particular number of agents required is uncertain because it depends on this private prior. In this paper, we present a robust BTS for the elicitation of binary information which is incentive compatible for every n >= 3, taking advantage of a particularity of the quadratic scoring rule. The robust BTS is the first peer prediction mechanism to provide strict incentive compatibility for every n >= 3 without relying on knowledge of the common prior. Moreover, and in contrast to the original BTS, our mechanism is numerically robust and ex post individually rational.

Objective Criminological researchers want people to reveal considerable private information when utilizing self-report surveys, such as involvement in crime, subjective attitudes and expectations, and probability judgments. Some of this private information is easily accessible for subjects and all that is required is for individuals to be honest, while other information requires mental effort and cognitive reflection. Though researchers generally provide little or no incentive to be honest and thoughtful, it is generally assumed that subjects do provide honest and accurate information. We assess the accuracy of deterrence measures by employing a scoring rule known as the Bayesian truth serum (BTS)—that incentivizes honesty and thoughtfulness among respondents.

Unraveling hypothetical bias in discrete choice experiments

This chapter discusses the use of archival data in psychological research. Archival data are existing data that were collected for a purpose other than its current use. First, we review several sources and types of archival data that may be of particular interest to applied psychology researchers, including social science data archives, public documents, datasets, official records, private documents or records, and mass media. Next we discuss advantages and disadvantages that are unique to archival data use in psychological research, such as the opportunity to access datasets with a specific methodology, design or specific population, along with the inherent challenges associated with navigating large, complex datasets with limited control over whether and how variables of interest are measured. Finally, we conclude with a series of recommendations to anyone who may be considering the use of archival data for a research project. We hope this chapter serves as a useful introduction to archival data and helpful resource to those seeking to use archival data in psychological research.

This paper aims to develop the insights into Bayesian truth serum (BTS) mechanism by postulating a sequence of seven natural conditions reminiscent of axioms in information theory. The condition that reduces a larger family of mechanisms to BTS is additivity, akin to the axiomatic development of entropy. The seven conditions identify BTS as the unique scoring rule for ranking respondents in situations in which respondents are asked to choose an alternative from a finite set and provide predictions of their peers’ propensities to choose, for finite or infinite sets of respondents.

The EU Broiler Directive came into force in the UK in June 2010 with the aim of setting new minimum standards, monitoring broiler welfare and addressing any welfare problems. A survey questionnaire was used to elicit information from a stratified sample of citizens in England and Wales regarding their willingness to pay for the provisions of the Directive, as an estimate of the consumer surplus associated with the legislation. We also explore the usefulness of Prelec's () Bayesian Truth Serum (BTS) in promoting respondents’ truthful reporting. A median willingness to pay of £21.50 per household per year (corrected for sample bias and possible ‘yea saying’) was estimated from 665 responses. This provides an estimated benefit of the legislation to citizens of over £503 million per year, equivalent to 5.3% of current consumer expenditure on chicken. This compares to an estimated £22 million per year cost of producers’ compliance and government enforcement associated with the legislation. No statistically significant differences in responses between respondents that did and did not have a BTS incentive to answer questions truthfully were found, which might reflect apparently truthful answers in this case, an insufficiently strong financial incentive or a weakened effect due to an element of disbelief in the BTS amongst the sample. The analysis suggests that the benefits of the Broiler Directive to citizens greatly outweigh the additional costs to producers, making a case for the legislation to be retained.

Intelligent systems often depend on data provided by information agents, for example, sensor data or crowdsourced human computation. Providing accurate and relevant data requires costly effort that agents may not always be willing to provide. Thus, it becomes important not only to verify the correctness of data, but also to provide incentives so that agents that provide high-quality data are rewarded while those that do not are discouraged by low rewards. We cover different settings and the assumptions they admit, including sensing, human computation, peer grading, reviews, and predictions. We survey different incentive mechanisms, including proper scoring rules, prediction markets and peer prediction, Bayesian Truth Serum, Peer Truth Serum, Correlated Agreement, and the settings where each of them would be suitable. As an alternative, we also consider reputation mechanisms. We complement the game-theoretic analysis with practical examples of applications in prediction platforms, community sensing, and peer grading.

The popularity and use of Bayesian methods have increased across many research domains. The current article demonstrates how some less familiar Bayesian methods can be used. Specifically, we applied expert elicitation, testing for prior-data conflicts, the Bayesian Truth Serum, and testing for replication effects via Bayes Factors in a series of four studies investigating the use of questionable research practices (QRPs). Scientifically fraudulent or unethical research practices have caused quite a stir in academia and beyond. Improving science starts with educating Ph.D. candidates: the scholars of tomorrow. In four studies concerning 765 Ph.D. candidates, we investigate whether Ph.D. candidates can differentiate between ethical and unethical or even fraudulent research practices. We probed the Ph.D.s’ willingness to publish research from such practices and tested whether this is influenced by (un)ethical behavior pressure from supervisors or peers. Furthermore, 36 academic leaders (deans, vice-deans, and heads of research) were interviewed and asked to predict what Ph.D.s would answer for different vignettes. Our study shows, and replicates, that some Ph.D. candidates are willing to publish results deriving from even blatant fraudulent behavior–data fabrication. Additionally, some academic leaders underestimated this behavior, which is alarming. Academic leaders have to keep in mind that Ph.D. candidates can be under more pressure than they realize and might be susceptible to using QRPs. As an inspiring example and to encourage others to make their Bayesian work reproducible, we published data, annotated scripts, and detailed output on the Open Science Framework (OSF).

Traditional ensemble methods rely on majority voting, which may fail to recognize correct answers held by a minority in scenarios requiring specialized knowledge. Therefore, this paper proposes two novel ensemble methods for supervised classification, named Confidence Truth Serum (CTS) and Confidence Truth Serum with Single Regression (CTS-SR). The former is based on the principles of Bayesian Truth Serum (BTS) and introduces classification confidence to calculate the prior and posterior probabilities of events, enabling the recovery of correct judgments provided by a confident minority beyond majority voting. CTS-SR further simplifies the algorithm by constructing a single regression model to reduce computational overhead, making it suitable for large-scale applications. Experiments are conducted on multiple binary classification datasets to evaluate CTS and CTS-SR. Experimental results demonstrate that, compared with existing ensemble methods, both of the proposed methods significantly outperform baseline algorithms in terms of accuracy and F1 scores. Specifically, there is an average improvement of 2–6% in accuracy and an average increase of 2–4% in F1 score. Notably, on the Musk and Hilly datasets, our method achieves a 5% improvement compared to the traditional majority voting approach. Particularly on the Hilly dataset, which generally exhibits the poorest classification performance and poses the greatest prediction challenges, our method demonstrates the best discriminative performance. validating the importance of confidence as a feature in ensemble learning.

PurposePrevious research has little specific guidance on how to improve large infrastructures’ risk analysis. This paper aims to propose a practical risk analysis framework across the project lifecycle with Bayesian Networks (BNs).Design/methodology/approachThe framework includes three phases. In the qualitative phase, primary risks were identified by literature reviews and interviews; questionnaires were used to determine key risks at each project stage and causal relationships between stage-related risks. In the quantitation, brainstorming and questionnaires, and techniques of ranked nodes/paths, risk map and Bayesian truth serum were adopted. Then, a BN-based risk assessment model was developed, and risk analysis was conducted with AgenaRisk software.FindingsTwenty key risks across the lifecycle were determined: some risks were recurring and different risks emerged at various stages with the construction and feasibility most risky. Results showed that previous stages’ risks significantly amplified subsequent stages’ risks. Based on the causality of stage-related risks, a qualitative model was easily constructed. Ranked nodes/paths facilitated the quantification by requiring less statistical knowledge and fewer parameters than traditional BNs. As articulated by a case, this model yielded very simple and easy-to-understand representations of risks and risk propagation pathways.Originality/valueRare research has developed a BN risk assessment model from the perspective of project stages. A structured model, a propagation network among individual risks, stage-related risks, and the final adverse consequence, has been designed. This research provides practitioners with a realistic risk assessment approach and further understanding of dynamic and stage-related risks throughout large infrastructures’ lifecycle. The framework can be modified and used in other real-world risk analysis where risks are complex and develop in stages.