Judge Advisor System Research Articles

In human–machine trust, humans rely on a simple averaging strategy

With the growing role of artificial intelligence (AI) in our lives, attention is increasingly turning to the way that humans and AI work together. A key aspect of human–AI collaboration is how people integrate judgements or recommendations from machine agents, when they differ from their own judgements. We investigated trust in human–machine teaming using a perceptual judgement task based on the judge–advisor system. Participants (n=89\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$n=89$$\\end{document}) estimated a perceptual quantity, then received a recommendation from a machine agent. The participants then made a second response which combined their first estimate and the machine’s recommendation. The degree to which participants shifted their second response in the direction of the recommendations provided a measure of their trust in the machine agent. We analysed the role of advice distance in people’s willingness to change their judgements. When a recommendation falls a long way from their initial judgement, do people come to doubt their own judgement, trusting the recommendation more, or do they doubt the machine agent, trusting the recommendation less? We found that although some participants exhibited these behaviours, the most common response was neither of these tendencies, and a simple model based on averaging accounted best for participants’ trust behaviour. We discuss implications for theories of trust, and human–machine teaming.

How transparency affects algorithmic advice utilization: The mediating roles of trusting beliefs

Although algorithms are increasingly used to support professional tasks and routine decision-making, their opaque nature invites resistance and results in suboptimal use of their advice. Scholars argue for transparency to enhance the acceptability of algorithmic advice. However, current research is limited in understanding how improved transparency enhances the use of algorithmic advice, such as the differences among various aspects of transparency and the underlying mechanism. In this paper, we investigate whether and how different aspects of algorithmic transparency (performance, process, and purpose) enhance the use of algorithmic advice. Drawing on the knowledge-based trust perspective, we examine the mediating roles of trusting beliefs in the relationships between transparency and the use of algorithmic advice. Using the “judge-advisor system” paradigm, we conduct a 2 × 2 × 2 experiment to manipulate the three aspects of transparency and examine their effects on the use of algorithmic advice. We find that performance and process transparency promote the use of algorithmic advice. However, the effect of process transparency gets attenuated when purpose transparency is high. Purpose transparency is only useful when process transparency is low. We also find that while all three aspects of transparency facilitate different trusting beliefs, only competence belief significantly promotes the use of algorithmic advice. It also fully mediates the facilitating effects of performance and process transparency. This study contributes to the emerging research on algorithmic decision support by empirically investigating the effects of transparency on the use of algorithmic advice and identifying the underlying mechanism. The findings also provide practical guidance on how to promote the acceptance of algorithmic advice that is valuable to both individual users and practitioners.

Are engineers more likely to avoid algorithms after they see them err? A longitudinal study

ABSTRACT Research suggests the superior predictive capabilities of algorithms compared to humans. However, people's reluctance to use algorithms after witnessing their inaccuracies has hindered their widespread adoption. Studies have explored this reluctance, but little is known about how different people use algorithms. We focused on algorithm utilisation by engineers, conducting two longitudinal ecological momentary assessment studies outside the lab to explore differences in how engineers and non-engineers engage with inaccurate algorithms. These studies involved 427 participants, predicting currency exchange rates or maximum weather temperatures over nine days based on the judge-advisor system framework. Our results showed a significant three-way interaction between the effects of advice source, whether participants were engineers or non-engineers, and time. Specifically, the trend of inaccurate algorithm use significantly decreased over time for engineers, highlighting the importance of considering the end-users when implementing algorithms.

GPT has become financially literate: Insights from financial literacy tests of GPT and a preliminary test of how people use it as a source of advice

We assess the ability of GPT–a large language model–to serve as a financial robo-advisor for the masses, by using a financial literacy test. Davinci and ChatGPT based on GPT-3.5 score 66% and 65% on the financial literacy test, respectively, compared to a baseline of 33%. However, ChatGPT based on GPT-4 achieves a near-perfect 99% score, pointing to financial literacy becoming an emergent ability of state-of-the-art models. We use the Judge-Advisor System and a savings dilemma to illustrate how researchers might assess advice-utilization from large language models. We also present a number of directions for future research.

Can you trust this source? Advice taking in borderline personality disorder

Research suggests that patients with borderline personality disorder (BPD) share a range of cognitive biases with patients with psychosis. As the disorder often manifests in dysfunctional social interactions, we assumed associated reasoning styles would be exaggerated in a social setting. For the present study, we applied the Judge-Advisor System by asking participants to provide initial estimates of a person’s age and presumed hostility based on a portrait photo. Afterwards, we presented additional cues/advice in the form of responses by anonymous previous respondents. Participants could revise their estimate, seek additional advice, or make a decision. Contrary to our preregistered hypothesis, patients with BPD (n = 38) performed similarly to healthy controls (n = 30). Patients sought the same number of pieces of advice, were equally confident, and used advice in similar ways to revise their estimates. Thus, patients with BPD did trust advice. However, patients gave higher hostility ratings to the portrayed persons. In conclusion, patients with BPD showed no cognitive biases in seeking, evaluating, and integrating socially provided information. While the study implies emotional rather than cognitive biases in the disorder, cognitive biases may still prove to be useful treatment targets in order to encourage delaying and reflecting on extreme emotional responses in social interactions.

A meta-analysis of the weight of advice in decision-making

The degree to which people take advice, and the factors that influence advice-taking, are of broad interest to laypersons, professionals, and policy-makers. This meta-analysis on 346 effect sizes from 129 independent datasets (N = 17, 296) assessed the weight of advice in the judge-advisor system paradigm, as well as the influence of sample and task characteristics. Information about the advisor(s) that is suggestive of advice quality was the only unique predictor of the overall pooled weight of advice. Individuals adjusted estimates by 32%, 37%, and 48% in response to advisors described in ways that suggest low, neutral, or high quality advice, respectively. This indicates that the benefits of compromise and averaging may be lost if accurate advice is perceived to be low quality, or too much weight is given to inaccurate advice that is perceived to be high quality. When examining the three levels of perceived quality separately, advice-taking was greater for subjective and uncertain estimates, relative to objective estimates, when information about the advisor was neutral in terms of advice quality. Sample characteristics had no effect on advice-taking, thus providing no evidence that age, gender, or individualism influence the weight of advice. The findings contribute to current theoretical debates and provide direction for future research.

The advice less taken: The consequences of receiving unexpected advice

AbstractAlthough new information technologies and social networks make a wide variety of opinions and advice easily accessible, one can never be sure to get support on a focal judgment task. Nevertheless, participants in traditional advice taking studies are by default informed in advance about the opportunity to revise their judgment in the light of advice. The expectation of advice, however, may affect the weight assigned to it. The present research therefore investigates whether the advice taking process depends on the expectation of advice in the judge-advisor system (JAS). Five preregistered experiments (totalN= 2019) compared low and high levels of advice expectation. While there was no evidence for expectation effects in three experiments with block-wise structure, we obtained support for a positive influence of advice expectation on advice weighting in two experiments implementing sequential advice taking. The paradigmatic disclosure of the full procedure to participants thus constitutes an important boundary condition for the ecological study of advice taking behavior. The results suggest that the conventional JAS procedure fails to capture a class of judgment processes where advice is unexpected and therefore weighted less.

Algorithmic versus Human Advice: Does Presenting Prediction Performance Matter for Algorithm Appreciation?

ABSTRACT We propose a theoretical model based on the judge-advisor system (JAS) and empirically examine how algorithmic advice, compared to identical advice from humans, influences human judgment. This effect is contingent on the level of transparency, which varies with whether and how the prediction performance of the advice source is presented. In a series of five controlled behavioral experiments, we show that individuals largely exhibit algorithm appreciation; that is, they follow algorithmic advice to a greater extent than identical human advice due to a higher trust in an algorithmic than human advisor. Interestingly, neither the extent of higher trust in algorithmic advisors nor the level of algorithm appreciation decreases when individuals are informed of the algorithm’s prediction errors (i.e., upon presenting prediction performance in an aggregated format). By contrast, algorithm appreciation declines when the transparency of the advice source’s prediction performance further increases through an elaborated format. This is plausibly because the greater cognitive load imposed by the elaborated format impedes advice taking. Finally, we identify a boundary condition: algorithm appreciation is reduced for individuals with a lower dispositional need for cognition. Our findings provide key implications for research and managerial practice.

Probing the Hypersalience Hypothesis-An Adapted Judge-Advisor System Tested in Individuals With Psychotic-Like Experiences.

Individuals with psychotic-like experiences and psychosis gather and use information differently than controls; in particular they seek and rely on less information or over-weight currently available information. A new paradigm, the judge-advisor system, has previously been used to investigate these processes. Results showed that psychosis-prone individuals tend to seek less advice but at the same time use the available advice more. Some theoretical models, like the hypersalience of evidence-matching hypothesis, predict that psychosis-prone individuals weight recently available information to a greater extent and thus provide an explanation for increased advice-weighting scores in psychosis-prone individuals. To test this model, we adapted the previously used judge-advisor system by letting participants receive consecutively multiple pieces of advice. To meet this aim, we recruited a large MTurk community sample (N = 1,396), which we split in a group with high levels of psychotic-like experiences (at least 2 SD above the mean, n = 80) and a group with low levels of psychotic-like experiences (maximum 0.5 SD above the mean, n = 1,107), using the Community Assessment of Psychic Experiences' positive subscale. First, participants estimated five people's age based on photographs. Then, they received consecutive advice in the form of manipulated age estimates by allegedly previous participants, with outliers in some trials. After each advice, participants could adjust their estimate. This procedure allowed us to investigate how participants weighted each currently presented advice. In addition to being more confident in their final estimates and in line with our preregistered hypothesis, participants with more frequent psychotic-like experiences did weight currently available advice more than participants with less frequent psychotic-like experiences. This effect was especially pronounced in response to outliers, as fine-grained post-hoc analysis suggested. Result thus support models predicting an overcorrection in response to new incoming information and challenges an assumed general belief inflexibility in people with psychotic experiences.

The role of decision confidence in advice-taking and trust formation.

In a world where ideas flow freely across multiple platforms, people must often rely on others' advice and opinions without an objective standard to judge whether this information is accurate. The present study explores the hypothesis that an individual's internal decision confidence can be used as a signal to learn the accuracy of others' advice, even in the absence of feedback. According to this "agreement-in-confidence" hypothesis, people can learn about an advisor's accuracy across multiple interactions according to whether the advice offered agrees with their own initial opinions, weighted by the confidence with which these initial opinions are held. We test this hypothesis using a judge-advisor system paradigm to precisely manipulate the profiles of virtual advisors in a perceptual decision-making task. We find that when advisors' and participants' judgments are independent, people can correctly learn advisors' features, like their accuracy and calibration, whether or not objective feedback is available. However, when their judgments (and thus errors) are correlated-as is the case in many real social contexts-predictable distortions in trust can be observed between feedback and feedback-free scenarios. Using agent-based simulations, we explore implications of these individual-level heuristics for network-level patterns of trust and belief formation. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Advice weighting as a novel measure for belief flexibility in people with psychotic-like experiences

Jumping to conclusions and bias against disconfirmatory evidence are two cognitive biases common in people with psychotic-like experiences and psychosis. However, many participants show comprehension problems doing traditional tasks; new paradigms with additional applied scenarios are thus needed. A large MTurk community sample (N = 1422) was recruited and subdivided into participants with high levels of psychotic-like experiences (at least 2 SD above the mean, n = 79) and participants with low levels of psychotic-like experiences (maximum 0.5 SD above the mean, n = 1110), based on the positive subscale of the Community Assessment of Psychic Experiences (CAPE). In the context of a judge-advisor system, participants made an initial estimate and then received advice that was either confirmatory or disconfirmatory. Participants then gave a new, possibly revised estimate and were allowed to seek additional advice. Participants with high levels of psychotic-like experiences gave their final assessment after receiving significantly less advice and were significantly more confident in their decision than participants with low psychotic-like experiences, in line with previous studies on jumping to conclusions and overconfidence. Contrary to the hypothesis and earlier studies, however, no deficit in belief revision was found. In fact, participants with high psychotic-like experiences weighted advice significantly higher in the condition with disconfirmatory advice, but only for the first advice they received. The increased weighting of a single piece of disconfirmatory advice can be explained by the hypersalience of evidence-hypothesis matches theory, according to which more weight is attached to the most recently available information.

The impact of advice distance on advice taking: Evidence from an ERP study

To use ERPs to investigate the impact of advice distance on advice taking in the Judge-Advisor System (JAS) paradigm, behavioural and ERP data were recorded from 20 subjects. The behavioural discrepancies and EEG characteristics of advice taking, as affected by the advice distances, were compared. The results showed that the relationship between the modification rate of a decision maker's initial estimation and the advice distance exhibited an inverse U-shaped curve. During the advice evaluation stage, different advice distances induced the feedback-related negativity (FRN) and subsequent P300 components. The amplitude of the FRN increased as the advice distance increased, while the amplitude of P300 in the zero distance condition was significantly larger than those in the intermediate and far distance conditions. These findings indicate that the advice evaluation process of a decision maker is related to conflict perception, prediction error and motivational or affective significance.

The Use of Recommendations on Physician Rating Websites: The Number of Raters Makes the Difference When Adjusting Decisions

ABSTRACTPhysician rating websites allow users to check physicians’ profiles, write reviews, or rate their performance. The opinion of other users regarding a physician can affect our decision to visit her/him. To investigate the specific role of the number of users rating a physician when choosing a physician with support of these platforms, we used a Judge-Advisor System in which participants answered their likelihood to visit a physician before and after seeing the recommendations of others. Within the experiment, three conditions were presented: high and low number of reviewers recommending a physician, and no recommendations. We found that the participants’ likelihood to visit a physician varied with respect to the displayed physician characteristics on the platform. Importantly, after the recommendation of others was presented, participants’ likelihood to visit the physician changed significantly. The participants’ adjusted response was significantly closer to the recommendation coming from a higher number of users, which indicate that this online, social media cue influences our decision to visit physicians. Comments and ratings on physician ratings are generally positive, but we show that negative ratings have a direct negative influence in the decision to visit a physician. We suggest administrators of these platforms to pay special attention to the content that users upload.

Are Depressed People More or Less Susceptible to Informational Social Influence?

When making judgments and decisions, people suffering from depression are often faced with opinions and advice from others (e.g., from their therapists) but it is unclear how their psychopathology alters the utilization of such information. This study is the first to examine whether depressed people are more or less susceptible to informational social influence. To this end, we employed the Judge–Advisor-System, which allows for a pure test of how people utilize information from others. We found that depressed participants had significantly higher advice taking values than non-depressed participants, which was mediated by self-esteem. A fine-grained analysis of these group differences revealed that depressed participants were more likely to revise their initial estimates after receiving advice than non-depressed people. Yet, once having decided to revise their estimates, depressed people did not weight advice more heavily. Theoretical implications concerning two qualitatively independent effects of depression on advice taking are discussed.

The Influence of Prime of Positive Self-label on Advice-taking

Advice taking, which is a decision-making process formulated by decision-makers with the reference of others' suggestions, has become a research hotspot as the interactive process between advisers and decision-makers, and receives more attention in the field of behavioral decision making. In this study, we proposed that the labeling for decision maker is a potential predictor in the advice taking process. Specifically, whether the decision-maker takes the advice from others will be influenced effectively by the descriptions(labeling stick to the decision maker). This study assumed that labeling in advance will exert an influence on advice taking by different mechanism in different decision-making scenarios. Using widely-used "Judge-Advisor System(JAS)" experimental model, this paper explored the influence of the prime of positive self-labeling on advice taking in three decision making scenarios which were decided by whether there are hopes to benefit or not, and whether they are public or in private.3 studies have been done in this research in a gradually deepening logical sequence. Study 1 included preliminary and formal experiments, and used 2(Positive Self-labeling Priming: Yes/No) × 2(Hope of Benefit: Yes/No) between-subject design to explore whether the advice taking process and the cognitive dissonance of the decision maker will be influenced by positive self-labeling in the public condition. Study 2 used between-subject design to further explore whether the prime of positive self-labeling will influence the advice taking in the private condition of the scenario that there is likely no benefit will be available. Further, study 3 explored whether the positive emotion of decision-makers will be promoted by the prime of positive self-labeling of decision makers, and whether positive meaning-finding acts as a mediator between the positive emotion and advice taking under the public condition of hopefully benefiting decision scene. The participants are university students and the number is 91(experiment 1), 135(experiment 2) and 96(experiment 3) respectively. The results found that the prime of the positive self-labeling of decision makers would promote them to take the advice no matter in the hopefully benefiting decision scene or in the likely no benefit scene, but it only caused cognitive dissonance of decision makers in no- benefiting decision scene in the public condition, and there is not a significant effect on the advice taking and cognitive dissonance. It also showed that positive meaning-finding mediated the effect between the positive emotion and advice taking under the public condition of hopefully benefiting decision scene. The implications, limitations and future directions of the study were discussed as well.

Factors affecting decision makers' preference for unshared information.

Two experiments examined solicitation of information in a group structured as a judge–advisor system (JAS) with 1 group member designated as the decision maker and the other 2 members as advisors. The decision maker solicited information from 2 advisors. One advisor’s information was shared in common with the decision maker, and the other’s information was predominantly unshared. In 2 experiments, decision makers asked for more information from the advisor with unshared information and rated this advisor’s information as more important and influential than the advisor with only redundant, shared information. When decision makers were not limited in the amount of information they could ask for, decision makers significantly increased requests for information from the advisor with shared information but not the advisor with unshared information. Experiment 2 found that whether or not an advisor agreed with the decision maker did not affect decision makers’ preference for the advisor with unshared information.

Do we listen to advice just because we paid for it? The impact of advice cost on its use

When facing a decision, people often rely on advice received from others. Previous studies have shown that people tend to discount others’ opinions. Yet, such discounting varies according to several factors. This paper isolates one of these factors: the cost of advice. Specifically, three experiments investigate whether the cost of advice, independent of its quality, affects how people use advice. The studies use the Judge–Advisor System (JAS) to investigate whether people value advice from others more when it costs money than when it is free, and examine the psychological processes that could account for this effect. The results show that people use paid advice significantly more than free advice and suggest that this effect is due to the same forces that have been documented in the literature to explain the sunk costs fallacy. Implications for circumstances under which people value others’ opinions are discussed.

Tell Me Something I Don't Know

This article examined strategic requests for information in a group structured as a judge advisor system (JAS) with one group member designated as the decision maker and the other two members as advisors. The decision maker could solicit advice from the two advisors. One advisor's advice contained only information shared in common with the decision maker, and the other's advice contained half unshared information. The authors proposed an alternative to research on information sampling and mutual enhancement and predicted that the decision maker would prefer information from the advisor with more unshared information. The decision maker solicited more information from the advisor with more unshared information and rated this advisor's information as more important and influential than the advisor with only shared information.

Factors affecting the acceptance of expert advice

This paper expands research on the judge advisor system (JAS) by examining advice utilization and trust. Experiment 1 examined five factors that could increase utilization of expert advice: judge's trust in the advisor, advisor confidence, advisor accuracy, judge's prior relationship with the advisor, and judge's power to set payment to the advisor. While judge's trust and advisor confidence correlated with the judge matching the advisor's advice, a stepwise regression found that of the five variables, advisor confidence was the only significant predictor of the judge matching the advisor. Experiment 2 examined trust without the role assignment to judge or advisor. While trust expressed in partner was not higher for the judge than the advisor in Experiment 1, in Experiment 2 trust in partner expressed by the low-expertise dyad member was higher than trust expressed by the high-expertise dyad member. Results from the two experiments are discussed in the context of Sniezek and Van Swol (2001).

Information Sampling and Confidence Within Groups and Judge Advisor Systems

This study compared how the judge advisor system (JAS) and unstructured groups discuss common and unique information. Three differences between the two were measured for their effect on communication of information. These were responsibility for decision, consensus seeking, and equity of participation. Participants were given information about two drugs and had to decide which of the two to market. Half met in unstructured 3-person groups and made the decision as a group. The rest met in JASs whereby the judge discussed the information with each advisor separately and then made the decision individually. Advisors mentioned but did not repeat a higher proportion of unique information than group members. Judges felt more responsible for, reported putting more effort toward, and had higher confidence in the decision than did group members. There was more inequity of participation and consensus seeking in JASs compared to groups. Differences are discussed in light of results found on information exchange.