Increased Decision Accuracy Research Articles

Trust and reliance on AI — An experimental study on the extent and costs of overreliance on AI

Decision-making is undergoing rapid changes due to the introduction of artificial intelligence (AI), as AI recommender systems can help mitigate human flaws and increase decision accuracy and efficiency. However, AI can also commit errors or suffer from algorithmic bias. Hence, blind trust in technologies carries risks, as users may follow detrimental advice resulting in undesired consequences. Building upon research on algorithm appreciation and trust in AI, the current study investigates whether users who receive AI advice in an uncertain situation overrely on this advice — to their own detriment and that of other parties. In a domain-independent, incentivized, and interactive behavioral experiment, we find that the mere knowledge of advice being generated by an AI causes people to overrely on it, that is, to follow AI advice even when it contradicts available contextual information as well as their own assessment. Frequently, this overreliance leads not only to inefficient outcomes for the advisee, but also to undesired effects regarding third parties. The results call into question how AI is being used in assisted decision making, emphasizing the importance of AI literacy and effective trust calibration for productive deployment of such systems.

Flexible visualization of a 3rd party Intrusion Prevention (Security) tool: A use case with the ELK stack

A difficult aspect of cyber security is the ability to achieve automated real time intrusion prevention across various sets of systems. To this extent, several companies are offering comprehensive solutions that leverage an “accuracy of scale” and moving much of the intelligence and detection on the Cloud, relying on an ever-growing set of data and analytics to increase decision accuracy. Often, they provide tools to visualize the decision workflows in attack prevention (as well as tune the algorithm) but those solutions are not always practical as companies see the problem as “global” that is, from a unified Cyber-security standpoint. However, a key to a successful Cyber-security program is transparency and trust: from an experimental team viewpoint, this specifically means having the ability to immediately see what and from where, who has been blocked and being able to inform the community in case of a revoked access without the need for filing a “ticket” (that may eventually be answered) – in other words, rapid response to their user-base is essential but solutions targeting “sub-groups” in an organization are not often available.We have come up with a versatile solution leveraging the ELK stack (Elasticsearch, Logstash, & Kibana) and an IPS (Intrusion Prevention System) based WAF (Web Application Firewall) from Signal Sciences. Signal Science allows the streaming of detailed logs in a Logstash format suitable for custom solutions for visualization. By combining these two tools, we have strengthened our security posture and enabled individual experiments to monitor their own traffic. Specifically, the IPS WAF provides unique data such as country of origin, protocol, response code, source IP, and paths accessed.In this contribution, we will show how we engineered a visualization solution so experiment groups could access a dashboard with predefined graphs but also, where they can create individual customizable dashboards used to display blocked traffic and troubleshoot latency issues. We will discuss the details and procedures for developing and configuring these tools and how it benefits cyber security postures across our scientific based environment.

Elastic Cloud Logs Traces, Storing and Replaying for Deep Machine Learning

Abstract Storing logs and retrieving it in cloud computing environment is a critical task for Deep Machine Learning models. Time to load large sizes of logs and pre-processing them (prepare logs to be used in learning phase) is a significant issue to machine learning modules that influence training and decision phases. Storing sampled logs in an optimal way that consumes minimum desk storage with accurate and fast replay will give any cloud management system an advantage to start processing logs immediately. Using a practical dimensional reduction method to reduce logs sizes and recover them will improve cloud management system in taking right scaling action and will reduce deep machine learning time complexity. There are many methods in literature for feature extraction and dimensional reduction methods such as Principle Components Analysis (PCA) and wavelet transform. These two methods have a good reputation in processing data set for different goals in machine learning methods like image recognition, signal processing and prediction. In this work, an analysis and comparison is made between PCA and wavelet as methods of cloud logs (considered as data set) dimensional reduction and compression to reduce logs size and logs recover time in order to be replayed in a faster way. This will reduce logs processing complexity, logs foot print and increase decision accuracy by focusing on logs main feature in cloud data center management system.

Unleashing the BEAST: a brief measure of human social information use

Abstract Social information enables individuals to reduce uncertainty and increase decision accuracy across a broad range of domains. Intriguingly, individuals and populations consistently differ in social information use. Understanding the underlying causes of this variation has proven challenging due to the lack of a standardized paradigm to quantify social information use. Here we introduce the BEAST (Berlin Estimate AdjuStment Task); a brief (∼5-min), simple, and incentive-compatible behavioural task to quantify individuals' propensities to use social information. In the task, participants observe an image with a number of animals and estimate the total number. Next, they receive another person's estimate, after which they provide a second estimate. An individual's average adjustment quantifies their propensity to use social information. We found that individuals' propensity to use social information is consistent within the task, has considerable test–retest reliability over 9 months, generalizes to other social learning tasks, and correlates with established self-reported measures of social conformity and social proximity. The BEAST thus reliably captures individual variation in social information use. We conclude by highlighting the BEAST's potential to serve as a flexible framework to assess the determinants of human social information use.

Measuring Decision Accuracy and Confidence of Mock Air Defence Operators

This study aimed to understand more fully some of the factors that influence decisions as related to air defence in a naval vessel's operation room. The study considered the impact of decision criticality (DC) and task load (TL) on measures of accuracy, confidence, and within-subjects confidence-accuracy (W-S C-A; a measure of metacognition). Personality constructs, workload, and situational awareness were also assessed. Participants were allocated to either a high, moderate, or low TL condition. Each took part in a computer-generated simulated air defence scenario where they were required to make a range of decisions and provide a corresponding confidence rating for each decision taken. Results showed that low DC increased confidence in decisions and high DC increased decision accuracy. Thus, DC significantly impacts decision confidence and decision accuracy. In addition, those less tolerant of ambiguity were less accurate in their decision-making. Future studies should take account of these factors.

Effects of deep brain stimulation of the subthalamic nucleus on perceptual decision making

When faced with difficult decisions, people prefer to stay with the default. This status quo bias often leads to suboptimal choice behavior. Neurophysiological evidence suggests a pivot role of the Subthalamic Nucleus (STN) for overcoming such status quo bias in difficult decisions, but causal evidence is lacking. The present study investigated whether subthalamic deep brain stimulation (DBS) in patients with Parkinson’s disease (PD) influences the status quo bias.Eighteen PD patients treated with STN-DBS performed a difficult perceptual decision task incorporating intrinsic status quo option. Patients were tested with (ON) and without (OFF) active STN stimulation.Our results show that DBS of the STN affected perceptual decision making in PD patients depending on the difficulty of decision. STN-DBS improved difficult perceptual decisions due to a selective increase in accuracy (hit rate) that was independent of response bias (no effect on false alarm rate). Furthermore, STN-DBS impacted status quo bias as a function of baseline impulsivity. In impulsive patients, STN-DBS increased the default bias, whereas in less impulsive PD patients, DBS of the STN reduced the status quo bias.In line with our hypothesis, STN-DBS selectively affected the tendency to stick with the default option on difficult decisions, and promoted increased decision accuracy. Moreover, we demonstrate the impact of baseline cognitive abilities on DBS-related performance changes in PD patients.

Collective intelligence meets medical decision-making: the collective outperforms the best radiologist.

While collective intelligence (CI) is a powerful approach to increase decision accuracy, few attempts have been made to unlock its potential in medical decision-making. Here we investigated the performance of three well-known collective intelligence rules (“majority”, “quorum”, and “weighted quorum”) when applied to mammography screening. For any particular mammogram, these rules aggregate the independent assessments of multiple radiologists into a single decision (recall the patient for additional workup or not). We found that, compared to single radiologists, any of these CI-rules both increases true positives (i.e., recalls of patients with cancer) and decreases false positives (i.e., recalls of patients without cancer), thereby overcoming one of the fundamental limitations to decision accuracy that individual radiologists face. Importantly, we find that all CI-rules systematically outperform even the best-performing individual radiologist in the respective group. Our findings demonstrate that CI can be employed to improve mammography screening; similarly, CI may have the potential to improve medical decision-making in a much wider range of contexts, including many areas of diagnostic imaging and, more generally, diagnostic decisions that are based on the subjective interpretation of evidence.

Accuracy of teachers’ tracking decisions: short- and long-term effects of accountability

Bias in teachers’ judgment formation and decision making has long been acknowledged. More specifically, studies have repeatedly demonstrated discrepancies between teacher ratings of minority and majority students with similar academic profiles. Studies have also demonstrated that increasing accountability reduced bias. Little is known, however, about the effect of accountability and bias on the accuracy of decisions. This study investigated the short- and long-term effect of accountability priming on the accuracy of transition decisions. It considered both the extent to which teacher decision accuracy differed for minority and majority students with similar academic profiles (accuracy bias) and differences in levels of confidence for accurate versus erroneous decisions (metacognitive judgment bias). In a longitudinal experimental design, we presented 38 primary school teachers with 9 student vignettes at 3 points in time (baseline, post priming, and 6-month follow-up), varying students’ ethnic background, and asked them to make a school tracking decision for each student. We measured decision accuracy as well as teachers’ level of confidence for each decision. Accuracy and confidence levels were combined to provide two indices of metacognitive judgment accuracy. Results confirmed the hypothesis that accuracy of decisions would improve as a result of increased level of accountability. More specifically, we found that teachers made more accurate decisions after priming, whereby ethnic background differences disappeared. In addition, teachers’ metacognitions varied, whereby after priming decision accuracy was better matched with teachers’ confidence levels. Although accuracy levels were still higher at follow-up than at pre-test, the ethnic bias recurred. This study shows that increased levels of accountability are associated with not only increased decision accuracy but also reduced metacognitive judgment bias, especially in regard to minority students. It also demonstrates accountability may be an effective way of reducing systematic errors in decision making. Findings are discussed in terms of theory and current changes in educational practice.

Information transmission via movement behaviour improves decision accuracy in human groups

A major advantage of group living is increased decision accuracy. In animal groups information is often transmitted via movement. For example, an individual quickly moving away from its group may indicate approaching predators. However, individuals also make mistakes which can initiate information cascades. How responsive should individuals then be to escaping group members? Increasing responsiveness increases true positives (i.e. escape when a predator is present) but at the cost of increased false positives (i.e. escape when a predator is absent). Conversely, reducing responsiveness decreases not only false positives but also true positives, resulting in a fundamental trade-off in decision accuracy. Here we investigated how socially responsive individuals are to information transmission via movement. We performed a simulated predator detection task using human groups in which humans stepped forward if they wanted to escape. We confirm that this simple movement mechanism allows individuals in groups to simultaneously increase true positives and decrease false positives. The increase in the number of escapees over time during collective decisions depended on the personal information of the group members. Individual predator detection by only a few group members rarely resulted in anyone stepping forward. Individual predator detection by a quarter of the group often resulted in the entire group escaping. Finally, individual predator detection by at least half of the group led to a rapid escape of the whole group. Overall, the increase in the number of escapees over time followed a linear response. Since information transmission via movement is widespread in animal groups, this mechanism is expected to be relevant for many animal groups to improve decision accuracy.

Humans use social information to adjust their quorum thresholds adaptively in a simulated predator detection experiment

Quorum sensing is used in many biological systems to increase decision accuracy. In quorum sensing, the probability that an individual adopts a behavior is a nonlinear function of the number of other individuals adopting this behavior. From an optimal decision-making perspective, individuals should adjust their quorum threshold to the particulars of the decision problem. Recent work predicts that a key factor here is the quality of social information. In particular, it is predicted that individuals should adjust their quorum thresholds such that it lies in between the average true and false positive rate of the other group members. We here test this prediction with a predator detection experiment. First, human groups observed a group of animals (projected on a white screen) in which a predator was present or absent, and each individual made an independent decision to escape or not. Second, individuals received social information on the decisions of their group members, after which individuals decided again. This social information, however, did not represent their own decisions but consisted of responses that either came from a high-performing group (i.e., many individuals detecting the predator) or from a low-performing group (i.e., few individuals detecting the predator). We found that individuals adaptively adjust their quorum threshold to the quality of the social information: when receiving social information from high-performing groups, individuals employed higher quorum thresholds than when receiving information from low-performing groups. Our study demonstrates that humans can quickly evaluate the quality of publicly available information and adaptively adjust their decision rules.

Incentive Mechanisms, Loan Decisions and Credit Rationing: A Framed Field Experiment on China's Responsibility System for Rural Credit

Incentive mechanisms are important attributes to financial decision-making. A framed field experiment was conducted to test loan decision behaviors. It showed that lenders’ risk-aversion and behavioral responses resulted in credit rationing under certain incentive schemes. In the experiment, loan officers from Rural Credit Cooperatives (RCCs) in Shandong, China were recruited to evaluate randomly selected loan applications and make lending decisions. All the loan files were previously approved with known performance and repayment status. Each loan officer was randomly assigned to one of two incentive groups. One was analogous to pure Personal Responsibility System (PRS), which provided bonuses to loan officers for approved loans that were in performance and imposed penalties on non-performing loans (NPLs). And the other was PRS with additional penalties for Type II error (i.e. rejecting loans that would have been good). The two groups were further randomized over prior knowledge about probability distribution of the application pool. Results showed that PRS made a risk-averse loan officer inclined to reject loans to avoid risk of penalty. This side effect generated credit rationing, increased Type II error in loan classification and lowered the interest returns of RCCs. Providing prior information about the application pool helped to increase decision accuracy. In theory, this study extended the incentive mechanism design under uncertainty to a behavioral scope. In practice, it can lead to increased profitability in financial institutes, alleviated credit rationing and stabilized credit supply in the market.

Using Salience to Guide Searches in Digital Displays

Technological innovations including software applications and the internet have increased the ability to collect and store information. However, the increasing amounts of information are used by operators who are constrained by their limited cognitive processing capacity. One common approach to improve the search for information in digital displays is the concept of increased stimulus-driven salience to guide the user. To examine search efficiency, sufficiency and decision accuracy we conducted an experiment to explore the effect of the congruence of salience and relevance and the effect of salience manipulation (position or size). Our study adapted the Wason selection task and manipulated salience by size in a tag cloud and position in a list. The results show that increased salience of relevant items does not insure an optimal search. Care should be taken to match salience with relevance; however, increased salience should not be expected to promote a sufficient search with increased decision accuracy.

Experiments with humans indicate that decision accuracy drives the evolution of niche width

One theory to explain the high incidence of niche specialization in many animals is that it reduces attentional load during resource-seeking behaviour and thus leads to more accurate resource selection. A recent neural network model refined the predictions of this theory, indicating that a cognitive advantage in specialists is likely to occur under realistic ecological conditions, namely when 'mistakes' (i.e. selection of non-host resources) contribute moderately but positively to fitness. Here, we present a formal empirical test of the predictions of this model. Using a human-computer interactive, we demonstrate that the central prediction of the model is supported: specialist humans are more accurate decision-makers than generalists when their mistakes are rewarded, but not when mistakes are punished. The idea that increased decision accuracy drives the evolution of niche width in animals has been supported in almost all empirical systems in which it has been investigated. Theoretical work supports the idea, and now the predictions of a key theoretical model have been demonstrated in a real biological information-processing system. Considering these interlocking pieces of evidence, we argue that specialization through increased decision accuracy may contribute significantly, along with other mechanisms, to promote niche specialization in animals.

Comparison of the Clinical Diagnosis and Subsequent Autopsy Findings in Medical Malpractice

The rate of diagnostic error has not declined over the last 30 years despite the improvement in diagnosis methods. Today, the probability to encounter a major diagnostic change in autopsy is still high. The special autopsy that provides clinicians the necessary feedback is performed seldom in our country. This study has been started upon the claim that the malpractice cases in scope of forensic cases could be a very useful database. About 525 cases with faulty medical application that resulted in death have been encountered according to the cases that have been sent to the Forensic Medical Institution, which has been the consultative authority of the Ministry of Justice between the years 2001 and 2005. Among these, 375 cases on which autopsies have been conducted are included in the scope of the study. The cases with and without major diagnostic changes have been compared in aspects of gender, age groups, and period spent in health units. When the concordance between autopsy and clinical diagnosis for the 375 cases that were included in the study is examined, it can be seen that clinical diagnosis is verified with autopsy in 49.1% of the cases whereas neither autopsy nor clinical diagnosis was encountered in 14.7% of them. The diseases where diagnostic faults are mostly encountered are ruptured aortic aneurysm, myocardial infarction and pneumonia. The majority of the cases without a clinical diagnosis comprises of complications that occur after operations. In the study, considerable major diagnostic changes were realized after autopsy, which shows that autopsy has an important place in Turkey in spite of the improved methods of diagnosis. Also, in the cases with major diagnostic changes, the medical malpractice rate was high. This shows that autopsy is influential in revealing medical malpractice failures. In the study, we found out that the number of autopsies was high in the cases with a malpractice claim. We aim to increase this number. We are sure that such increment would increase decision accuracy.

Linkage analysis in cases of serial burglary: comparing the performance of university students, police professionals, and a logistic regression model

University students, police professionals, and a logistic regression model were provided with information on 38 pairs of burglaries, 20% of which were committed by the same offender, in order to examine their ability to accurately identify linked serial burglaries. For each offense pair, the information included: (1) the offense locations as points on a map, (2) the distance (in km) between the two offenses, (3) entry methods, (4) target characteristics, and (5) property stolen. Half of the participants received training informing them that the likelihood of two offenses being committed by the same offender increases as the distance between the offenses decreases. Results showed that students outperformed police professionals, that training increased decision accuracy, and that the logistic regression model achieved the highest rate of success. Potential explanations for these results are presented, focusing primarily on the participants' use of offense information, and their implications are discussed.

Multimodal mixed messages: the use of multiple cues allows greater accuracy in social recognition and predator detection decisions in the mosquitofish, Gambusia holbrooki

Information gathered by animals through different sensory modalities at any given point may be aligned, that is, consistent across different senses, or in conflict, where different senses provide different information. This latter instance may occur for a variety of reasons, including differences in speed of dispersal and persistence of some cues relative to others. By gathering information using multiple sensory modalities simultaneously, animals may be able to mediate this conflict and increase decision accuracy. This study examined the use of different sensory modalities over a short spatial range by mosquitofish in locating conspecifics and avoiding a cryptic sympatric predator. Fish were provided with visual and/or chemical cues either separately or simultaneously. Where visual and chemical cues were provided simultaneously, they were either aligned or in conflict. As expected, fish with access to aligned cues generally performed best, approaching a conspecific shoal sooner and spending longer in proximity to the shoal and avoiding approaching the predators for longer and spending less time in proximity to them. Fish with access to conflicting cues performed as well as those with access to aligned cues in overall time spent in proximity to predators, while they were more wary of conspecific visual cues presented in association with predator chemical cues. However, fish with access to conflicting cues were slower to approach conspecific visual cues and, crucially, faster to approach predator visual cues than fish with access to aligned cues. This suggests that although multimodal cue use generally promotes accuracy, the potential remains for conflicting cues to generate risky decisions. Key words: chemosense, decision-making, fish, olfaction, shoaling, vision. [Behav Ecol]

An Experimental Examination of Information Technology and Compensation Structure Complementarities in an Expert System Context

This paper investigates the interaction between compensation structures and expert system technology. One hundred twenty-two participants selected a contract (fixed pay or performance-contingent incentives) and performed one of two risk-rating tasks either in the presence or absence of an externally valid expert system. For both tasks, the expert system increased decision accuracy only for participants selecting performance-contingent incentives, and these incentives increased decision accuracy only in the presence of the expert system, consistent with a complementary interaction. The results support the view that expert system technology changes task requirements and, hence, may also change self-selection and effort compensation contracting effects. The results provide evidence about the interdependence of organization, system, and individual factors in determining task performance, and suggest that organizational performance may be improved through the joint implementation of expert systems and performance-contingent incentives.

Improving decision accuracy where base rates matter: The prediction of violent recidivism

AbstractBase rates are vital in predicting violent criminal recidivism. However, both lay people given simulated prediction tasks and professionals milking real life predictions appear insensitive to variations in the base rate of violent recidivism. Although there are techniques to help decision makers attend to base rates, increased decision accuracy is better sought in improved actuarial models as opposed to improved clinicians.

Interaction with Others Increases Decision Confidence but Not Decision Quality: Evidence against Information Collection Views of Interactive Decision Making

We present three studies of interactive decision making, where decision makers interact with others before making a final decision alone. Because the theories of lay observers and social psychologists emphasize the role of information collection in interaction, we developed a series of tests of information collection. Two studies with sports collection show that interaction does not increase decision accuracy or meta-knowledge (calibration or resolution). The simplest test of information collection is responsiveness - that people should respond to information against their position by modifying their choices or at least lowering their confidence. Studies using traditional scenarios from the group polarization literature show little responsiveness, and even "deviants," who interact with others who unanimously disagree with their choice, frequently fail to respond to the information they collect. The most consistent finding is that interaction increases people′s confidence in their decisions in both sports predictions and risky shift dilemmas. For predictions, confidence increases are not justified by increased accuracy. These results question theories of interaction which assume that people collect information during interaction (e.g., Persuasive Arguments Theory). They also question the labeling of previous results as "shifts" or "polarization." We suggest that interaction is better understood as rationale construction than as information collection - interaction forces people to explain their choices to others, and a variety of previous research in social psychology has shown that explanation generation leads to increased confidence. In Study 3, we provide a preliminary test of rationale construction by showing that people increase in confidence when they construct a case for their position individually, without interaction.