Numerical Probability Research Articles

Cultivating credibility with probability words and numbers

AbstractRecent research suggests that communicating probabilities numerically rather than verbally benefits forecasters’ credibility. In two experiments, we tested the reproducibility of this communication-format effect. The effect was replicated under comparable conditions (low-probability, inaccurate forecasts), but it was reversed for low-probability accurate forecasts and eliminated for high-probability forecasts. Experiment 2 further showed that verbal probabilities convey implicit recommendations more clearly than probability information, whereas numeric probabilities do the opposite. Descriptively, the findings indicate that the effect of probability words versus numbers on credibility depends on how these formats convey directionality differently, how directionality implies recommendations even when none are explicitly given, and how such recommendations correspond with outcomes. Prescriptively, we propose that experts distinguish forecasts from advice, using numeric probabilities for the former and well-reasoned arguments for the latter.

A Coherentist Justification of Induction

In this paper I offer a coherentist justification of induction along the lines of a Sellarsian coherence theory. On this coherence theory, a proposition (or a hypothesis) is justified if we can answer all objections raised against it in our social practice of demanding justification and responding to such demands. On the basis of this theory of justification, I argue that we are justified in accepting the uniformity of nature partly because we have no alternative but to accept it for rationally pursuing our epistemic goal. In addition, my coherentist view explains which inductive inferences it is rational to accept. Furthermore, my coherentist view also explains why we can hardly determine a particular numerical value x such that we can draw the conclusion ‘it is rational to accept that p’ from the premise ‘the numerical probability of p is greater than or equal to x’.

What’s in a number? Communicating risk through real-world examples

“Don’t worry, it’s more dangerous driving here than having the anaesthetic”. This statement may be reassuring, but it isn’t true. Despite its falsehood, it is often quoted as it evokes a visceral response that suggests the procedure is low risk.Communicating risks and probabilities to our patients is an everyday occurrence for clinicians - but are we correctly understood? Are we over-reliant on numerical expressions of risk? Could comparative examples of probabilities aid the understanding of risk?In this article the communication of numerical probability in healthcare is examined and a table of commonly used probabilities aims to improve understanding by converting numerical expressions into more tangible, real-world, examples.To allay a patient’s fear of awareness it is perhaps reassuring to know that you are more likely to guess my four-digit bank personal identification number first time than experience awareness during a general anaesthetic. Conversely, and providing much less reassurance, taking ten trips into space with NASA is safer, in terms of 30-day mortality, than an emergency laparotomy in the UK.Clinicians are invited to trial this method for communicating probability, in carefully chosen circumstances, and read the cues and outcomes from the communication that follows.

Keynes’s Method Has Nothing to Do With a Common Discourse or Ordinary Language Logic: Keynes’s Method, Which Involved the Use of Inexact Measurement in Probability and Statistics, Based on Approximation, Was Based Directly on Boole’s Mathematical Logic and Algebra

It is impossible to correctly grasp Keynes’s method of analysis in the A Treatise on Probability in 1921 if the work of G Boole is ignored. Unfortunately, all Post Keynesian, Institutionalist and Heterodox economists , who have published work on Keynes in the 20th and 21st centuries, have done just that. George Boole, and not J M Keynes in his 1921 A Treatise on Probability, put forth the first technically advanced mathematical and logical treatment of a logical theory of probability in 1854 in his The Laws of Thought that was based on a logic of propositions about events or outcomes and not the events or outcomes themselves. This logic is a mathematical logic and has absolutely nothing to do with an ordinary discourse human logic, which involves the use of a common sense language between humans. Given that Keynes built his A Treatise on Probability directly on the mathematical and logical approach and foundation of G Boole’s Boolean algebra and logic, it is simply impossible for Keynes’s approach in his A Treatise on Probability to have been based on a logic of ordinary language as claimed by Carabelli (1985, 1988, 2003), Chick (1998), and Chick and Dow (2001). Keynes is supposed to have had some kind of unique, unclear and peculiar approach to analysis based on intuition that can’t be discerned, according to Anna Carabelli (1985, 1988, 2003) and other heterodox economists. Carabelli argues that Keynes was anti-logicist, anti-empiricist, anti-positivist, anti-rationalist, and anti-formalist in his method, as well as being anti-mathematical. It is quite impossible for Keynes to have opposed all of these positions and still write Parts II and V of the A Treatise on Probability, which provide formal, mathematical, and logicist underpinnings to his approach of Inexact measurement and Approximation that leads directly to Keynes’s specification of lower and upper bounds on all probabilities and outcomes except for areas of application involving his Principle of Indifference and relative frequencies that have passed an application of the Lexis-Q test (exploratory data analysis and /or goodness of fit tests). Keynes’s inductive logic of Part III of the A Treatise on Probability is built directly on the method of inexact measurement and approximation of Part II of the A Treatise on Probability. This involves Keynes’s use of a modified version of Boole’s Problem X that he solved on pp.192-194 of the A Treatise on Probability and used on pp.234-237 and 254-257. Keynes’s development of the concept of finite probability, applicable to both numerical and non numerical probabilities, was a necessary prerequisite for understanding Keynes’s work in Part III on induction and analogy. Given Keynes’s work on the relationship between probability and induction in Part III of the A Treatise on Probability it is impossible for Keynes to have been a rationalist, as claimed by R.O.Donnell. Keynes’s work in Part III of the A Treatise on Probability is then a prerequisite for his work in Part V of the A Treatise on Probability. We can now see that it is impossible to grasp Keynes’s work in Part III of the A Treatise on Probability unless Part II of the A Treatise on Probability is understood and it is not possible to grasp Part V unless Part III of the A Treatise on Probability has been digested. Heterodox economists, in general, study only chapter III of Part I of the A Treatise on Probability. No Heterodox economist has ever studied Part II of the A Treatise on Probability.

Keynes Demonstrated in Chapter 15 of the A Treatise on Probability That His Non Numerical Probabilities Are Identical to Boole’s Constituent Probabilities: It Is Mathematically Impossible for Keynes’s Non Numerical Probabilities to Be Ordinal Probabilities

The claim that Keynes’s non numerical probabilities are ordinal probabilities was shown to be mathematically impossible by Keynes in chapter 15 of the A Treatise on Probability on pp.162-163 and in chapter 17 on pp.186-194. Keynes’s non numerical probabilities are identical to Boole’s constituent probabilities. Keynes improved on Boole’s technique and was able to solve Boolean problems much quicker than it took Boole to solve the problems. Part II of the A Treatise on Probability is nearly identical to the analysis provided in his two Cambridge University Fellowships in 1907 and 1908.

The Foundation for Keynes’s Decision Theoretic Approach to Uncertainty in the General Theory (1936) and the 1937 Quarterly Journal of Economics Reply Article Was Acknowledged by Keynes to Be Based on His General Theory of Decision Making Presented in the A Treatise on Probability (1921) in the 1937–38 Correspondence with H. Townshend

Keynes made it clear to Townshend in their 1937-38 exchanges that Townshend’s assessment, that Keynes ‘s theory of liquidity preference in the General Theory was based on Keynes’s non numerical probabilities and weight of evidence(argument)analysis from the A Treatise on Probability, was correct. No mention at all is made about Frank Ramsey and subjective probability. No mention at all is made about the 1937 QJE article. No mention at all is made by either Townshend or Keynes about fundamental (irreducible, deep, radical, genuine, etc.) uncertainty. What is mentioned explicitly by Keynes to Townshend is the importance of Keynes’s comments on pages 148 and 240 of the General Theory. There is absolutely no support for any heterodox and/or Post Keynesian claim, based on a number of assessments made by Joan Robinson and G L S Shackle over their lifetimes, that the 1937 QJE article represents a major or fundamental change in Keynes’s view about uncertainty from those expressed in the A Treatise on Probability in 1921 and the General Theory in 1936.

Combining Probability Forecasts: 60% and 60% Is 60%, but Likely and Likely Is Very Likely

How do we combine others’ probability forecasts? Prior research has shown that when advisors provide numeric probability forecasts, people typically average them (i.e., they move closer to the average advisor’s forecast). However, what if the advisors say that an event is “likely” or “probable?” In eight studies (N = 7,334), we find that people are more likely to act as if they “count” verbal probabilities (i.e., they move closer to certainty than any individual advisor’s forecast) than they are to “count” numeric probabilities. For example, when the advisors both say an event is “likely,” participants will say that it is “very likely.” This effect occurs for both probabilities above and below 50%, for hypothetical scenarios and real events, and when presenting the others’ forecasts simultaneously or sequentially. We also show that this combination strategy carries over to subsequent consumer decisions that rely on advisors’ likelihood judgments. We discuss and rule out several candidate mechanisms for our effect.

Keynes’s Canonical Statements on Uncertainty Appear in His 1921 a Treatise on Probability in Chapter 26 on Pages 309 -312: Statements about Uncertainty in the General Theory in 1936 and the 1937 Quarterly Journal of Economics Article Are, at Best, Very Small Footnotes to Chapter 26 of the A Treatise on Probability

The Townshend–Keynes exchanges over decision making, weight of the argument (evidence), non numerical probabilities (Keynes’s term for Boole’s constituent probabilities, used in The Laws of Thought in 1854, that appears on page 163 of the A Treatise on Probability in chapter 15 on inexact measurement and approximation), and the connection between the A Treatise on Probability, the General Theory,and the 1937 Quarterly Journal of Economics, reveal that Keynes’s discussions about uncertainty in the General Theory, and the 1937 Quarterly Journal of Economics article are simply small ,minor footnotes to the A Treatise on Probability. Nowhere in the exchanges between Keynes and Townshend in 1937 and 1938 does either Keynes or Townshend mention or make any reference to the 1937 Quarterly Journal of Economics article, which Joan Robinson, G L S Shackle, Paul Davidson, and Post Keynesian and heterodox economists claim marked a major change in Keynes’s approach to decision making. Keynes, in his replies to Townshend, simply reinforced the strong connections that existed between his concepts of non numerical probability, weight,and his liquidity preference theory of the rate of interest on pages 148 and 240 of the General Theory. There is nothing in the 1937 Quarterly Journal of Economics article that is innovative or original that is not already discussed in far greater depth in chapter 26 of A Treatise on Probability. At best, the 1937 Quarterly Journal of Economics article is a minor footnote to pages 309-312 of chapter 26 of the A Treatise on Probability, which applies to all decision making. The 1937 QJE article emphasizes specifically economic decision making, but does not differ in any way from Keynes’s discussion in 1921. Heterodox economists have simply read into the 1937 QJE article Shackle’s contradictory concept of complete and total uncertainty, which heterodox economists call radical, fundamental, irreducible, genuine, or their latest term, deep, uncertainty. Shackle’s chaotic – kaledic view of a world of complete and total uncertainty was completely rejected by Keynes in his lifetime. Nowhere is this viewpoint even mentioned in the 1937-1938 Keynes-Townshend exchanges. Shackle’s chaotic – kaledic view of a world of complete and total uncertainty appears to be related to the great emphasis he places on the imagination of the decision maker, where decision makers are supposed to make things up based on their constantly changing dreams of the future.

Study on the Relationship between Worker States and Unsafe Behaviours in Coal Mine Accidents Based on a Bayesian Networks Model

Unsafe behaviours, such as violations of rules and procedures, are commonly identified as important causal factors in coal mine accidents. Meanwhile, a recurring conclusion of accident investigations is that worker states, such as mental fatigue, illness, physiological fatigue, etc., are important contributory factors to unsafe behaviour. In this article, we seek to provide a quantitative analysis on the relationship between the worker state and unsafe behaviours in coal mine accidents, based on a case study drawn from Chinese practice. Using Bayesian networks (BN), a graphical structure of the network was designed with the help of three experts from a coal mine safety bureau. In particular, we propose a verbal versus numerical fuzzy probability assessment method to elicit the conditional probability of the Bayesian network. The junction tree algorithm is further employed to accomplish this analysis. According to the BN established by expert knowledge, the results show that when the worker is in a poor state, the most vulnerable unsafe behaviour is violation, followed by decision-making error. Furthermore, insufficient experience may be the most significant contributory factor to unsafe behaviour, and poor fitness for duty may be the principal state that causes unsafe behaviours.

Do clinicians convey what they intend? Lay interpretation of verbal risk labels used in decision encounters

ObjectiveTo assess the numerical probabilities that individuals associate with frequently-used verbal labels relating to treatment outcomes and their association with medical context, age, gender, educational level, health literacy, and numeracy. MethodsVerbal labels (N = 11) were extracted from N = 90 audiotaped decision encounters in oncology. Three hundred Dutch adults, as proxies for newly-diagnosed cancer patients, assigned numerical probabilities to the labels in the context of cancer recurrence or nausea, and completed questions on their socio-demographic characteristics, health literacy and numeracy. ResultsWe found considerable variation in how individuals interpreted the verbal labels. Participants’ probability estimates of verbal labels was lower in the context of (the more serious) cancer recurrence compared to (less serious) nausea. Lower numerate participants differentiated less between labels. There was no association between participants’ estimates and age, gender, educational level or health literacy. ConclusionThere is considerable variation in how individuals interpret verbal labels frequently-used in decision encounters. Individuals seem to take base rates and severity of outcomes into account. Verbal labels may be less helpful to lower numerate individuals. Practice implicationsTo minimize misinterpretation and to improve patient-clinician decision making about health and care, we recommend to avoid the use of verbal labels only.

Assessing and Communicating Uncertainty Effectively in a Rapidly Changing World

Command and policy decision-making alike depend on timely and accurate intelligence. As Sherman Kent had noted in the 1950s and 60s, intelligence assessments are seldom cold, hard facts but rather judgments made by experts under conditions of uncertainty. Since Kent, history has taught us that misjudging and miscommunicating uncertainties threaten prospects for operational and strategic successes. Nevertheless, NATO and its members' intelligence communities have persisted in using inadequate methods for assessing and communicating uncertainty, which rely on vague linguistic probabilities (e.g., “likely” or “unlikely”). Here, drawing on recent scientific evidence including from NATO SAS-114 and other research, I describe the principal reasons why the intelligence community should change course and consider using numeric probabilities for estimates that support important decisions. This is arguably more important now than ever since changes in the global security environment, which augment the importance of non-munitions targeting, call for characterization of deep uncertainties related to second- and higher-order effects.

Keynes’s Logical Theory of Probability Is Concerned with Rational Degrees of Belief Based on Cognition, Which Are Not Mysterious or Mystical, Rank Ordered or Qualitative: F P Ramsey Failed to Grasp Keynes’s Use of Non–Additivity in His Reviews of the a Treatise on Probability

Recent exchanges between two academics on the Internet ,both experts in conventional and unconventional approaches to decision theory, reveal an astounding lack of knowledge and understanding regarding the makeup of Keynes’s logical theory of probability as put forth by Keynes in 1921, approximately 100 years ago. Given this lacuna concerning Keynes, even after nearly 100 years has passed by, there is thus a need to restate some basic fundamental facts about Keynes's approach that can only be attained by reading beyond Part I of the A Treatise on Probability or Chapter III of the A Treatise on Probability. Unfortunately, what academics mean when they say that they have read the A Treatise on Probability is that they have read Part I and/or Chapter III of A Treatise on Probability or Chapter III of the A Treatise on Probability. Parts II, III, IV and V are simply skipped or skimmed over, as was done by F P Ramsey in his reviews of 1922 and 1926. Keynes’s logical theory of probability is based on the upper–lower probability approach of George Boole. Boole is the explicit founder of the logical theory of probability that Keynes built on when writing the TP. Keynes called his approach inexact measurement and approximation. Keynes’s interval valued theory was designed, like Boole’s, to explicitly deal with non measurability, non comparability and incommensurability. Qualitative probability, comparative probability, ordinal probability, or rank ordered probability, just like numerical probability, can’t deal with non measurability, non comparability and incommensurability. The failure to absorb this basic point lies at the heart of all Post Keynesian and Fundamentalist Keynesian interpretations that all ignore George Boole’s fundamental contributions to Keynes’s logical theory of probability.

Accelerating Learning in Active Management: The Alpha-Brier Process

Forecasting tournaments have shown it is possible to improve the accuracy of forecasts of real-world events that sophisticated observers often supposed are too idiosyncratic to be pinned down by numerical probabilities. Building on this research program, the authors propose the Alpha-Brier process to help firms improve forecasts guiding investment decisions. Alpha-Brier offers four major benefits: (1) It reduces the noisiness of vague-verbiage forecasts, such as X “might” or “could” happen, that are open to varying interpretations; (2) it makes it easier to distinguish skill from luck as drivers of portfolio performance; (3) it gives investors faster feedback on the accuracy of their probability judgments; and (4) it lets managers test which methods of training, incentivizing, and aggregating judgments deliver the biggest boosts to accuracy. Paradoxically, the more pervasive the skepticism toward the core premise of Alpha-Brier that probability judgments can be extended to superficially unique events, the more opportunities there will be for firms to exploit market inefficiencies linked to overreliance on vague-verbiage forecasts. TOPICS:Statistical methods, portfolio construction, performance measurement

A Prediction Tournament Paradox

In a prediction tournament, contestants “forecast” by asserting a numerical probability for each of (say) 100 future real-world events. The scoring system is designed so that (regardless of the unknown true probabilities) more accurate forecasters will likely score better. This is true for one-on-one comparisons between contestants. But consider a realistic-size tournament with many contestants, with a range of accuracies. It may seem self-evident that the winner will likely be one of the most accurate forecasters. But, in the setting where the range extends to very accurate forecasters, simulations show this is mathematically false, within a somewhat plausible model. Even outside that setting the winner is less likely than intuition suggests to be one of the handful of best forecasters. Though implicit in recent technical papers, this paradox has apparently not been explicitly pointed out before, though is easily explained. It perhaps has implications for the ongoing IARPA-sponsored research programs involving forecasting.

Variants of Vague Verbiage: Intelligence Community Methods for Communicating Probability

Debates over uncertainty communication have long pervaded the intelligence discourse and became acute following major intelligence failures; namely, the 9/11 attacks and Iraq WMD fiasco. To address the goal of mitigating subjectivity and the potential for miscommunication, some intelligence organizations have developed standardized lexicons for communicating estimative probability. However, as is the case with standards developed for other facets of uncertainty communication in intelligence (e.g., information credibility, source reliability, analytic confidence), these standards are rarely grounded in empirical research, and they may in fact undermine rather than improve communication fidelity. One overriding reason for our pessimistic outlook is that, for the most part, the set of current standards represents various versions of vague verbiage: all current intelligence standards for communicating estimative probabilities commit to using verbal probabilities (words such as “likely” or phrases like “realistic possibility”) and they shun the use of numerical probabilities either as precise estimates (e.g., 73% chance) or imprecise estimates (e.g., a 60%-80% chance). In this chapter, we present an annotated collection of the estimative probability standards gathered by members and affiliates of NATO’s SAS-114 Research Task Group on Assessment and Communication of Uncertainty in Intelligence to Support Decision-Making. These include standards used in intelligence production, as well as in other domains such as defence and security risk management and climate science. After reviewing this non-exhaustive collection of standards, we discuss common problematic features and how they might compromise efforts to support decision making.

How Uncertainty Influences Lay People's Attitudes and Risk Perceptions Concerning Predictive Genetic Testing and Risk Communication.

The interpretation of genetic information in clinical settings raises moral issues about adequate risk communication and individual responsibility about one’s health behavior. However, it is not well-known what role numeric probabilities and/or the conception of disease and genetics play in the lay understanding of predictive genetic diagnostics. This is an important question because lay understanding of genetic risk information might have particular implications for self-responsibility of the patients.Aim: Analysis of lay attitudes and risk perceptions of German lay people on genetic testing with a special focus on how they deal with the numerical information.Methods: We conducted and analyzed seven focus group discussions (FG) with lay people (n = 43).Results: Our participants showed a positive attitude toward predictive genetic testing. We identified four main topics: (1) Anumeric risk instead of statistical information; (2) Treatment options as a factor for risk evaluation; (3) Epistemic and aleatory uncertainty as moral criticism; (4) Ambivalence as a sign of uncertainty.Conclusion: For lay people, risk information, including the statistical numeric part, is perceived as highly normatively charged, often as an emotionally significant threat. It seems necessary to provide lay people with a deeper understanding of risk information and of the limitations of genetic knowledge with respect to one’s own health responsibility.

How Keynes Solved the ‘Mystery’ of the Diagram on Page 39 (Page 42 of the 1973 CWJMK Edition) of the A Treatise on Probability in Part II in Chapter 15 on pp.161–163 Just As He Had Foretold on pp. 37–38 of Chapter III

On pp.37-38 of the A Treatise on Probability ,Keynes was very clear that he was only going to present a 'brief' analysis of non numerical probability graphically as an illustration. However, in Part II, he stated he would present a 'detailed' analysis. This detailed mathematical and logical analysis involved the application of his modified approach to inexact measurement using the method of approximation that had first been used by Boole in 1854. Keynes demonstrated and presented a simple, illustrated version of his interval valued approach in Chapter III of the A Treatise on Probability. In Chapter 15, he provided the detailed mathematical application of his logical theory of probability. Keynes followed through on his promise to demonstrate a purely mathematical treatment of non numerical probabilities.The real mystery is the complete and total failure of the Post Keynesians and Keynesian Fundamentalists to grasp Keynes’s completely clear analysis 98 YEARS after the publication of the A Treatise on Probability in 1921. So Keynes actually did precisely what he said he would do. The only real remaining mystery is why no philosopher, mathematician, or economist in the 20th or 21st centuries ever read chapters 15, 16, and 17 of the A Treatise on Probability. It has been 98 years since Keynes published his book in 1921. The time has come for serious scholars to move beyond Keynes’s introduction to measurement in chapter III of the A Treatise on Probability, using diagrams, to Keynes’s full scale mathematical and logical application of his inexact measurement approach that he called approximation based on Boole’s upper and lower, interval valued, estimate approach to probability and decision making.

Given Keynes's Definitions of Logical Probability and Evidential Weight, It Is Impossible for Keynes’s Approach to Measurement to Be an Ordinal Theory; His 'Non Numerical' Probabilities Must Be Based on Inexact and Imprecise Measurement Using Approximation Involving Boolean Upper and Lower Probability Bounds and Bounded Outcomes

Keynes spent all of his introductory Chapter One of the A Treatise on Probability emphasizing that his logical relation of probability (a relation of similarity – dissimilarity based on analogy) P(a/h)=α, where α came in degrees and 0≤α≤1. In chapter 6, Keynes introduced his logical relation of the evidential weight of the argument, V(a/h), but did not specify what it equaled, although he clearly stated that he would integrate both probability and weight together later. In chapter 26, Keynes specified that V(a/h)= w, where we came in degrees and 0≤w≤1. Keynes showed how to integrate both probability and weight together in his conventional coefficient of weight and risk, c, in chapter 26 of the A Treatise on Probability. Probability and weight were also integrated together by Keynes using the theory of inexact measurement or approximation based on intervals with upper and lower bounds or limits. Ordinal probability can do none of these things. Advocates of the claim that Keynes’s theory was an ordinal theory of probability concentrate exclusively on a misinterpreted diagram that appears near the end of chapter 3 of the A Treatise on Probability. Thus, while Keynes’s theory can easily deal with ordinal probability with the aid of the Principle of Indifference, it is not part of Keynes’s main or major analysis in the A Treatise on Probability, which was interval probability. The same conclusion holds for numerical probability and rational expectations. Rational expectations are a very special case that occurs if w=1. The heterodox, institutionalist, or Post Keynesian approaches, involving irreducible, fundamental, or radical uncertainty, are a very, very special case that occurs only if w=0 in the very long run. Their argument, that measurement is not possible at all, be it exact or inexact, is rejected by Keynes in chapter 4 of the General Theory. Thus, while Keynes’s theory can easily deal with ordinal probability, it is impossible for the ordinal probability to deal with inexact and imprecise interval-valued probability measurement. Keynes’s theory of measurement involves interval-valued probability in the A Treatise on Probability with the numerical and ordinal probability being very special cases of his general theory of interval-valued probability. Keynes’s approach to measurement in the General Theory also involved inexact and imprecise measurement as deployed by Keynes in chapter 4 of the General Theory. No ordinal probability approach to measurement is used by Keynes in the General Theory. The great value of the 1936-1938 Keynes-Townshend exchanges is that they show that Keynes presented an approach to measurement that was imprecise and inexact in both the A Treatise on Probability and the General Theory.

On Discrete Probability Approximations for Transaction Cost Problems

In this paper we consider a discrete-time formulation of dynamic transaction cost problems. We examine applicability of numerical discrete probability approximation as an alternative simplistic approach to solve dynamic transaction cost problems. We provide a computational study of a lattice-based heuristic method on simple transaction cost models and highlight its many advantages. The solution of these problems provides a dynamic investor with important insights as to how the portfolio should be re-balanced when faced with transaction costs.

Improving information evaluation for intelligence production

ABSTRACT National security decision-making is informed by intelligence assessments, which in turn depend on sound information evaluation. We critically examine information evaluation methods, arguing that they mask rather than effectively guide subjectivity in intelligence assessment. Drawing on the guidance metaphor, we propose that rigid ‘all-purpose’ information evaluation methods be replaced by flexible ‘context-sensitive’ guidelines aimed at improving the soundness, precision, accuracy and clarity of irreducibly subjective judgments. Specific guidelines, supported by empirical evidence, include use of numeric probability estimates to quantify the judged likelihood of information accuracy, promoting collector-analyst collaboration and periodic revaluation of information as new information is acquired.