Reference Class Problem Research Articles

Searching for the General Science of Evidence: Venn on Probability and Induction

In this paper Venn's account of probability inference and induction is examined, tracing their differences as well as how they ‘co-operate’ in inferences from particulars to particulars. We discuss the role of mathematical idealizations in making probability inferences, the celebrated rule of succession and we delve into the nature of the reference class problem arguing that for Venn it is a common problem for both induction and inference in probability. Our approach is both historical and philosophical attempting to sketch Venn's position both in the philosophy of probability and induction of his time and in relation to the twentieth-century frequentism.

The Reference Class Problem and Probabilities in the Individual Case: A Response to Fuller

AbstractIn a recent article on the interpretation of probability in evidence-based medical practice, Jonathan Fuller argues that we should interpret probabilities as credences in individual cases because this avoids some important problems. In this article, I argue that Fuller misidentifies the real issue and so fails to offer a meaningful solution to it. The real problem with making probability judgments in individual cases is deciding which objective considerations ought to constrain our formation of credences. This leads us to the reference class problem, which, as Alan Hajek argues, is a problem for any interpretation of probability.

The deep neural network approach to the reference class problem

Methods of machine learning (ML) are gradually complementing and sometimes even replacing methods of classical statistics in science. This raises the question whether ML faces the same methodological problems as classical statistics. This paper sheds light on this question by investigating a long-standing challenge to classical statistics: the reference class problem (RCP). It arises whenever statistical evidence is applied to an individual object, since the individual belongs to several reference classes and evidence might vary across them. Thus, the problem consists in choosing a suitable reference class for the individual. I argue that deep neural networks (DNNs) are able to overcome specific instantiations of the RCP. Whereas the criteria of narrowness, reliability, and homogeneity, that have been proposed to determine a suitable reference class, pose an inextricable tradeoff to classical statistics, DNNs are able to satisfy them in some situations. On the one hand, they can exploit the high dimensionality in big-data settings. I argue that this corresponds to the criteria of narrowness and reliability. On the other hand, ML research indicates that DNNs are generally not susceptible to overfitting. I argue that this property is related to a particular form of homogeneity. Taking both aspects together reveals that there are specific settings in which DNNs can overcome the RCP.

The Specific Evidence Rule. Reference Classes – Individuals – Personal Autonomy

This paper grapples with the issue of naked statistical evidence in general and the reference class problem (RCP) in particular. By analysing the reasoning patterns underlying the RCP, I will show, first, that the RCP rests on theoretical presuppositions which we are by no means willing to accept. Such a presupposition is the wholesale approach in decision-making. Secondly, I will show that the very effort to increase the level of precision to a maximum so that a reference class contains a single member only is theoretically inconsistent insofar, as it deprives reference classes of their general (and thus scientific) character. Thereupon, I will argue, thirdly, that the decision to enact a specific evidence rule is a political one and reflects deep moral and jurisprudential values, not scientific propositions. Such a value is personal autonomy, which I go on to illuminate briefly. Whether the trier of fact will treat cases in a wholesale approach or not depends on constitutional arrangements and legal values putting emphasis on the individual and the latter’s dignity.

The Reference Class Problem for Credit Valuation in Science

Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theoretic models of credit seeking in science. The reference class problem for credit valuation in science is the problem of determining to which of an agent’s communities—which reference class—credit determinations should be indexed for an act under some state of nature. Solving this problem requires developing rich, mutually informed theories of community and credit that are sensitive to the structure and status systems of complex, heterogeneous scholarly networks.

Generalizations and Reference Classes

Legal scholarship exploring the nature of evidence and the process of juridical proof has had a complex relationship with formal modeling. An example of this complex relationship concerns attempts by scholars to use mathematical models to quantify the probative value of evidence in order to study decision-making from that perspective. As with earlier attempts to apply probability theory to juridical proof, this scholarship is interesting, instructive, and insightful. However, it also suffers from a deep conceptual problem that makes ambiguous the lessons that can be drawn from it — the problem of reference classes. In this chapter, we examine the implications of the reference-class problem for attempts to model the probative value of items of evidence. This chapter makes three contributions. First, and most importantly, it is a further demonstration of the problematic relationship between algorithmic tools and aspects of legal decision-making. Second, it points out serious pitfalls to be avoided for analytical or empirical studies of juridical proof. Third, it indicates when algorithmic tools may be more or less useful in the evidentiary process. At the highest level of generality, this chapter is another demonstration of the very complex set of relationships involving human knowledge and rationality, on the one hand, and attempts to reduce either to a set of formal concepts, on the other.

Objective consequentialism and the plurality of chances

I claim that objective consequentialism (OC) faces a problem stemming from the existence in some situations of a plurality of chances relevant to the outcomes of an agent’s acts. I suggest that this phenomenon bears structural resemblance to the well-known Reference Class problem. I outline a few ways in which one could attempt to deal with the issue, suggesting that it is the higher-level chance that should be employed by OC.

On Luck and Modality

The modal account of luck is the predominant account of luck in epistemology and ethics. In the first half of this paper, I discuss three possible interpretations of the modal account (proportional, distance, and density-based views) and raise objections to each. I then raise an objection to all plausible versions of the modal account, that is, that whether an event is lucky or the extent to which it is a matter of luck will depend on what initial conditions or features of the event one holds constant across nearby possible worlds. However, there is often reasonable disagreement about what the relevant initial conditions of an event are, and the modal account of luck has no means of determining which description of the event is correct. As such, the modal account is subject to a kind of reference class problem, and the view cannot actually tell us the extent to which certain events are a matter of luck.

A Formal Solution to Reichenbach's Reference Class Problem

Following Reichenbach, it is widely held that in making a direct inference, one should base one’s conclusion on a relevant frequency statement concerning the most specific reference class for which one is able to make a warranted and relatively precise-valued frequency judgment. In cases where one has accurate and precise-valued frequency information for two relevant reference classes, R1 and R2, and one lacks accurate and precise-valued frequency information concerning their intersection, R1 intersect R2, it is widely held, following Reichenbach, that no inference may be drawn. In contradiction to Reichenbach and the common wisdom, I argue for the view that it is often possible to draw a reasonable informative conclusion, in such circumstances. As a basis for drawing such a conclusion, I show that one is generally in a position to formulate a reasonable direct inference for a reference class that is more specific than either of R1 and R2.

Natural selection and the reference grain problem

Over the last 20 years, the concept of natural selection has been highly debated in the philosophy of biology. Yet, most discussions on this topic have focused on the questions of whether natural selection is a causal process and whether it can be distinguished from drift. In this paper, I identify another sort of problem with respect to natural selection. I show that, in so far as a classical definition of fitness includes the transmission of a type between generations as part of the definition, it seems difficult to see how the fitness of an entity, following this definition, could be description independent. In fact, I show that by including type transmission as part of the definition of fitness, changing the grain at which the type of an entity is described can change the fitness of that entity. If fitness is not grain-of-description independent, this further propagates to the process of natural selection itself. I call this problem the ‘reference grain problem’. I show that it can be linked to the reference class problem in probability theory. I tentatively propose two solutions to it.

Induction by Direct Inference Meets the Goodman Problem

Abstract I here aim to show that a particular approach to the problem of induction, which I will call "induction by direct inference", comfortably handles Goodman's problem of induction. I begin the article by describing induction by direct inference. After introducing induction by direct inference, I briefly introduce the Goodman problem, and explain why it is, prima facie, an obstacle to the proposed approach. I then show how one may address the Goodman problem, assuming one adopts induction by direct inference as an approach to the problem of induction. In particular, I show that a relatively standard treatment of what some have called the \Reference Class Problem" addresses the Goodman Problem. Indeed, plausible and relatively standard principles of direct inference yield the conclusion that the Goodman inference (involving the grue predicate) is defeated, so it is unnecessary to invoke considerations of `projectibility' in order to address the Goodman problem. I conclude the article by discussing the generality of the proposed approach, in dealing with variants of Goodman's example.

Resolving the observer reference class problem in cosmology

The assumption that we are typical observers plays a core role in attempts to make multiverse theories empirically testable. A widely shared worry about this assumption is that it suffers from systematic ambiguity concerning the reference class of observers with respect to which typicality is assumed. As a way out, Srednicki and Hartle recommend that we empirically test typicality with respect to different candidate reference classes in analogy to how we test physical theories. Unfortunately, as this paper argues, this idea fails because typicality is not the kind of assumption that can be subjected to empirical tests. As an alternative, a background information constraint on observer reference class choice is suggested according to which the observer reference class should be chosen such that it includes precisely those observers who one could possibly be, given one's assumed background information.

Selection in a Complex World: Deriving Causality from Stable Equilibrium

It is an ongoing controversy whether natural selection is a cause of population change, or a mere statistical description of how individual births and deaths accumulate. In this paper I restate the problem in terms of the reference class problem, and propose how the structure of stable equilibrium can provide a solution in continuity with biological practice. Insofar natural selection can be understood as a tendency towards equilibrium, key statisticalist criticisms are avoided. Further, in a modification of the Newtonian-force analogy, it can be suggested that a better metaphor for natural selection is that of an emergent force, similar in nature to entropic forces: with magnitude and direction, but lacking a spatiotemporal origin or point of application.

Using group data to treat individuals: understanding heterogeneous treatment effects in the age of precision medicine and patient-centred evidence.

Although often conflated, determining the best treatment for an individual (the task of a doctor) is fundamentally different from determining the average effect of treatment in a population (the purpose of a trial). In this paper, we review concepts of heterogeneity of treatment effects (HTE) essential in providing the evidence base for precision medicine and patient-centred care, and explore some inherent limitations of using group data (e.g. from a randomized trial) to guide treatment decisions for individuals. We distinguish between person-level HTE (i.e. that individuals experience different effects from a treatment) and group-level HTE (i.e. that subgroups have different average treatment effects), and discuss the reference class problem, engendered by the large number of potentially informative subgroupings of a study population (each of which may lead to applying a different estimated effect to the same patient), and the scale dependence of group-level HTE. We also review the limitations of conventional 'one-variable-at-a-time' subgroup analyses and discuss the potential benefits of using more comprehensive subgrouping schemes that incorporate information on multiple variables, such as those based on predicted outcome risk. Understanding the conceptual underpinnings of HTE is critical for understanding how studies can be designed, analysed, and interpreted to better inform individualized clinical decisions.

Continued misinterpretation of confidence intervals: response to Miller and Ulrich.

Miller and Ulrich (2015) critique our claim (Hoekstra et al., Psychonomic Bulletin & Review, 21(5), 1157–1164, 2014), based on a survey given to researchers and students, of widespread misunderstanding of confidence intervals (CIs). They suggest that survey respondents may have interpreted the statements in the survey that we deemed incorrect in an idiosyncratic, but correct, way, thus calling into question the conclusion that the results indicate that respondents could not properly interpret CIs. Their alternative interpretations, while correct, cannot be deemed acceptable renderings of the questions in the survey due to the well-known reference class problem. Moreover, there is no support in the data for their contention that participants may have had their alternative interpretations in mind. Finally, their alternative interpretations are merely trivial restatements of the definition of a confidence interval, and have no implications for the location of a parameter.

Disease-mongering through clinical trials

Our goal in this paper is to articulate a precise concept of at least a certain kind of disease-mongering, showing how pharmaceutical marketing can commercially exploit certain diseases when their best definition is given through the success of a treatment in a clinical trial. We distinguish two types of disease-mongering according to the way they exploit the definition of the trial population for marketing purposes. We argue that behind these two forms of disease-mongering there are two well-known problems in the statistical methodology of clinical trials (the reference class problem and the distinction between statistical and clinical significance). Overcoming them is far from simple.

Identifying economic crises: insights from history

Economists have been blamed for their inability to forecast and address crises. This article attributes this inability to intertwined factors: the lack of a coherent definition of crises, the reference-class problem, the lack of imagination regarding the nature of future crises and sample-selection biases. Specifically, economists tend to adapt their views on crises to recent episodes, and omit averted and potential crises. Threshold-based definitions of crises run the risk of being ad hoc. Using historical examples, this article highlights some epistemological shortcomings of the current approach.

Structuring inferential reasoning in criminal fact finding: an analogical theory

The article proposes a normative theory of inferential reasoning for criminal fact finding, centred on the concept of ‘analogy’. While evidence law scholars have devoted little attention to the topic, the article maintains that analogy deserves more consideration. In particular, it argues that an analogical theory of inferential reasoning has three main advantages. First, the theory makes it possible to incorporate within a single coherent framework the important insights of different approaches to ‘reasoning under uncertainty’; indeed, it welcomes both the Pascalian notion of ‘relevance’ based on the Bayesian likelihood ratio and the Baconian concept of ‘weight’. Secondly, it helps advance the conventional understanding of the reference class problem, an evidential conundrum widely discussed in the recent legal scholarship. Finally, the theory allows for a functional taxonomy of reasonable doubts.

Individual Risk

J Clin Hypertens (Greenwich). 2012;14:261–264. ©2012 Wiley Periodicals, Inc.Patients don’t have an “individual risk” or unique probability of an outcome. Outside Mendelian inheritance, risks are conditional probabilities and differ as the risk factors included differ, at times substantially. This lack of reliability is an inherent limitation and is not resolved by including additional risk factors. Groups of like individuals need to be assembled to measure the probability of an outcome. Many groups, like any individual, can be identified, eg, groups of the same age, sex, race, or any combination of these attributes (or any others). That each of these groups may have different risk means there is no such thing as individual risk. This issue was identified by John Venn in 1866 and is known as the reference class problem. Models relate risk factors to outcomes in populations. The number calculated for an individual should not be reported as their individual or true risk, nor should it be used as the sole criterion for clinical decisions. Instead, Feinstein proposed relying on clinically important subgroups. An example would be utilizing an individual’s blood pressure as the primary determinant of hypertension treatment decisions, not an unreliable individual risk estimate.

A Third Route to the Doomsday Argument

In this paper, I present a solution to the Doomsday argument based on a third type of solution, by contrast to on the one hand, the Carter-Leslie view and on the other hand, the Eckhardt et al. analysis. The present line of thought is based on the fact that both aforementioned analyses are based on an inaccurate analogy. After discussing the imperfections of both models, I present then a two-sided model that fits more adequately with the human situation corresponding to DA and encapsulates both Carter-Leslie's and Eckhardt et al.'s models. I argue then that this new analogy also holds when one takes into account the issue of indeterminism and the reference class problem. This leads finally to a novel formulation of the argument that could well be more consensual than the original one.