Comments by J.Q. Smith on Goldstein and Rougier

Abstract

The authors are to be commended for a very imaginative paper. They bravely attempt to formalise and then propose solutions to the very real problem of how to capture the scientist's beliefs about the discrepancy between simulator outputs and reality. Their approach is laudably subjectivist and they have not been overly distracted, as many authors have been in the recent past in trying to obtain a spurious objectivity in their analysis. They also rightly criticise the appropriateness of the conditional independence assumption they and previous authors havemade in the past to address this issue. They convincingly demonstrate that this assumption is untenable both from a practical and a theoretical perspective. Although there is much I like about this paper I hope I will be forgiven for concentratingmy comments on the difficulties I havewith their proposed revisedmethodology. I begin with a philosophical point. For me, the demands of coherence—forcefully made throughout the paper—only make sense when there is someone (or group of people) who owns the probability model: i.e. someone who believes what is expressed through that model. My first question is therefore who owns the probability statements being made? In particular, who owns the conditional independence statements at the core of this methodology? Who is their “modeller”, the “our” who owns the distribution: the statistician or the domain expert(s), or someone else? This is obviously a critical question in the interpretation and plausibility of their model, on the input of expert judgement and its calibration. Henceforth I assume that by “we” the authors intend a synonym to be “Michael and Jonty” and the priors they use in their model represent what they believe, having elicited information from domain experts. Thus Michael and Jonty believe in a world where predictions of a reality y can be fully expressed as the output of a single simulator f ∗(x,w) where (x,w) is an input vector of all possible measurements that they could in principle collect and f ∗ a complicated function encapsulating the whole of relevant knowledge and theory that they could in principle absorb and code. My second question: How do they even think of events concerning {f ∗(x,w), (x,w)}? Do they list all the possible inputs they could collect, all the possible differential equations they could in principle use? Specifically, how do they satisfy De Finetti's clarity principle about {f ∗(x,w), (x,w)} (especially w) that allows them to even write down statements like (8)? Or are they abandoning the De Finetti principles that previsions (and hence conditional independence statements) should only be specified on observables? Incidentally, I do not myself believe that the totality of scientific knowledge about something is expressible within a single coherent model. The best I believe I can do is to express some of my currently integrated knowledge (not all that Imight learn or even all I might loosely grasp) within a coherent framework so that it is unambiguous and can be scrutinised and criticised by others. I therefore find the authors' initial premise implausible. Letus assume that the conditional independence statement (8) is actually formallydefensible.My thirdquestion is:Whyshould they believe the conditional independence statement (8) any more than (1)—the one they have so convincingly demonstrated as fundamentally flawed? The argument they give us in the paper is that this is credible “due to the high accuracy of f ∗”—see Section 3.1. But formally the existenceof a conditional independence relation (8) cannotbe invoked just because f ∗(x∗,w∗) could,withhigh probability, be close to y. Obviously in this case ∗ could still be highly dependent on {f , f ∗, x∗,w∗}. Their conditional independence cannot be deduced through arguments of accuracy. In my opinion this “pragmatic” assumption is no better supported than (1). However, let us predict that the authors will in the future present a proper argument for why we might expect that (8) is at least a plausible approximate working hypothesis. There then appears to be a very interesting link here with work on causal Bayesian Networks: see e.g. Spirtes et al. (1993), Pearl (2000, 2003), and Dawid (2002). As here implicitly the world is assumed to be run by a complicated simulator: but now decomposable into a network of component simulators. Within this (albeit usually non-dynamic) simulator network it is typically assumed that some outputs (and hence inputs) of the component simulators are missing (like w in this paper). The issue is to make predictions about the effect on y after the system has been manipulated.