Comparative Forecast Evaluation: Graphical Gaussian Models and Sufficiency Relations

Abstract

This paper deals with the comparative evaluation of categorical forecasts supposing that forecasts and observations are continuous variables and have a jointly normal distribution. An information content approach based on the well-established covariance fitting technique of graphical Gaussian modeling is proposed to evaluate the possibly correlated random errors in competing forecasts. Suppose that two alternative forecasting systems deliver forecasts, say fa and fb, for a scalar variable θ. Two questions are relevant when using these forecasts: 1) Is one forecasting system definitely better than the other? 2) Knowing the forecasts of the better system, can additional information be obtained from also consulting the second system? The main part of this paper addresses the second question. If, for instance, the forecasts fb are redundant given the value of fa, the forecasts fa are sufficient for the pair of forecasts (fa, fb). The appropriate statistical concept to describe this situation is conditional independence of fb and θ given fa. Pairwise conditional independences in a dataset can conveniently be displayed in a graph by a lack of direct connection between nodes representing the corresponding variables. For multivariate normal data missing links in the graph are characterized by zero elements of the inverse variance–covariance matrix. This study applies a known maximum likelihood technique of fitting graphical models to data in order to specify the amount of incremental information in fb. A prototypical example is elaborated that indicates a potential of graphical modeling for evaluating the dependence structure in a set of multisite forecasts. Several studies have examined a different sufficiency concept to identify which of two given forecasting systems is unambiguously more useful to any user. The forecasts fa are termed sufficient for the forecasts fb if the statistical properties of fb can be simulated by additionally randomizing the forecasts fa. Assuming joint normal distributions of forecasts and corresponding observations, this randomization translates into a simple reduction of explained variance. In this study the relation between fa being sufficient for fb and it being sufficient for the pair (fa, fb) is elucidated.