Linear Opinion Pool Research Articles

Measuring the Output Gap in Switzerland with Linear Opinion Pools

We use the recently proposed linear opinion pool methodology of Garratt et al. (2014) to construct real-time ensemble nowcast densities of the output gap for Switzerland over an out-of-sample period from 2003:Q1 to 2015:Q4. The model space consists of a large number of bivariate VAR specifications for inflation and the output gap, with each specification using a different estimate of the output gap, lag order in the VAR, and structural break information. The ensemble nowcast densities for the output gap are constructed by combining the predictive densities of the individual VAR specifications, weighted by their ability to provide accurate density forecasts for inflation. The overall performance of the linear opinion pool is assessed by its real-time output gap nowcasts and by the size of the ex post revisions to the output gap nowcasts. We find that the linear opinion pool does not produce any more accurate density or point forecasts of inflation than a number of simple univariate benchmark models that condition on the same structural break information. Further, the linear opinion pool’s real-time estimate of the output gap is no more robust to ex post re visions than the real-time estimates of the individual univariate output gaps. The fact that Swiss GDP price deflator data are subject to large revisions, complicates the measurement and forecasting of inflation.

Terms-based discriminative information space for robust text classification

With the popularity of Web 2.0, there has been a phenomenal increase in the utility of text classification in applications like document filtering and sentiment categorization. Many of these applications demand that the classification method be efficient and robust, yet produce accurate categorizations by using the terms in the documents only. In this paper, we propose a novel and efficient method using terms-based discriminative information space for robust text classification. Terms in the documents are assigned weights according to the discrimination information they provide for one category over the others. These weights also serve to partition the terms into category sets. A linear opinion pool is adopted for combining the discrimination information provided by each set of terms to yield a feature space (discriminative information space) having dimensions equal to the number of classes. Subsequently, a discriminant function is learned to categorize the documents in the feature space. This classification methodology relies upon corpus information only, and is robust to distribution shifts and noise. We develop theoretical parallels of our methodology with generative, discriminative, and hybrid classifiers. We evaluate our methodology extensively with five different discriminative term weighting schemes on six data sets from different application areas. We give a side-by-side comparison with four well-known text classification techniques. The results show that our methodology consistently outperforms the rest, especially when there is a distribution shift from training to test sets. Moreover, our methodology is simple and effective for different application domains and training set sizes. It is also fast with a small and tunable memory footprint.

Reliability updating in linear opinion pooling for multiple decision makers

Accurate information sources are vital prerequisites for good decision making. Here, we consider a multiple participant setting, where all decision makers have a collection of neighbours with whom they share their beliefs about some common relevant uncertain quantity. When determining which course of action to follow, a decision maker takes into account all the information received from her neighbours. Over time, in light of the returns observed from choices made, decision makers update their own beliefs over the uncertain event and also adjust the degree of consideration they afford to the opinions of each neighbour based on the level of reliability their information is ascertained to have. In this article, we derive and discuss a methodology, the plug-in approach, by which this can be done, and discuss some suitable justification for this. We include an illustrative risk-based example which further demonstrates the merits of the method, before concluding with a discussion on further work and potential alternatives.

Prediction of categorical spatial data via Bayesian updating

ABSTRACTThis study introduces a transition probability-based Bayesian updating (BU) approach for spatial classification through expert system. Transition probabilities are interpreted as expert opinions for updating the prior marginal probabilities of categorical response variables. The main objective of this paper is to provide a spatial categorical variable prediction method which has a solid theoretical foundation and yields relatively higher classification accuracy compared with conventional ones. The basic idea is to first build a linear Bayesian updating (LBU) model that corresponds to an application of Bayes’ theorem. Since the linear opinion pool is intrinsically suboptimal and underconfident, the beta-transformed Bayesian updating (BBU) model is proposed to overcome this limitation. Another type of BU approach, conditional independent Bayesian updating (CIBU), is derived based on conditional independent experts. It is shown that traditional Markovian-type categorical prediction (MCP) is equivalent to a particular CIBU model with specific parameters. As three variants of the BU method, these techniques are illustrated in synthetic and real-world case studies, comparison results with both the LBU and MCP favor the BBU model.

A Framework for Multi-Model Ensembling

As reliance on computer modeling increases, scientists and engineers often produce ensembles of model runs from multiple models. Combining such multi-model ensembles while managing uncertainties is a challenging problem. We embed the problem into a Bayesian framework. Specifically, a Bayesian is to update prior beliefs based on information regarding the probability distributions from a collection of experts. These experts may be individuals, research centers, computer modeling teams, etc. In this article we use one well-known approach to combining probability distributions, the so-called linear opinion pool, to provide a basis for suggesting approaches for processing multi-model ensembles. First, had the experts' probability distributions been available, we consider treating their combination as if it is a likelihood for the quantities of interest and combine it with our prior distribution by formal application of Bayes' theorem to produce our posterior distribution. Second, we will act as if the ensemble...

Assessing uncertainty in sighting records: an example of the Barbary lion.

As species become rare and approach extinction, purported sightings can be controversial, especially when scarce management resources are at stake. We consider the probability that each individual sighting of a series is valid. Obtaining these probabilities requires a strict framework to ensure that they are as accurately representative as possible. We used a process, which has proven to provide accurate estimates from a group of experts, to obtain probabilities for the validation of 32 sightings of the Barbary lion. We consider the scenario where experts are simply asked whether a sighting was valid, as well as asking them to score the sighting based on distinguishablity, observer competence, and verifiability. We find that asking experts to provide scores for these three aspects resulted in each sighting being considered more individually, meaning that this new questioning method provides very different estimated probabilities that a sighting is valid, which greatly affects the outcome from an extinction model. We consider linear opinion pooling and logarithm opinion pooling to combine the three scores, and also to combine opinions on each sighting. We find the two methods produce similar outcomes, allowing the user to focus on chosen features of each method, such as satisfying the marginalisation property or being externally Bayesian.

Calibration, sharpness and the weighting of experts in a linear opinion pool

Linear opinion pools are the most common form of aggregating the probabilistic judgments of multiple experts. Here, the performance of such an aggregation is examined in terms of the calibration and sharpness of the component judgments. The performance is measured through the average quadratic score of the aggregate. Trade-offs between calibration and sharpness are examined and an expression for the optimal weighting of two dependent experts in a linear combination is given. Circumstances where one expert would be disqualified are investigated. Optimal weights for the multiple, dependent experts are found through a concave quadratic program.

Modelling Uncertainty: An Application to Forecasting International Reserves

Being able to accurately predict international reserves is an extremely important task for pegged exchange rate economies. Barbados’ 1992 experience with vanishing reserves has left its mark on the economy, being brought to mind most recently when reserve levels began to slide sharply in 2013. However, forecasting remains an approximate science and it has become increasingly important to recognize and capture the uncertainties involved in model-based forecasting. This paper attempts to capture parameter uncertainty, model selection uncertainty, and future uncertainty by employing a combination of regression ensembles, randomized model selection, linear opinion pooling and equal or RMSE weights. The importance of density forecasting in predicting international reserves is displayed by using the 2013 unexpected and unexplained decline in net international reserves as the out-of-sample test. This paper agrees with the proposal that density forecasting should be employed to provide policy makers with greater information describing the various uncertainties in economic forecasting. Notably, being able to understand the uncertainty of forecasting international reserves would have provided policymakers with a head start in dealing with declining reserves in 2013.

A Multiple Training Image Approach for Spatial Modeling of Geologic Domains

Characterization of complex geological features and patterns remains one of the most challenging tasks in geostatistics. Multiple point statistics (MPS) simulation offers an alternative to accomplish this aim by going beyond classical two-point statistics. Reproduction of features in the final realizations is achieved by borrowing high-order spatial statistics from a training image. Most MPS algorithms use one training image at a time chosen by the geomodeler. This paper proposes the use of multiple training images simultaneously for spatial modeling through a scheme of data integration for conditional probabilities known as a linear opinion pool. The training images (TIs) are based on the available information and not on conceptual geological models; one image comes from modeling the categories by a deterministic approach and another comes from the application of conventional sequential indicator simulation. The first is too continuous and the second too random. The mixing of TIs requires weights for each of them. A methodology for calibrating the weights based on the available drillholes is proposed. A measure of multipoint entropy along the drillholes is matched by the combination of the two TIs. The proposed methodology reproduces geologic features from both TIs with the correct amount of continuity and variability. There is no need for a conceptual training image from another modeling technique; the data-driven TIs permit a robust inference of spatial structure from reasonably spaced drillhole data.

A logarithmic opinion pool based STAPLE algorithm for the fusion of segmentations with associated reliability weights.

Pelvic floor dysfunction is common in women after childbirth and precise segmentation of magnetic resonance images (MRI) of the pelvic floor may facilitate diagnosis and treatment of patients. However, because of the complexity of its structures, manual segmentation of the pelvic floor is challenging and suffers from high inter and intra-rater variability of expert raters. Multiple template fusion algorithms are promising segmentation techniques for these types of applications, but they have been limited by imperfections in the alignment of templates to the target, and by template segmentation errors. A number of algorithms sought to improve segmentation performance by combining image intensities and template labels as two independent sources of information, carrying out fusion through local intensity weighted voting schemes. This class of approach is a form of linear opinion pooling, and achieves unsatisfactory performance for this application. We hypothesized that better decision fusion could be achieved by assessing the contribution of each template in comparison to a reference standard segmentation of the target image and developed a novel segmentation algorithm to enable automatic segmentation of MRI of the female pelvic floor. The algorithm achieves high performance by estimating and compensating for both imperfect registration of the templates to the target image and template segmentation inaccuracies. A local image similarity measure is used to infer a local reliability weight, which contributes to the fusion through a novel logarithmic opinion pooling. We evaluated our new algorithm in comparison to nine state-of-the-art segmentation methods and demonstrated our algorithm achieves the highest performance.

Belief Elicitation to Populate Health Economic Models of Medical Diagnostic Devices in Development

Bayesian methods can be used to elicit experts' beliefs about the clinical value of healthcare technologies. This study investigates a belief-elicitation method for estimating diagnostic performance in an early stage of development of photoacoustic mammography (PAM) imaging versus magnetic resonance imaging (MRI) for detecting breast cancer. Eighteen experienced radiologists ranked tumor characteristics regarding their importance to detect malignancies. With reference to MRI, radiologists estimated the true positives and negatives of PAM using the variable interval method. An overall probability density function was determined using linear opinion pooling, weighted for individual experts' experience. The most important tumor characteristics are mass margins and mass shape. Respondents considered MRI the better technology to visualize these characteristics. Belief elicitation confirmed this by providing an overall sensitivity of PAM ranging from 58.9 to 85.1% (mode 75.6%) and specificity ranging from 52.2 to 77.6% (mode 66.5%). Belief elicitation allowed estimates to be obtained for the expected diagnostic performance of PAM, although radiologists expressed difficulties in doing so. Heterogeneity within and between experts reflects this uncertainty and the infancy of PAM. Further clinical trials are required to validate the extent to which this belief-elicitation method is predictive for observed test performance.

Assessing the risk of Nipah virus establishment in Australian flying-foxes.

Nipah virus (NiV) is a recently emerged zoonotic virus that causes severe disease in humans. The reservoir hosts for NiV, bats of the genus Pteropus (known as flying-foxes) are found across the Asia-Pacific including Australia. While NiV has not been detected in Australia, evidence for NiV infection has been found in flying-foxes in some of Australia's closest neighbours. A qualitative risk assessment was undertaken to assess the risk of NiV establishing in Australian flying-foxes through flying-fox movements from nearby regions. Events surrounding the emergence of new diseases are typically uncertain and in this study an expert opinion workshop was used to address gaps in knowledge. Given the difficulties in combining expert opinion, five different combination methods were analysed to assess their influence on the risk outcome. Under the baseline scenario where the median was used to combine opinions, the risk was estimated to be very low. However, this risk increased when the mean and linear opinion pooling combination methods were used. This assessment highlights the effects that different methods for combining expert opinion have on final risk estimates and the caution needed when interpreting these outcomes given the high degree of uncertainty in expert opinion. This work has provided a flexible model framework for assessing the risk of NiV establishment in Australian flying-foxes through bat movements which can be updated when new data become available.

Measuring output gap nowcast uncertainty

We propose a methodology for gauging the uncertainty in output gap nowcasts across a large number of commonly-deployed vector autoregressive (VAR) specifications for inflation and the output gap. Our approach utilises many output gap measures to construct ensemble nowcasts for inflation using a linear opinion pool. The predictive densities for the latent output gap utilise weights based on the ability of each specification to provide accurate probabilistic forecasts of inflation. In an application based on US real-time data, nowcasting over the out-of-sample evaluation period from 1991q2 to 2010q1, we demonstrate that a system of bivariate VARs produces well-calibrated ensemble densities for inflation, in contrast to univariate autoregressive benchmarks. The implied nowcast densities for the output gap are multimodal and indicate a considerable degree of uncertainty. For example, we assess the probability of a negative output gap at around 45% between 2004 and 2007. Despite the Greenspan policy regime, there still remained a substantial risk that the nowcast for output was below potential in real time. We extend our methodology to include distinct output gap measures, based on alternative filters, and show that, in our application, the nowcast density for the output gap is sensitive to the detrending method.

Forecast densities for economic aggregates from disaggregate ensembles

AbstractWe extend the “bottom up” approach for forecasting economic aggregates with disaggregates to probability forecasting. Our methodology utilises a linear opinion pool to combine the forecast densities from many disaggregate forecasting specifications, using weights based on the continuous ranked probability score. We also adopt a post-processing step prior to forecast combination. These methods are adapted from the meteorology literature. In our application, we use our approach to forecast US Personal Consumption Expenditure inflation from 1990q1 to 2009q4. Our ensemble combining the evidence from 16 disaggregate PCE series outperforms an integrated moving average specification for aggregate inflation in terms of density forecasting.

The effect of the number of seed variables on the performance of Cooke′s classical model

In risk analysis, Cooke′s classical model for aggregating expert judgment has been widely used for over 20 years. However, the validity of this model has been the subject of much debate. Critics assert that this model′s scoring rule may unintentionally reward experts who manipulate their quantile estimates in order to receive a greater weight. In addition, the question of the number of seed variables required to ensure adequate performance of Cooke′s classical model remains unanswered. In this study, we conduct a comprehensive examination of the model through an iterative, cross validation test to perform an out-of-sample comparison between Cooke′s classical model and the equal-weight linear opinion pool method on almost all of the expert judgment studies compiled by Cooke and colleagues to date. Our results indicate that Cooke′s classical model significantly outperforms equally weighting expert judgment, regardless of the number of seed variables used; however, there may, in fact, be a maximum number of seed variables beyond which Cooke′s model cannot outperform an equally-weighted panel.

An expert elicitation of climate, energy and economic uncertainties

Critical energy policy decisions rely on expert assessments of key future uncertainties. But existing modelling techniques that help form these expert assessments often ignore the existence of uncertainty. Consequently, techniques to measure these uncertainties are of increasing importance. We use one technique, expert elicitation, to assess six key uncertain parameters with 25 UK energy experts across academia, government and industry. We obtain qualitative descriptions of the uncertain parameters and a novel data set of probability distributions describing individual expert beliefs. We conduct a sensitivity analysis on weights for a linear opinion pool and show that aggregated median beliefs in 2030 are: for oil price $120/barrel (90% CI: 51, 272); for greenhouse gas price $34/tCO2e (90% CI: 5, 256) and for levelised cost of low-carbon electricity 17.1 US cents/kWh (90% CI: 8.3, 31.0). The quantitative results could inform model validation, help benchmark policy makers’ beliefs or provide probabilistic inputs to models.

Universal algebra for general aggregation theory: Many-valued propositional-attitude aggregators as MV-homomorphisms

This article continues Dietrich and List's (2010) work on propositional-attitude aggregation theory, which is a generalized unification of the judgement aggregation and probabilistic opinion-pooling literatures. We first propose an algebraic framework for an analysis of (many-valued) propositional-attitude aggregation problems. Then we shall show that systematic propositional-attitude aggregators can be viewed as homomorphisms—algebraically structure-preserving maps—in the category of Chang's (1958a, Trans. Am. Math. Soc., 88, 467–490) MV-algebras. (Proof idea: Systematic aggregators are induced by maps satisfying certain functional equations, which in turn can be verified to entail homomorphy identities.) Since the 2-element Boolean algebra as well as the real unit interval can be endowed with an MV-algebra structure, we obtain as natural corollaries two famous theorems: Arrow's theorem for judgement aggregation as well as McConway's (1981, J. Am. Stat. Assoc., 76, 410–414) characterization of linear opinion pools. Conceptually, this characterization of aggregators can be seen as justifying a certain structuralist interpretation of social choice. Technically and perhaps more importantly, it opens up a new methodology to social-choice theorists: the analysis of general aggregation problems by means of universal algebra.

Trimmed Opinion Pools and the Crowd's Calibration Problem

We introduce an alternative to the popular linear opinion pool for combining individual probability forecasts. One of the well-known problems with the linear opinion pool is that it can be poorly calibrated. It tends toward underconfidence as the crowd's diversity increases, i.e., as the variance in the individuals' means increases. To address this calibration problem, we propose the exterior-trimmed opinion pool. To form this pool, forecasts with low and high means, or cumulative distribution function (cdf) values, are trimmed away from a linear opinion pool. Exterior trimming decreases the pool's variance and improves its calibration. A linear opinion pool, however, will remain overconfident when individuals are overconfident and not very diverse. For these situations, we suggest trimming away forecasts with moderate means or cdf values. This interior trimming increases variance and reduces overconfidence. Using probability forecast data from U.S. and European Surveys of Professional Forecasters, we present empirical evidence that trimmed opinion pools can outperform the linear opinion pool.This paper was accepted by Rakesh Sarin, decision analysis.

Trimmed Opinion Pools and the Crowd's Calibration Problem

We introduce an alternative to the popular linear opinion pool for combining individual probability forecasts. One of the well-known problems with the linear opinion pool is that it can be poorly calibrated. It tends toward underconfidence as the crowd’s diversity increases, i.e., as the variance in the individuals’ means increases. To address this calibration problem, we propose the exterior-trimmed opinion pool. To form this pool, forecasts with low and high means, or cumulative distribution function (cdf) values, are trimmed away from a linear opinion pool. Exterior-trimming decreases the pool’s variance and improves its calibration. A linear opinion pool, however, will remain overconfident when individuals are overconfident and not very diverse. For these situations, we suggest trimming away forecasts with moderate means, or cdf values. This interior-trimming increases variance and reduces overconfidence. Using probability forecast data from US and European Surveys of Professional Forecasters, we present empirical evidence that trimmed opinion pools can outperform the linear opinion pool.

PMD4 Expert Elicitation to Populate Early Health Economic Models of Medical Diagnostic Devices in Development

Purpose: During the development of new diagnostic and therapeutic devices, it is desirable to indicate the cost-effectiveness through modeling and to establish its potential clinical value to guide further developments. However, in these early stages of development, there are usually no or sparse clinical data available. In this study, expert elicitation was used as a method to estimate uncertain priors of the diagnostic performance of a new imaging device, i.e. Photoacoustic Mammography (PAM). We compared PAM as an alternative to Magnetic Resonance Imaging (MRI) as a second line diagnostic in the detection of breast cancer. Method: Expert elicitation was used as a method to formulate the knowledge and beliefs of experts regarding the future performance of PAM and to quantify this information into probability distributions. 18 experienced radiologists (specialized, in examining MR-images of breasts) were asked to estimate the importance of different tumor characteristics in the examination of images of breasts. Following this, the performance of visualizing these characteristics were estimated for both MRI and PAM. Using the mathematical approach to elicitation, the radiologists estimated the true positive rate (TPR) and true negative rate (TNR) based on existing MRI data (with a TPR of 263 out of 292, and a TNR of 214 out of 308) and specified the mode (the most likely value), the lower, and the upper boundaries (a 95% credible interval). An overall probability density function (PDF) was determined using the linear opinion pooling method in which weighting is applied to reflect the performance of individual experts. Result: The elicited judgments show that the most important characteristics in the discrimination between benign and malign tissue are mass margins (30.44%) and mass shape (28.6%). The oxygen saturation (2.49%) and mechanical properties (9.48%) were less important as there is limited information available about the added value of these characteristics. The performance of MRI on visualizing mass margins and mass shape was estimated to be higher than PAM, where PAM scored higher in the performance of displaying oxygen saturation and mechanical properties. An overall score of MRI (82.28) and PAM (54.03) indicates that MRI performs best in visualizing lesions of the breast. From the expert elicitation process an overall sensitivity was estimated ranging from 58.9% to 85.1%, with a mode of 75.6%. The specificity ranged from 52.2% to 77.6%, with a mode of 66.5%. Radiologists expressed difficulties making the estimations, as they felt there was insufficient data about the manner in which PAM visualizes different tumor types. Conclusion: The examination of tumor characteristics indicates that PAM is inferior over MRI. However, if oxygen saturation and mechanical properties are more important in the examination of images of breasts, this results in higher performance of PAM. Using expert elicitation in the absence of clinical data, prior distributions of the range of sensitivity and specificity can be obtained. Theoretically, this data can be fed into early health economic models. There were, however, difficulties expressed by experts in estimating the performance of PAM, given the limited existing evidence and clinical experience. The expression of uncertainty surrounding their beliefs should reflect the infancy of the diagnostic method, however further clinical trials should be commissioned to indicate whether these results are valid. Before that, the use of the elicited priors in health economic models requires careful consideration.