Expert Judgment Elicitation Research Articles

Forecasting adversarial actions using judgment decomposition-recomposition

In domains such as homeland security, cybersecurity, and competitive marketing, it is frequently the case that analysts need to forecast actions by other intelligent agents that impact the problem of interest. Standard structured expert judgment elicitation techniques may fall short in this type of problem as they do not explicitly take into account intentionality. We present a decomposition technique based on adversarial risk analysis followed by a behavioural recomposition using discrete choice models that facilitate such elicitation process and illustrate its reasonable performance through behavioural experiments.

Aso volcano, Japan: assessing the 100-year probability of a new caldera-forming eruption based on expert judgements with Bayes Net and Importance Sampling uncertainty analysis

The Aso-4 explosive eruption on Kyushu, Japan, 89,500 years ago was one of the biggest eruptions in the last one hundred millennia, with a magnitude of approximately M8. Modern society requires the likelihood of natural events with potentially disastrous consequences to be evaluated, even if probabilities of occurrence are diminishingly small. For some situations, it is not satisfactory to assert an event scenario probability is “negligible” or can be “ignored”. Judicial hearings or litigation may require risk levels to be quantified, in which case, statements of scientific confidence could be decisive. Internationally, e.g., for nuclear site safety evaluations, event likelihoods on order of 10–7/year are often considered for quantitative assessment. At such hazard levels, this might include evaluating the proposition that a particular volcano can deliver a future super-eruption, a supposition that could be attached to Aso volcano. But, simplistically taking the average recurrence interval between past caldera-forming eruptions at a given volcano is an unreliable guide to the likelihood of a future repeat: each past event represented a unique set of tectonic and magmatic conditions within a continually evolving volcanic system. Such processes are not temporally stationary nor statistically uniform. To evaluate the probability of a new M8 event at Aso, within the next 100 years, we performed a comprehensive stochastic probability uncertainty analysis using a model implemented with advanced computational Bayes Net (BN) software. Our eruption process model is informed by multiple strands of evidence from volcanology, petrology, geochemistry and geophysics, together with estimates of epistemic (knowledge) uncertainty, adduced from reviews of published data, modelling and from expert judgement elicitation. Several lines of evidence characterise the likely structure, magmatic composition and eruptive state of the present-day Aso volcano, which has had numerous smaller eruptions since Aso-4. To calculate the probability of another M8 eruption of Aso, we implemented probabilistic ‘Importance Sampling’ in our model. With this approach, we find the chance of an Aso-4 scale eruption (characterised by mean volume 500 km3 DRE and approximate 90% credible interval [210 ‥ 1200] km3 DRE) is less than 1–in–1 billion in the next 100 years (i.e., < 10–9 probability). Based on current volcanological understanding and evidence, we believe this probability estimate is robust to within an order of magnitude.

Structured Expert Judgment Elicitation in Conceptual Aircraft Design

Disruptive technologies and novel aircraft generations represent a potential approach to address the ambitious emission reduction goals in aviation. However, the introduction of innovative concepts is a time-consuming process, which might not necessarily yield an optimal design for a given flight mission and within the defined time frame. In order to address the need for a structured and more exhaustive search for novel concept generations, the Advanced Morphological Approach (AMA) and its further enhancement was introduced earlier. It implies the decomposition of design problems into functional attributes and appropriate technological alternatives. Subsequently, these are evaluated and combined into solutions, which are then projected onto a solution space. The current paper focuses on the technology evaluation step by deriving and integrating structured expert judgment elicitation (SEJE) techniques into conceptual aircraft design with the AMA. For this purpose, the first aim of the work is to justify the developed method by giving an overview and discussing the most prominent SEJE methods and their applications in aerospace. Then, the derived SEJE concept is described and applied in the form of an expert workshop on the use case of wing morphing architecture. As a result, a solution space of wing morphing architecture configurations is generated and analyzed. The workshop conduction and the expert feedback serve as valuable findings for both the further AMA enhancement and similar research.

Graded Approach Establishment for the HTGR Maintenance Activities Using Modified Fuzzy FMEA & Expert Judgement Methodology

Grading is an important step of the Nuclear Power Plant (NPP) operation & maintenance activities. However, there are several grading difficulties for the High Temperature Gas Reactor (HTGR) as well as the other type of NPPs causing by the lack of operational experiences and availability of the reliability data. Failure Mode & Effects Analysis (FMEA) is one of the mature techniques that are commonly used to solve such kinds of difficulties. Nevertheless, traditional FMEA has several issues and possibly become an obstacle in the grading process. The modified FMEA by utilizing expert judgment elicitation techniques combined with the fuzzy logic theory is proposed to solve those issues. As a study practice, the proposed methodology is applied by examining Japanese’s HTGR, Gas Turbine High Temperature Reactor 300 for Cogeneration (GTHTR300C) design carefully. This study establishing good practice especially for the future advanced NPP maintenance activities development.

Assessing the validity of navigation risk assessments: A study of offshore wind farms in the UK

The developments of offshore wind farms can place increased pressures on conflicting marine users, particularly in already crowded waterways. Risk analysis of potential hazard scenarios are conducted by developers and regulators in the form of Navigation Risk Assessments which seek to identify, measure and mitigate impacts through data collection, consultation, modelling and risk assessment. These activities have inherent uncertainties and limitations which are rarely discussed and have the potential to undermine the value and credibility of the risk assessment. To evaluate the accuracy of Navigation Risk Assessments, their predictions are compared with the historical incident record of accidents involving wind farms. This review identifies significant methodological limitations and sources of uncertainty endemic in the Navigation Risk Assessment process which results in an over-estimation of risk. These include a lack of inclusion of historical evidence, issues during elicitation of expert judgment and methodological limitations of both quantitative risk models and the underlying risk assessment. Based on our evaluation, future research directions are highlighted to support decision makers on marine spatial planning by increasing the robustness of Navigation Risk Assessments.

One Coin, Two Sides: Eliciting Expert Knowledge From Training Participants in a Capacity-Building Program for Veterinary Professionals.

Scientific research may include the elicitation of judgment from non-academic subject-matter experts in order to improve the quality and/or impact of research studies. Elicitation of expert knowledge or judgment is used when data are missing, incomplete, or not representative for the specific setting and processes being studied. Rigorous methods are crucial to ensure robust study results, and yet the quality of the elicitation can be affected by a number of practical constraints, including the understanding that subject-matter experts have of the elicitation process itself. In this paper, we present a case of expert elicitation embedded within an extended training course for veterinary professionals as an example of overcoming these constraints. The coupling of the two activities enabled extended opportunities for training and a relationship of mutual respect to be the foundation for the elicitation process. In addition, the participatory research activities reinforced knowledge synthesis objectives of the educational program. Finally, the synergy between the two concurrent objectives may produce benefits which transcend either independent activity: solutions and ideas built by local professionals, evolving collaborative research and training approaches, and a network of diverse academic and practicing professionals. This approach has the versatility to be adapted to many training and research opportunities.

A structured expert judgement elicitation approach: how can it inform sound intervention decision-making to support household food security?

To examine structured expert judgement (SEJ) elicitation as a method to provide robust, defensible data for three determinants of household food security (food cost, household disposable income and physical access) for quantifying a proof-of-concept integrating decision support system for food security. SEJ elicitation is a validated method for obtaining unavailable data, but its use in household food security in high-income countries is novel. Investigate Discuss Estimate Aggregate (IDEA) elicitation protocol was implemented, including quantitative and qualitative elements. Using specific questions related to three determinants, food security experts were encouraged to Investigate - estimate individual first-round responses to these questions, Discuss - with each other evidence on the reasoning and logic of their estimates, Estimate - second-round responses, following which these judgements were combined using mathematical Aggregation. Victoria, Australia. Five experts with a range of expertise in the area of household food insecurity participated in the SEJ elicitation process. The experts' ability to provide reliable estimates was tested and informed the aggregation of the collection of individual estimates into a single quantity of interest for use in decision support. The results of the quantitative elicitation show the impact of combinations of varying household income, food cost and physical access on household food security status and severity and is supported by the experts reasoning during elicitation. This research provides insight to the application of SEJ where elicited data can inform and support intervention decision-making specific to household food security, especially where evidence is absent or of poor quality.

Elicitation of US and Chinese expert judgments show consistent views on solar geoengineering

Expert judgments on solar geoengineering (SG) inform policy decisions and influence public opinions. We performed face-to-face interviews using formal expert elicitation methods with 13 US and 13 Chinese climate experts randomly selected from IPCC authors or supplemented by snowball sampling. We compare their judgments on climate change, SG research, governance, and deployment. In contrast to existing literature that often stress factors that might differentiate China from western democracies on SG, we found few significant differences between quantitative judgments of US and Chinese experts. US and Chinese experts differed on topics, such as desired climate scenario and the preferred venue for international regulation of SG, providing some insight into divergent judgments that might shape future negotiations about SG policy. We also gathered closed-form survey results from 19 experts with >10 publications on SG. Both expert groups supported greatly increased research, recommending SG research funding of ~5% on average (10th–90th percentile range was 1–10%) of climate science budgets compared to actual budgets of <0.3% in 2018. Climate experts chose far less SG deployment in future climate policies than did SG experts.

A decision support system for liability in civil litigation: a case study from an insurance company

The use of statistical and AI methods in civil litigation is an area likely to expand. As with many areas of social science, the data requirements are high but complex, because of the complexity of the legal process and the nature of the causal connections. This paper looks at the early stage of the process where the initial establishment of liability acts as a legal triage which affects the route through the litigation process. A simple model is used in which the training set is the assessment of the probability of liability given hypothetical scenarios in road traffic accidents. The model is augmented by additional “weight of evidence” assessments. The model, once built, is used as a decision support system for claim handlers on a routine basis. The methods can be seen as a way of utilising a special type of expert judgment elicitation.

Explaining successful and failed investments in U.S. carbon capture and storage using empirical and expert assessments

Most studies of deep decarbonization find that a diverse portfolio of low-carbon energy technologies will be required, including carbon capture and storage (CCS) that mitigates emissions from fossil fuel power plants and industrial sources. While many projects essential to commercializing the technology have been proposed, most (>80%) end in failure. Here we analyze the full universe of CCS projects attempted in the U.S. that have sufficient documentation (N=39)—the largest sample ever studied systematically. We quantify 12 project attributes that the literature has identified as possible determinants of project outcome. In addition to costs and technological readiness, which prior research has emphasized, we develop metrics for attributes that are widely thought to be important yet have eluded systematic measurement, such as the credibility of project revenues and policy incentives, and the role of regulatory complexity and public opposition. We build three models—two statistical and one derived through the elicitation of expert judgment—to evaluate the relative influence of these 12 attributes in explaining project outcome. Across models, we find the credibility of revenues and incentives to be among the most important attributes, along with capital cost and technological readiness. We therefore develop and elicit experts’ judgment of 14 types of policy incentives that could alter these attributes and improve the prospects for investment in CCS. Knowing which attributes have been most responsible for past successes and failures allows developers to avoid past mistakes and identify clusters of near-term CCS projects that are more likely to succeed.

Anticipated impacts of Brexit scenarios on UK food prices and implications for policies on poverty and health: a structured expert judgement approach

IntroductionFood insecurity is associated with increased risk for several health conditions and with poor chronic disease management. Key determinants for household food insecurity are income and food costs. Whereas short-term...

Dynamic probabilistic analysis of accidents in construction projects by combining precursor data and expert judgments

Construction accident occurrences are essentially rare, stochastic, and dynamic. This study proposes a method for accident prediction that fully captures these natures based on historical data and prior knowledge. The method utilizes the relatively high occurrence frequency of precursor events and the dependency between precursors and accidents. The modeling approach consists of three steps: (1) characterize the stochastic occurrences of precursor events over time based on precursor data; (2) estimate the failure rate of the Poisson model which is assumed to be a prior distribution of accident occurrences; and (3) elicit the expert knowledge about the stochastic dependency between near miss occurrences and accident occurrences. A copula-based Markov model is used to develop the time series model of precursors while a copula-based protocol is proposed to aid expert judgment elicitation and quantification. The probability of accident occurrence is then dynamically updated according to the observed historical near miss numbers. The proposed method is applied to a metro construction project. A five-year long near miss data were collected and used as accident precursor data, while three experts were invited to provide relevant information. The developed accident model is used to predict the accident-prone periods, which are consistent with the months that the observed near miss occurrence frequency deviates significantly from normality. Thus, the model can be used to support the planning of necessary safety improvement programs before the accident risk increased.

Designing Monitoring Programs for Marine Protected Areas Within an Evidence Based Decision Making Paradigm

The Evidence Based Decision Making (EBDM) paradigm encourages managers to base their decisions on the strongest available evidence, but it has been criticized for placing too much emphasis on the choice of study design method without considering the types of questions that are being addressed as well as other relevant factors such as how well a study is implemented. Here we review the objectives of Australia’s Marine Park network, and identify the types of questions and data analysis that would address these objectives. Critically, we consider how the design of a monitoring program influences our ability to adequately answer these questions, using the strength of evidence hierarchy from the EBDM paradigm to assess the adequacy of different design strategies and other sources of information. It is important for conservation managers to recognize that the types of questions monitoring programs are able to answer depends on how they are designed and how the collected data are analyzed. The socio-political process that dictates where protected areas are placed typically excludes the strongest types of evidence, Random Controlled Trials (RCTs), for certain questions. Evidence bases that are stronger than ones commonly employed to date, however, could be used to provide a causal inference, including for those questions where RCTs are excluded, but only if appropriate designs such as cohort or case-control studies are used, and supported where relevant by appropriate sample frames. Randomized, spatially balanced sampling, together with careful selection of control sites, and more extensive use of propensity scores and structured elicitation of expert judgment, are also practical ways to improve the evidence base for answering the questions that underlie marine park objectives and motivate long-term monitoring programs.

An assessment of the risk, cost-effectiveness, and perceived benefits of anti-parvovirus B19 tested blood products.

Parvovirus B19 (B19V) can cause severe anemia, hydrops foetalis, and even death in vulnerable patients. To prevent transfusion-transmitted B19V infection of at-risk patients, B19V antibody screening of blood donors was implemented. The cost-effectiveness of this intervention is unclear, as the likelihood of transmission through blood and subsequent complications for recipients are unknown. This study estimates the cost-effectiveness of anti-B19V donor screening in the Netherlands. The estimates needed for the cost-effectiveness model were: the occurrence of B19V in Dutch blood donors, the number of anti-B19V tested products required by hospitals, the likelihood of morbidity and mortality given B19V infection, treatment costs, and screening costs. These estimates were obtained from literature and observational data. When data were unavailable, structured expert judgment elicitation and statistical modeling were applied. The costs of preventing one transfusion transmitted B19V infection are estimated at €68,942 (€42,045 - €102,080). On average, 1.25 cases of morbidity and 0.12 cases of mortality are prevented annually. Although the perceived risk of transfusion transmitted B19V infection was low, half of the treating physicians favored anti-B19V screening. The estimated mortality and morbidity caused by B19V infection was low in the risk groups. The cost-effectiveness ratio is similar to other blood safety screening measures. No guidance exists to evaluate the acceptability of this ratio. The explicit overview of costs and effects may further guide the discussion of the desirability of B19V safe blood products.

Assessment of safety barriers for the prevention of cascading events in oil and gas offshore installations operating in harsh environment

Fire in the topside production area of offshore Oil and Gas (O&G) platforms has the potential to trigger cascading events and may lead to catastrophic consequences for operators, environment and asset. Safety barriers, such as hardware systems and emergency response, are critical elements aimed at preventing the propagation of this type of scenarios. Harsh environmental conditions may strongly affect the integrity of protective equipment and significantly hinder an effective and timely emergency response, with a consequent increase in the risk level. The present study is aimed at developing a structured methodology to the quantitative performance assessment of safety barriers, specifically addressing the analysis of offshore facilities operating in harsh and sensitive environment. An expert judgment elicitation procedure is adopted to score external factors associated with harsh environmental conditions. Simplified relationships are established for availability and effectiveness evaluation of safety barriers in offshore O&G platforms. A dedicated framework is developed for the analysis of emergency response. The approach is tested through the application to a case study, aimed at the assessment of the influence of harsh environmental conditions on the risk due to cascading events in a reference offshore installation.

Group elicitations yield more consistent, yet more uncertain experts in understanding risks to ecosystem services in New Zealand bays.

The elicitation of expert judgment is an important tool for assessment of risks and impacts in environmental management contexts, and especially important as decision-makers face novel challenges where prior empirical research is lacking or insufficient. Evidence-driven elicitation approaches typically involve techniques to derive more accurate probability distributions under fairly specific contexts. Experts are, however, prone to overconfidence in their judgements. Group elicitations with diverse experts can reduce expert overconfidence by allowing cross-examination and reassessment of prior judgements, but groups are also prone to uncritical “groupthink” errors. When the problem context is underspecified the probability that experts commit groupthink errors may increase. This study addresses how structured workshops affect expert variability among and certainty within responses in a New Zealand case study. We find that experts’ risk estimates before and after a workshop differ, and that group elicitations provided greater consistency of estimates, yet also greater uncertainty among experts, when addressing prominent impacts to four different ecosystem services in coastal New Zealand. After group workshops, experts provided more consistent ranking of risks and more consistent best estimates of impact through increased clarity in terminology and dampening of extreme positions, yet probability distributions for impacts widened. The results from this case study suggest that group elicitations have favorable consequences for the quality and uncertainty of risk judgments within and across experts, making group elicitation techniques invaluable tools in contexts of limited data.

A comparative social life cycle assessment of urban domestic water reuse alternatives

The social benefits and impacts of four alternative approaches to urban domestic non-potable water reuse were compared: (1) central wastewater treatment, no urban reuse. Reclaimed water is discharged to nature; (2) central wastewater treatment and urban reuse of the wastewater treatment plant’s tertiary effluent; (3) semi-distributed greywater treatment and reuse, at cluster scale (8 residential buildings); (4) distributed greywater treatment and reuse, within each apartment building. The impacts of the four aforementioned approaches to water reuse on three relevant stakeholders were investigated: public, community, and consumer. A hierarchical structure of impact categories and sub-categories under these stakeholder groups was established. Expert judgment elicitation was used to attribute weights to the social criteria, through an analytical hierarchy process (AHP). AHP is an established multi-criteria decision analysis method, based on series of pairwise comparisons. It was also used to evaluate impact intensities for both quantitative and qualitative social indicators. All expert judgements were integrated into an overall weight vector, and a comprehensive social score was calculated for each compared scenario. Public commitment to water saving was ranked as the most important factor in assessing the social impacts of urban domestic reuse, with a weight of 29.6%. Two sub-categories of the community category ranked second: urban landscape and community engagement (12.6% and 12.0%, respectively). The two distributed alternatives are advantageous over the other two approaches in terms of water saving and urban landscape. The semi-distributed alternative has a significant benefit of community engagement, which the other three lack. The business-as-usual (BAU, no reuse) scenario scored highest in the categories: public equality, consumer health concerns, and consumption habits. Final scores for the compared scenarios indicate that central reuse is somewhat more socially beneficial than no urban reuse, but the two distributed alternatives are far better. In social life cycle assessment (SLCA) often quantitative and qualitative criteria/indicators exist side by side and their harmonious integration is challenging. The challenge arising is twofold: quantifying qualitative indicators and homogenizing all indicator evaluations into a uniform comparable scale. Both tasks were achieved in this study using the AHP. The AHP also successfully served as a platform for participatory processes of eliciting expert judgments regarding criteria weights and may be very useful for stakeholder participation in any social assessment. Regarding the case study—distributed urban water reuse was found to be socially beneficial, both in terms of promoting public commitment to conservation of natural water resources and in advancing community engagement.

Assessing transition probability of bridge deterioration using Dempster–Shafer theory of evidence

Reliable deterioration models for forecasting the condition of bridge elements, systems, and networks are essential components of any bridge management system. Estimating deterioration transition probabilities is the basis of the popular Markov chain deterioration model. Transition probabilities ha ve been produced in the literature by statistical and probabilistic analysis performed on available bridge inspection and condition data. The available data may be limited or inconsistent with the Markov chain transition period. In addition, the conventional probability theory has limitations when applied to problems with a stochastic nature such as bridge deterioration. This paper introduces a novel method based on the theory of evidence for bridge deterioration modeling through expert judgment elicitation. The advantages of the theory of evidence over the traditional probability theory are discussed and the process for the theory implementation is demonstrated with a case study to validate the application of Dempster–Shafer theory of evidence to estimate the transition probabilities. Based on the results, the theory of evidence is proposed as a scientific expert judgement elicitation technique in the area of bridge condition rating and deterioration modeling. Expert judgment elicitation and theory of evidence application hold potential in the field of bridge management and require further investigation and research.

Characterization of Precipitation through Copulas and Expert Judgement for Risk Assessment of Infrastructure

In this paper two methodologies are investigated that contribute to better assessment of risks related to extreme rainfall events. Firstly, one-parameter bivariate copulas are used to analyze rain gauge data in the Netherlands. Out of three models considered, the Gumbel copula, which indicates upper tail dependence, represents the data most accurately for all 33 stations in the Netherlands. Seasonal variability is noticeable, with rank correlation reaching maximum in winter and minimum in summer as well as other temporal and spatial patterns. Secondly, an expert judgment elicitation was undertaken. The experts’ opinions were combined using Cooke’s classical method in order to obtain estimates of future changes in precipitation patterns. Experts predicted mostly an approximate 10% increase in rain amount, duration, intensity and the dependence between amount and duration. The results were in line with official national climate change scenarios, based on numerical modelling. Applicability of both methods was presented based on an example of an existing tunnel in the Netherlands, contributing to better estimates of the tunnel’s limit state function and therefore the probability of failure.

Predicting in shock: on the impact of negative, extreme, rare, and short lived events on judgmental forecasts

The occurrence of unexpected events that are extreme in magnitude, rare in frequency, and short-lived in duration poses distinctive challenges to decision makers and planners. In this paper we examine the impact of negative versions of these events, which we term “shocks”, on the judgmental forecasts of subjects experiencing them. A behavioral experiment asking participants to forecast monthly time series in the presence of temporary but extreme decreases in those series is used. Average changes to annual prediction intervals and 1-month ahead forecasts were much smaller than the magnitude of the shock and occurred in proportion to the size of the shock. Changes to prediction intervals were more persistent for moderate than large shocks, and larger for shocks occurring a second time. Our results provide supporting evidence for the view that decision makers underweight rare and extreme events rather than overweight them, consistent with a discounting or forgetting effect. The behavioral findings are relevant to operations researchers involved in expert judgment elicitation and in supporting decision making.