Use Of Expert Judgment Research Articles

An allision risk model for passing vessels and offshore oil and gas installations on the Norwegian Continental Shelf

This article presents a new risk model for estimating the probability of allision risk (the impact between a ship under way and a stationary installation) from passing vessels on the Norwegian Continental Shelf (NCS). Offshore petroleum operators on the NCS are required by the Norwegian Petroleum Safety Authority (PSA) to perform risk assessments to estimate the probability of impacts between ships and offshore installations, both for field related and passing (merchant) vessels. This has typically been done using the aging industry standard COLLIDE risk model, but this article presents a new risk model based on a Bayesian Belief Network (BBN) that can replace the old COLLIDE model for passing vessels. The new risk model incorporates a wider range of risk influencing factors (RIFs) and enables a holistic and detailed analysis of risk factors, barrier elements and dependencies. Even though the risk of allision with passing vessels is very small, the potential consequences can be critical. The new risk model is more transparent and provides a better understanding of the mechanisms behind allision risk calculations. The results from the new model are aligned with industry expectations, indicating an overall satisfactory performance. The article discusses several key elements, such as the use of expert judgement to estimate RIFs when no empirical data is available, model sensitivity, and a comparative assessment of the new risk model to the old COLLIDE model.

A critical analysis of multi-criteria models for the prioritisation of health threats

Multi-criteria assessments are increasingly being employed in the prioritisation of health threats, supporting decision processes related to health risk management. The use of multi-criteria analysis in this context is welcome, as it facilitates the consideration of multiple impacts of health threats, it can encompass the use of expert judgment to complement and amalgamate the evidence available, and it permits the modelling of policy makers’ priorities. However, these assessments often lack a clear multi-criteria conceptual framework, in terms of both axiomatic rigour and adequate procedures for preference modelling. Such assessments are ad hoc from a multi-criteria decision analysis perspective, despite the strong health expertise used in constructing these models. In this paper we critically examine some key assumptions and modelling choices made in these assessments, comparing them with the best practices of multi-attribute value analysis. Furthermore, we suggest a set of guidelines on how simulation studies might be employed to assess the impact of these modelling choices. We apply these guidelines to two relevant studies available in the health threat prioritisation domain. We identify severe variability in our simulations due to poor modelling choices, which could cause changes in the ranking of threats being assessed and thus lead to alternative policy recommendations than those suggested in their reports. Our results confirm the importance of carefully designing multi-criteria evaluation models for the prioritisation of health threats.

Expert Elicitation to Inform a Cost-Effectiveness Analysis of Screening for Renal Cancer

BackgroundPopulation screening for renal cell carcinoma (RCC) using ultrasound has the potential to improve survival outcomes; however, a cost-effectiveness analysis (CEA) has yet to be performed. Owing to the lack of existing evidence, we performed structured expert elicitation to derive unknown quantities to inform the CEA. ObjectiveTo elicit the cancer stage distribution (proportion of individuals with each stage of cancer) for different RCC screening scenarios and the annual transition probabilities for undiagnosed disease becoming diagnosed in the National Health Service. MethodsThe study design and reporting adhered to the Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations. The elicitation was conducted face-to-face or via telephone between each individual expert and the facilitator, aided by online material. For multinomial data, Connor-Mosimann and modified Connor-Mosimann distributions were fitted for each expert and for all experts combined using mathematical linear pooling. ResultsA total of 24 clinical experts were invited, and 71% participated (7 urologists, 6 oncologists, 4 radiologists). The modified Connor-Mosimann distribution provided the best fit for most elicited quantities. Greater uncertainty was noted for the elicited transition probabilities compared with the elicited stage distributions. ConclusionWe performed the first expert elicitation of RCC screening parameters, crucial information that will inform the CEA of screening. In addition, the elicited quantities may enable future health economic evaluations assessing the value of diagnostic tools and pathways in RCC.

Expert judgments calibration and combination for assessment of river levee failure probability

In civil engineering, the utilisation of expert judgement is important for evaluating the reliability of structures whose failure behaviour is little known and difficult to quantify. This article presents the application of the approaches of elicitation, calibration and aggregation of expert opinions to evaluate the reliability of river levees. These approaches have seldom been used in the field of French hydraulic structures whereas there is growing interest for them in other countries, concerning structures reliability and other fields such as aerospace industry, nuclear industry, hydrology, statistics, economics, psychology etc. This article proposes a quantitative approach to the use of expert judgement for evaluating river levee failure probability. An application to the case of an existing river levee is presented. The approach developed assesses the expert opinions according to their calibration (statistical accuracy) and their informativeness, then aggregates them in a single calibrated opinion. Applying the proposed weighting and aggregation procedure reduces variability in the probability of failure estimate, then it provides a better probabilistic distribution of aggregated expert opinion than of individual expert opinions. The results allow identifying a trend of aggregated expert opinions that point towards over- or lack of confidence.

Correction to: Use of Expert Judgement Across NICE Guidance‑Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation

In Sect. 3.3.1, the second sentence of the second paragraph.

Disaggregated Short-Term Inflation Forecast (STIF) for Monetary Policy Decision in Sierra Leone

The study focuses on developing a model of short-term inflation forecasting (STIF) in the context of monetary policy conceptualization. The research toolkit is accompanied using the Box-Jenkins Time Series Approach (ARIMA) to analyze inflation and its risks in Sierra Leone. The proposed model is based on diagrams characterizing the constituents of the Consumer Price Index (CPI) as the primary means of communication to inform the general public about the uncertainty surrounding the dynamics of prices in Sierra Leone. The author points out that such an approach to monetary policy forecasting will allow the use of expert judgment to stabilize the economy in the formation of key areas of monetary policy functioning and development. The article focuses on the interpretation of risks for each of the components of the CPI, which will enhance confidence in the financial and economic decisions made by the financial management of the Bank of Sierra Leone [BSL]. Empirical findings have shown that CPI components such as food and non-alcoholic beverages, housing and health indicate that financial shocks occurring in or outside Sierra Leone can significantly affect the overall CPI. It is stated that such a trend causes a fleeting effect of rising prices on consumer spending in the short term. The study postulates the uniqueness of the component model in the context of turning the key focus of financial management of business structures and government organizations to target sectoral events. As a result of the study, the conclusion is drawn about the need to monitor price volatility by empirically evaluating the components of the CPI basket in the forward-looking perspective. The importance of forecast is been resounded all around the globe, given its relevance in addressing the dynamics of macroeconomic variables and their significant implications for economic stabilisation and decision-making. Keywords: financial management, inflation, forecasting, consumer price index, CPI components; Sierra Leone.

A Multi Criteria Assessment Method for Identifying Vulnerable Marine Ecosystems in the North-East Atlantic

In international fisheries management, scientific advice on the presence of ‘Vulnerable Marine Ecosystems’ (VMEs) per United Nations resolutions, has generally used qualitative assessments based on expert judgement of the occurrence of indicator taxa such as cold-water corals and sponges. Use of expert judgement alone can be criticised for inconsistency and sometimes a lack of transparency; therefore, development of robust and repeatable numeric methods to detect the presence of VMEs would be advantageous. Here, we present a multi-criteria assessment (MCA) method to evaluate how likely a given area of seafloor represents a VME. The MCA is a taxa-dependent spatial method that accounts for both the quantity and data quality available. This was applied to a database of records of VMEs built, held and compiled by the International Council for the Exploration of the Sea (ICES). A VME index was generated which ranged from 1.51 to 4.52, with 5.0 being reserved for confirmed VME habitats. An index of confidence was also computed that ranged from 0.0 to 0.75, with 1 being reserved for those confirmed VME habitats. Overall the MCA captured the important elements of the ICES VME database and provided a simplified, spatially aggregated and weighted estimate of how likely a given area is to contain VMEs. The associated estimate of confidence gave an indication of how uncertain that assessment was for the same given area. This methodology provides a more systematic and standardised approach for assessing the likelihood of presence of VMEs in the North-East Atlantic.

Sequential Refined Partitioning for Probabilistic Dependence Assessment.

Modeling dependence probabilistically is crucial for many applications in risk assessment and decision making under uncertainty. Neglecting dependence between multivariate uncertainties can distort model output and prevent a proper understanding of the overall risk. Whenever relevant data for quantifying and modeling dependence between uncertain variables are lacking, expert judgment might be sought to assess a joint distribution. Key challenges for the use of expert judgment for dependence modeling are over- and underspecification. An expert can provide assessments that are infeasible, i.e., not consistent with any probability distribution (overspecification), and on the other hand, without making very restrictive parametric assumptions an expert cannot fully define a probability distribution (underspecification). The sequential refined partitioning method addresses over- and underspecification while allowing for flexibility about which part of a joint distribution is assessed and its level of detail. Potential overspecification is avoided by ensuring low cognitive complexity for experts through eliciting single conditioning sets and by offering feasible assessment ranges. The feasible range of any (sequential) assessment can be derived by solving a linear programming problem. Underspecification is addressed by modeling the density of directly and indirectly assessed distribution parts as minimally informative given their constraints. Hence, our method allows for modeling the whole distribution feasibly and in accordance with experts' information. A nonparametric way of assessing and modeling dependence flexibly in such detail has not been presented in the expert judgment literature for probabilistic dependence models so far. We provide an example of assessing terrorism risk in insurance underwriting.

Use of Expert Judgement Across NICE Guidance-Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation.

ObjectivesThis study aimed to review current use of experts within National Institute for Health and Care Excellence (NICE) guidance-making programmes, identify improvements in use of expert judgement, and to assess tools and protocols to support the elicitation of information from experts for use by NICE.MethodsThe study comprised a review of NICE process guides; semi-structured interviews with individuals representing each NICE guidance-making programme and a comparison of the suitability of currently available tools and protocols for expert elicitation to the requirements of NICE programmes.ResultsInformation elicited from experts and the way in which it is used varies across NICE guidance-making programmes. Experts’ involvement can be as intensive as being a member of a committee and thus having direct influence on recommendations or limited one-off consultations on specific parameters. We identified 16 tools for expert elicitation that were potentially relevant. None fully met the requirements of NICE, although an existing tool could be potentially adapted. Ongoing research to develop a reference protocol for expert elicitation in healthcare decision making may be of use in future.ConclusionsNICE uses expert judgement across all its guidance-making programmes, but its uses vary considerably. There is no currently available tool for expert elicitation suitable for use by NICE. However, adaptation of an existing tool or ongoing research in the area could address this in the future.

Transmission of climate risks across sectors and borders.

Systemic climate risks, which result from the potential for cascading impacts through inter-related systems, pose particular challenges to risk assessment, especially when risks are transmitted across sectors and international boundaries. Most impacts of climate variability and change affect regions and jurisdictions in complex ways, and techniques for assessing this transmission of risk are still somewhat limited. Here, we begin to define new approaches to risk assessment that can account for transboundary and trans-sector risk transmission, by presenting: (i) a typology of risk transmission that distinguishes clearly the role of climate versus the role of the social and economic systems that distribute resources; (ii) a review of existing modelling, qualitative and systems-based methods of assessing risk and risk transmission; and (iii) case studies that examine risk transmission in human displacement, food, water and energy security. The case studies show that policies and institutions can attenuate risks significantly through cooperation that can be mutually beneficial to all parties. We conclude with some suggestions for assessment of complex risk transmission mechanisms: use of expert judgement; interactive scenario building; global systems science and big data; innovative use of climate and integrated assessment models; and methods to understand societal responses to climate risk. These approaches aim to inform both research and national-level risk assessment.

Structure activity relationship (SAR) toxicological assessments: The role of expert judgment

Structure activity relationships (SAR) and read-across are widely used animal alternative approaches for filling toxicological data gaps. A framework describing the use of expert judgment in evaluating analogs for SAR has been published and widely cited, however, reliance on expert judgment can introduce inconsistent results across experts and hinder transparency. Here we explore the use of a quantitative similarity score between an analog and a Structure of Interest (SOI) to see if these scores correlate with the expert judgement-based suitability rankings. We find these global similarity scores representing a “whole-molecule” view of similarity to be insensitive to differences in local structure which may be important for toxicity, and, therefore, cannot be substituted for expert judgement-based similarity rankings. In this paper, we suggest that the next step in the progression of SAR approaches retains the insights from expert judgment, but facilitates consistency and transparency through the development of rating “rules”. This report outlines and defines analog rating rules for several compound categories. While not comprehensive, the exercises demonstrate the development of rules for categories with a large spread in molecular weight and alkyl chain length and explains the advantages that we see in this approach compared to relying solely on a computational approach or an unstructured expert judgement approach. These rules may be incorporated into analog searching work flows to define boundaries for analogs “suitable” for read-across.

Good practice guide to setting inputs for operational risk models

AbstractThis paper seeks to establish good practice in setting inputs for operational risk models for banks, insurers and other financial service firms. It reviews Basel, Solvency II and other regulatory requirements as well as publicly available literature on operational risk modelling. It recommends a combination of historic loss data and scenario analysis for modelling of individual risks, setting out issues with these data, and outlining good practice for loss data collection and scenario analysis. It recommends the use of expert judgement for setting correlations, and addresses information requirements for risk mitigation allowances and capital allocation, before briefly covering Bayesian network methods for modelling operational risks.

Reporting Guidelines for the Use of Expert Judgement in Model-Based Economic Evaluations.

Expert judgement has a role in model-based economic evaluations (EEs) of healthcare interventions. This study aimed to produce reporting criteria for two types of study design to use expert judgement in model-based EE: (i) an expert elicitation (quantitative) study; and (ii) a Delphi study to collate (qualitative) expert opinion. A two-round online Delphi process identified the degree of consensus for four core definitions (expert; expert parameter values; expert elicitation study; expert opinion) and two sets of reporting criteria in a purposive sample of experts. The initial set of reporting criteria comprised 17 statements for reporting a study to elicit parameter values and/or distributions and 11 statements for reporting a Delphi survey to obtain expert opinion. Fifty experts were invited to become members of the Delphi process panel by e-mail. Data analysis summarised the extent of agreement (using a pre-defined 75% 'consensus' threshold) on the definitions and suggested reporting criteria. Free-text comments were analysed using thematic analysis. The final panel comprised 12 experts. Consensus was achieved for the definitions of expert (88%); expert parameter values (83%); and expert elicitation study (83%). The panel recommended criteria to use when reporting an expert elicitation study (16 criteria) and a Delphi study to collate expert opinion (11 criteria). This study has produced guidelines for reporting two types of study design to use expert judgement in model-based EE: (i) an expert elicitation study requiring 16 reporting criteria; and (ii) a Delphi study to collate expert opinion requiring 11 reporting criteria.

Expert judgement

AbstractExpert judgement has been used since the actuarial profession was founded. In the past, there has often been a lack of transparency regarding the use of expert judgement, even though those judgements could have a very significant impact on the outputs of calculations and the decisions made by organisations. The lack of transparency has a number of dimensions, including the nature of the underlying judgements, as well as the process used to derive those judgements. This paper aims to provide a practical framework regarding expert judgement processes, and how those processes may be validated. It includes a worked example illustrating how the process could be used for setting a particular assumption. It concludes with some suggested tools for use within expert judgement. Although primarily focussed on the insurance sector, the proposed process framework could be applied more widely without the need for significant changes.

An investigation of dependence in expert judgement studies with multiple experts

Expert judgement plays an important role in forecasting and elsewhere, as it can be used to quantify models when no data are available and to improve predictions from models when combined with data. In order to provide defensible estimates of unknowns in an analysis, the judgements of multiple experts can be elicited. Mathematical aggregation methods can then be used to combine these individual judgements into a single judgement for the decision maker. However, most mathematical aggregation methods assume that such judgements come from experts who are independent, which is unlikely to be the case in practice. This paper investigates dependence in expert judgement studies, both within and between experts. It provides the most comprehensive analysis to date by considering all studies in the TU Delft database. It then assesses the practical significance of the dependencies identified in the studies by comparing the performances of several mathematical aggregation methods with varying dependence assumptions. Between-expert correlations were more prevalent than within-expert correlations. For studies that contained between-expert correlations, models which include these produced better forecasts. The implications of this for the use of expert judgement in forecasting are discussed.

The User, not the Tool: Perceptions of Credibility and Relevance Affect the Uptake of Prioritisation

Prioritisation methods have been used in conservation planning for over 20 years. The scientific literature focuses on the technical aspects of prioritisation, providing limited information on factors affecting the uptake of priorities. We focused on the Back on Track species prioritisation program in Queensland, Australia, used to prioritise species conservation efforts across Queensland from 2005. The program had low uptake by intended users. Our study aimed to identify the perceived limitations in the technical-scientific quality of this species-based prioritisation process and its outcomes in terms of credibility (scientific adequacy of the technical evidence) and relevance (of information to the needs of decision-makers). These criteria have been used to understand the uptake of scientific information in policy. We interviewed 73 key informants. Perceptions of credibility were affected by concerns related to the use of expert judgement (rather than empirical evidence) to assess species, impressions that key experts were not included in the planning process, and the lack of confidence in the information supporting prioritisation. We identified several trade-offs and synergies between the credibility and relevance of priorities to potential users. The relevance of the output plans was negatively affected by the lack of clarity about who were potential users and implementers of the priorities identified. We conclude with recommendations to enhance the credibility and relevance of such initiatives.

Adversarial Risk Analysis for Enhancing Combat Simulation Models

Abstract Adversarial Risk Analysis (ARA) builds on statistical risk analysis and game theory to analyze decision situations involving two or more intelligent opponents who make decisions under uncertainty. During the past few years, the ARA approach-which is based on the explicit modelling of the decision making processes of a rational opponent-has been applied extensively in areas such as counterterrorism and corporate competition. In the context of military combat modelling, however, ARA has not been used systematically, even if there have been attempts to predict the opponent’s decisions based on wargaming, application of game theoretic equilibria, and the use of expert judgements. Against this backdrop, we argue that combining ARA with military combat modelling holds promise for enhancing the capabilities of combat modelling tools. We identify ways of combining ARA with combat modelling and give an illustrative example of how ARA can provide insights into a problem where the defender needs to estimate the utility gained from hiding its troop movements from the attacker. Even if the ARA approach can be challenging to apply, it can be instructive in that relevant assumptions about the resources, expectations and goals that guide the adversary’s decisions must be explicated.

Survey of prediction capabilities of three nuclear data libraries for a PWR application

We survey prediction capabilities of ENDF/B-VII.1, JEFF-3.2 and JENDL-4.0u nuclear data libraries (NDLs) for the application of generating two-group homogenized assembly constants for a steady state diffusion model in the context of UAM-LWR (Uncertainty Analysis in Best-Estimate Modeling for Design, Operation and Safety Analysis of LWRs) Benchmark. We consider two different fuel assembly test cases representing a PWR. State of uncertainty quantification in each NDLs is presented for the application. We expect small differences between the NDLs due to the use of expert judgment in the evaluation processes, and identify several order-of-magnitude differences between the NDLs for significant contributors to uncertainty. We also quantify the contribution from cross-material correlations to the uncertainties.

The reliability of setting grade boundaries using comparative judgement

In recent years the use of expert judgement to set and maintain examination standards has been increasingly criticised in favour of approaches based on statistical modelling. This paper reviews existing research on this controversy and attempts to unify the evidence within a framework where expertise is utilised in the form of comparative judgement. Initially, the paper introduces a mathematical model for the way in which comparative judgement may operate. Data from existing studies of comparative judgement are then used to estimate suitable parameters for this model. Having derived a working mathematical model for the operation of expert judgement, the paper will then demonstrate that this model provides results that are broadly consistent with existing research, including both studies that are critical and studies that are supportive of the use of expert judgement. The model will then be applied to examine the required scale and design of expert-driven approaches to standard maintaining in order for these to have any chance of being as reliable as the currently favoured, purely statistical approaches. The paper also discusses the minimum conditions required for this approach to designing expert-driven approaches to be appropriate.

AN ANALYTIC HIERARCHY PROCESS (AHP) APPROACH IN THE SELECTION OF SUSTAINABLE MANUFACTURING INITIATIVES: A CASE IN A SEMICONDUCTOR MANUFACTURING FIRM IN THE PHILIPPINES

This paper proposes an evaluation framework of sustainable manufacturing (SM) initiatives using the hierarchical structure of sustainability indicators set adopted by the US National Institute of Standards and Technology (US NIST) in the context of the Analytic Hierarchy Process (AHP). Evaluating SM initiatives developed by manufacturing firms is crucial for resource allocation, and ensuring that investments enhance the sustainability performance of the firm. This evaluation is a challenge because of the multi-criteria nature of the problem and the presence of subjective criteria for which little or no information on their measurement systems is available. Thus, this study is appropriate due to the following reasons: (1) US NIST provides a comprehensive evaluation model of sustainability with its four-level hierarchy that provides evidence of depth and details of sustainability evaluation, and (2) AHP has the capability to handle multi-level decision-making structure with the use of expert judgments in a pairwise comparison process. A case study of a semiconductor manufacturing firm is presented to illustrate the proposed evaluation framework. Results show that firms must strengthen their financial base through programs that improve efficiency, quality and productivity before carrying out initiatives that address the environment and the immediate community. This work presents a framework that could guide decision-makers, in a way that is simple and comprehensive in their attempt to promote sustainability. http://dx.doi.org/10.13033/ijahp.v7i1.223