Expert Opinion Elicitation Research Articles

An AHP-Delphi multi-criteria location planning model with application to Whey Protein Production Facility decisions

Whey is produced as a by-product from the manufacture of cheese or caseins and its disposal is potentially a serious environmental problem. This paper proposes establishing a Whey Protein Production Facility (WPPF) as an independent facility that would be supplied with whey by-products from Hamedan province cheese factories. The alternative sites for a WPPF are industrial complexes in the Hamedan region. The Delphi technique was used to elicit expert opinions about criteria for evaluating the locations. The criteria identified by the experts were land expenses, industrial complex facilities, convenience of access to sources of whey protein, lack of competing factories and transportation. The Analytic Hierarchy Process (AHP) and Expert Choice software were used to compare the alternatives on the criteria derived from the Delphi process.

Application of the RAND/UCLA Appropriateness Method to Evaluate an Informative System for Liver Transplantation in Adult and Pediatric Recipients

The Delphi Method (DM) is the most frequently used technique to acquire structured expert-opinion elicitation (EOE). It has been increasingly applied to construct guidelines in medicine and to evaluate the appropriateness of clinical procedures. In this study, the RAND/UCLA appropriateness method was used as a structured EOE process to evaluate the appropriateness of a dataset concerning liver transplantation in adult and pediatric recipients for an information system funded by the Italian Ministry of Health. The original dataset was obtained using an interdisciplinary pool of regional experts ( n = 60). This dataset held 280 items stratified into three groups: I. pretransplant items ( n = 123); II. transplant items ( n = 65); III. early posttransplant and follow-up items ( n = 92). In the second DM round, the dataset was subjected to an extraregional panel of independent experts ( n = 9) to assess a score ranging from 1 to 9 on each item based on increasing appropriateness, according to the RAND/UCLA Appropriateness Method. Overall agreement, uncertainty, and disagreement between experts was 95.89%, 3.12%, and 0.99%, respectively. For each group, agreement-uncertainty-disagreement were 99.35%/0.65%/0% (group I), 91.53%/5.30%/3.17% (group II), and 96.87%/3.13%/0% (group III), respectively. This study supported the use of a structured EOE process to evaluate the appropriateness of a large dataset for liver transplantation activity.

Eliciting Expert Opinion for Economic Models: An Applied Example

Objective Expert opinion is considered as a legitimate source of information for decision-analytic modeling where required data are unavailable. Our objective was to develop a practical computer-based tool for eliciting expert opinion about the shape of the uncertainty distribution around individual model parameters. Methods We first developed a prepilot survey with departmental colleagues to test a number of alternative approaches to eliciting opinions on the shape of the uncertainty distribution around individual parameters. This information was used to develop a survey instrument for an applied clinical example. This involved eliciting opinions from experts to inform a number of parameters involving Bernoulli processes in an economic model evaluating DNA testing for families with a genetic disease, hypertrophic cardiomyopathy. The experts were cardiologists, clinical geneticists, and laboratory scientists working with cardiomyopathy patient populations and DNA testing. Results Our initial prepilot work suggested that the more complex elicitation techniques advocated in the literature were difficult to use in practice. In contrast, our approach achieved a reasonable response rate (50%), provided logical answers, and was generally rated as easy to use by respondents. The computer software user interface permitted graphical feedback throughout the elicitation process. The distributions obtained were incorporated into the model, enabling the use of probabilistic sensitivity analysis. Conclusion There is clearly a gap in the literature between theoretical elicitation techniques and tools that can be used in applied decision-analytic models. The results of this methodological study are potentially valuable for other decision analysts deriving expert opinion.

Geographically assisted elicitation of expert opinion for regression models

One of the perceived strengths of Bayesian modelling is the ability to include prior information. Although objective or noninformative priors might be preferred in some situations, in many other applications the Bayesian framework offers a real opportunity to formally combine data with information available from experts. The question addressed in this paper is how to elicit this information in a form suitable for prior modelling. Particular attention is paid to geographic data for which maps might be used to assist in the elicitation. Two case studies are used to illustrate the methodology: estimation of city house prices and prediction of presence of a rare species.

A framework to implement QCS through process cost modeling

PurposeThe purpose of this paper is to describe a structured framework to implement and sustain a quality costing system (QCS) based on process cost modeling (PCM) in process industries.Design/methodology/approachAfter reviewing and analyzing various cost accounting methodologies practiced by companies the research objectives were achieved by acknowledging the need to attach fuzziness to notion of “quality”. The imprecise, vague, and complex information related to cost items under Prevention, Appraisal and Failure (PAF) segments is synthesized using well‐established fuzzy principles. A case based approach from process industry is discussed to implement and sustain quality costing system after prioritizing the processes.FindingsWhile conforming on the results of prior research on practice of quality costing approaches and the problems faced by the companies in implementing a quality management system the fuzzy approach (owing to its sound logic and effectiveness in identifying the vagueness and imprecision in human judgment) is successfully applied to elicit expert opinion regarding the importance of cost items. The information so obtained after fuzzy synthesis is used to set up priority with respect to the processes which can provide necessary help to managers/practioneners to invest efforts in reduction of cost of non‐conformances (CONC) and optimal allocation of resources.Practical implicationsThe approach discussed in the paper will be helpful to managers; quality practitioners to set up/improve various quality improvement initiatives for successful implementation of quality costing system.Originality/valueThe framework discussed in the paper provides a novel approach to implement QCS by using PCM after judicious selection of the processes and cost items.

Comment: Expert Elicitation for Reliable System Design

The paper by Bedford, Quigley and Walls has re viewed the role of expert judgements in reliability as sessment within the engineering design process. It dif fers from many statistically based papers that focus mainly on statistical models, in that it addresses issues related to the process, the concept and principles. The paper itself is comprehensive in coverage and offers general guidance for reliability assessment at the sys tem design stage. However, to some extent, the tech nical side of the statistical techniques used in expert opinion elicitation in reliability analysis has not been fully explored. A practitioner who wants to conduct re search in the area still needs to search methodologies from other recommended sources in the paper to find practical solutions. This is perhaps the purpose of the paper. In my discussion, two techniques that are particu larly useful to subjective data elicitation, but are not discussed in the Bedford, Quigley and Walls paper, namely, empirical Bayes (EB; Robbins, 1955) and evi dential reasoning (ER; Dempster, 1966), will be briefly discussed. It is hoped that this will shed some light on the problem of expert data elicitation in general and re liability assessment in design in particular.

Multi‐period model for disruptive events in interdependent systems

AbstractThis paper develops the Multi‐Period Model for Disruptive Events in Interdependent Systems (MPMDEIS). The analysis is divided into two principal periods: (1) an event period that lasts for the duration of the active perturbation and results in the degradation of the interdependent systems, and (2) a recovery period during which the systems evolves to a new equilibrium. The event period perturbation is modeled as a shock that cascades among the interdependent systems and reduces their production capacity. The state of the systems at the end of the event period is specified by its operational unavailability vector, which is given by the initial perturbation strength vector multiplied by an exponential matrix with the interdependency matrix P as the exponent. The coefficients of P represent the relative contribution of the loss of capacity of one system to the loss of another system as the perturbation propagates from one to the other. The quantification of the shock and interdependencies is critical for the realistic assessment of the inflicted damage and formulating effective defensive strategies against terrorist attacks, natural hazards, and technological failures or accidents. The recovery period depends on the implemented response actions and it can be modeled using a systems dynamics approach. The standard economic input‐output tables are unlikely to be appropriate for the events of interest. However, there are probabilistic risk analysis techniques such as eliciting expert opinions and analytic methods which can be used to reliably assess these data. The MPMDEIS is compared with the Inoperability Input‐Output Model (IIM) and Dynamic Input‐Output Inoperability Model (DIIM) and it is demonstrated to avoid IIM and DIIM assumptions that can lead to significant errors for disruptive events in interdependent systems. © 2006 Wiley Periodicals, Inc. Syst Eng 9: 281– 295, 2006

Preliminary results of an expert-opinion elicitation process to prioritize an informative system funded by Italian ministry of health for cadaveric donor management, organ allocation, and transplantation activity

Expert-opinion elicitation (EOE) is a heuristic process for gathering evidence and data or answering questions on issues/problems of concern. The Delphi method (DM) is the most frequent technique used to obtain structured elicitation of expert opinions. It has been increasingly applied in medicine to produce guidelines and to evaluate the appropriateness of diagnostic procedures. In this study, a DM has been used as structured EOE process to assess the appropriateness and clinical priority of a data set–based informative system in the context of the Liguria-Trento Transplant Network (LTTN) Project, funded by Italian Ministry of Health. The original data set was obtained by using an interdisciplinary pool of regional experts (n = 60). This data set held 1506 items stratified in 21 categories at various surgical phases (preoperative, intraoperative, and postoperative) and transplantation types (liver, kidney, and kidney/pancreas) in adult and pediatric recipients. Some categories included cadaveric donor management, organ allocation, and acute liver failure. In the second DM round, the data set was subjected to a panel of extraregional, independent experts (n = 9) to assess scores ranging from 1 to 9 on each item, based on increasing appropriateness/priority, according to RAND/UCLA Appropriateness Method. The overall agreement between experts was 95.88%, whereas disagreement and uncertainty were 0.13% and 3.98%, respectively. A major uncertainty occurred for the data set concerning the multiorgan cadaveric donor, for liver transplantation, and for kidney transplantation in adult recipients. The use of a structured EOE process may represent an effective strategy to define the appropriateness and prioritization criteria of a large data set in the field of solid organ transplantation.

Key components of drug treatment provision for young people: a Delphi approach

To date, most drug treatment services have dealt with adult clients, often stretching their services to include a younger age group. As drug services specifically for young people are being developed, there is scant evidence of the effectiveness of treatment provision on which to draw. Consequently, policy makers, service commissioners, and service providers face the problem of trying to make decisions in a situation where there is little information concerning the key components of accessible and effective services for young drug users. The current authoritative guidance is contained within the Health Advisory Service (HAS) Report (1996), and the subsequent Health Advisory Service (HAS) Review (2001). The HAS Report and the HAS Review were generated by consensus methodologies based on eliciting expert opinion. The aim of this study was to obtain information by using a different, and arguably more rigorous, method to the consensus methodologies used in the HAS Report and the HAS Review. A Delphi consensus methodology was used in this study in order to corroborate (or otherwise) the current HAS guidance, and thus better inform service provision. The Delphi panel was mainly in agreement with the recommendations of the HAS Report and the HAS Review, but placed less weight on physiological and pharmaceutical interventions and more emphasis on interventions that may lead to social integration. The implications with regard to future service development are discussed in light of the findings of this Delphi study.

A ranking of software engineering measures based on expert opinion

This research proposes a framework based on expert opinion elicitation, developed to select the software engineering measures which are the best software reliability indicators. The current research is based on the top 30 measures identified in an earlier study conducted by Lawrence Livermore National Laboratory. A set of ranking criteria and their levels were identified. The score of each measure for each ranking criterion was elicited through expert opinion and then aggregated into a single score using multiattribute utility theory. The basic aggregation scheme selected was a linear additive scheme. A comprehensive sensitivity analysis was carried out. The sensitivity analysis included: variation of the ranking criteria levels, variation of the weights, variation of the aggregation schemes. The top-ranked measures were identified. Use of these measures in each software development phase can lead to a more reliable quantitative prediction of software reliability.

회색문헌 통합관리 체계에 관한 연구

본 연구는 국내 각 기관에서 소장하거나 생산하고 있는 회색문헌을 체계적으로 수집하고 공유하여 활용할 수 있는 방안을 제시하였다. 이를 위해서 회색문헌이 갖고 있는 특성과 유형을 분석하였으며, 회색문헌이 갖는 특성 가운데 회색문헌의 공유를 저해하는 열람제한의 특징에 대해서도 조사하였다. 또한 이러한 제한을 극복하기 위한 방법을 도출하고 실제적인 회색문헌 집중관리 시스템을 개발하기 위해 필요한 기초 정보를 조사하였다. 기초자료와 정보를 입수하기 위해 회색문헌을 수집, 처리 및 서비스하는 기관의 정보관리 담당자를 대상으로 설문조사를 실시하였다. 설문결과 정보관리담당자들은 보고서를 회색문헌의 여러 종류중에서 가장 중요하다고 판단하였으며, 지도와 학위논문과 같은 자료에 대해서는 그 중요도를 상대적으로 낮게 보고 있었다. 또한 자관 회색문헌에 대해서는 보수적인 입장을 견지하고 있으며 외부 자료에 대해서는 열람을 희망하고 있었다. 따라서 회색문헌 공유를 위한 유인책으로는 경제적 지원보다 자료의 공유라는 유인책이 효과적임을 확인하였다. The purposes of the study were to identify the characteristics and types of grey literatures, to investigate the factors restricting the sharing of grey literatures in Korea, and to make recommendations for the efficient collection and utilization system of grey literatures being produced and managed by either government agencies or government-supported research institutes in Korea. A survey instrument was designed to measure the current status of grey literature management systems and to elicit expert opinions from information professionals in information centers for an efficient system of grey literatures. It was found that information professionals considered the technical reports as the most important type of grey literatures, but maps and dissertations as the least important type of information value. Related to the participation for the central system of grey literatures, many respondents preferred to utilize the grey literatures owned by other information centers, but they showed a neutral interest level in sharing their information centers' grey literatures with other centers.

Elicitation of Expert Opinion in Benefit Transfer of Environmental Goods

Benefit transfer is a method for estimating the value of environmental goods that involves the use of past information on identical or similar goods. This paper considers the extent to which benefit transfer can be based on prior distributions elicited from expert opinion. We propose two alternative methods to elicit the parameters of a prior distribution from experts on environmental valuation. An experiment is carried out on the value of National Parks in Spain. The results from the elicited distributions are compared with the information provided by onsite samples of visitors. The results indicate that individual experts made different predictions about the potential value of the policy areas that were diverse and unable to accurately predict the value for each policy site. However, the average across the elicited distributions approaches the estimated distribution with empirical data and accurately predicts the relative values for the two policy sites considered.

Strategic Implementation of Infrastructure Priority Projects: Case Study in Palestine

A strategy was developed for implementation and management of multisector urban infrastructure projects. The strategy includes risk-based analytical hierarchy process (AHP) for project prioritization that is based on project deliverables and project life-cycle and implementation guidelines. The expert-opinion elicitation process used for this study consists of a variation of the Delphi technique, scenario analysis, civil works, and nuclear industry recommendations. The AHP methodology utilizes a multicriteria decision-making technique that allows the consideration of both objective and subjective factors in obtaining cardinal priority ranking of infrastructure projects. The methodology, which deals with different fields of infrastructure, can incorporate uncertainty in the process and can be implemented using simple spreadsheet format. The methodology was developed for a group of players (methodology implementers) involved in implementation and management of urban infrastructure projects. These include the decision makers, the stakeholders who can influence the decision and/or be affected by it, and the analysts. The methodology was demonstrated in this paper using real-life applications for effectiveness in prioritizing infrastructure projects from mixed infrastructure sectors in Palestine.

Differences and similarities among experts’ opinions on Salmonella enterica dynamics in swine pre-harvest

A workshop was conducted to elicit expert opinion on infection status and transmission of salmonella in pigs at the farm of origin, during transport and during lairage. A second objective was to compare opinions regarding risk factors for salmonella introduction and control at the farm level between experts from different countries. Thirty-six experts from 11 countries filled in a paper-and-pencil questionnaire during an international conference. Experts from all countries agreed on the risk of salmonella introduction related to live animals and the importance of general hygiene and all-in/all-out management for salmonella control. However, workshop participants from Denmark put more weight on factors related to feed while experts from USA rated factors related to rodents and people contact higher. The experts believed that 21–33% of pigs coming from a chronically infected farm would be infected with salmonella, but only one-third of the infected pigs would be shedders. Regarding transport and lairage, the US participants believed that contamination was occurring in the majority of pigs regardless of initial infection status and particularly during lairage; in contrast, the Danish experts were more optimistic with respect to the contamination and infection risk. US experts believed that the incidence of salmonella shedding would be high among pigs infected during transport and lairage and that an important proportion of ‘carrier’ pigs would start shedding. Our results reflect the differences in the level of salmonella infection and implemented control strategies between countries as well as the different philosophies that professionals have. The differences in opinion regarding salmonella dynamics could be due either to true differences in risk as a consequence of distinct management and transport practices in Denmark and USA or to a difference in perception.

Delphi Method in Web Site Selection: Using the Experts

Summary The development of Web sites is proceeding at an ever-quickening pace. Adult learners can benefit from the wealth of information accessible in both new sites and changing older sites. However, that excitement is tempered by frustration in following blind leads or encountering questionable quality. One approach is to stay abreast of the latest material by attempting to look at and read every new and modified site. However, time is finite, and many of our students have lives filled with conflicting needs. The Delphi method, a sociological research technique first developed in defense research, is used to elicit expert opinion as a short cut to evaluating Web sites. A specific example with business-related sites is presented as a test case in developing and applying the methodology. The use of Delphi, an oracle-based technique, seems quite appropriate as we assist our adult students to allocate their time wisely.

Conceptual and methodological issues in public health performance measurement: results from a computer-assisted expert panel process.

As part of the developmental process for the National Public Health Performance Standards Program, the CDC convened a group of experts in the fields of public health practice and research to evaluate key conceptual and methodological issues involved in measuring the performance of public health organizations. Participants engaged in a nominal group process and an electronic polling exercise designed to elicit expert opinions about these issues. Results revealed broad consensus around the need for measurement systems that support quality improvement and accountability applications, with scientific investigation viewed as an important but secondary objective of measurement. Substantial variation was observed in perceptions about the importance of specific measurement concepts and methods. Results highlight the need for performance measurement systems to reflect multiple organizational perspectives in their design and implementation.

Experiences in elicitation [Read before The Royal Statistical Society at a meeting on 'Elicitation' on Wednesday, April 16th, 1997, the President, Professor A. F. M. Smith in the Chair

Summary. Elicitation of expert opinion is becoming increasingly important in the elicitation of prior distributions. In this paper, the psychology of elicitation and the currently available methods are briefly reviewed, but the primary discussion is on the distinction between 'general' elicitation methods for a class of problems and 'application-specific' methods which are useful only once. Examples of both types of elicitation are given, along with a discussion about general versus application-specific methods, and predictive versus structural elicitation.

Two methods for the structured assessment of model uncertainty by experts in performance assessments of radioactive waste repositories

The assessment of the performance of high-level radioactive waste repositories is based on the use of models for predicting system behaviour. The complexity of the system together with the large spatial and temporal scales imposed by the regulations introduce large uncertainties in the analysis. The difficulty of validating the relevant models creates the need of assessing their validity by means of expert judgments. This paper addresses the problem of model uncertainty both from a theoretical and a practical point of view and presents two mathematical approaches to the treatment of model uncertainty that can assist the experts in the formulation of their judgments. The formal elicitation of expert judgments is investigated within the Technical Facilitator/Integrator (TFI) framework that has been proposed by the Senior Seismic Hazard Analysis Committee. Within this framework, the mathematical formulations for the treatment of model uncertainty are regarded as tools for sensitivity analyses that give insights into the model characteristics and are helpful in structuring the expert opinion elicitation process itself. The first approach, referred to as the alternate-hypotheses formulation, amounts to constructing a suitable set of plausible hypotheses and evaluating their validity. The second approach to model uncertainty is referred to as the adjustment-factor formulation and it requires that a reference model be identified and its predictions be directly modified through an adjustment factor that accounts for the uncertainty in the models. Furthermore, both approaches require a clear understanding of the distinction between aleatory and epistemic uncertainties. The implications that these two formulations have on, and the issues that they raise in, the elicitation of expert opinions are explored. A case study of model uncertainty regarding alternative models for the description of groundwater flow and contaminant transport in unsaturated, fractured tuff is presented.

Bayes binomial sampling by attributes with a general-beta prior distribution

In binomial sampling the standard beta is frequently used as a prior due to its conjugate property for a sample from a Bernoulli distribution. This article looks into the case when this prior is a general beta. Practical advantages in the elicitation of expert opinion (to obtain the prior) and a convenient expression of the posterior are presented. A computer program (in double precision) on an IBM-PC (or compatible) permits the simple use of the general beta. This program was used to solve the numerical examples, showing the advantages of using the general beta as a prior.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Expert judgment in maintenance optimization

A comprehensive method for the use of expert opinion for obtaining lifetime distributions required for maintenance optimization is proposed. The method includes procedures for the elicitation of discretized lifetime distributions from several experts, the combination of the elicited expert opinion into a consensus distribution, and the updating of the consensus distribution with failure and maintenance data. The development of the method was prompted by the lack of statistical training of the experts and the high demands on their time. The use of a discretized life distribution provides more flexibility, is more comprehendible by the experts in the elicitation stage, and greatly reduces the computation in the combination and updating stages. The methodology is Bayes, using the Dirichlet distribution as the prior distribution for the elicited discrete lifetime distribution. Methods are described for incorporating information concerning the expertise of the experts into the analysis.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>