Knowledge Of Multiple Experts Research Articles

OP141 Expert Knowledge Elicitation in Health Technology Assessment: Our Experience Using the Sheffield Elicitation Framework

IntroductionExpert judgement has an important role in health technology assessment (HTA), including as a source of evidence to inform economic modeling when published data are lacking. Quantitative information may be elicited from experts to inform model inputs and associated uncertainty using one of many expert elicitation methodologies. Here, the feasibility and potential benefits of one expert elicitation method, the Sheffield Elicitation Framework (SHELF), to the HTA process is examined.MethodsThe SHELF method seeks to express the knowledge of multiple experts in the form of a subjective probability distribution. Eliciting a subjective probability distribution allows the uncertainty of experts to be included in probabilistic sensitivity analysis, which is becoming an increasingly prominent feature of HTAs. The individual knowledge of participating experts is combined through behavioral aggregation, where experts participate in a discussion before being asked to provide judgments from the perspective of a rational impartial observer. The whole process is led by a facilitator who ensures all participants contribute and confirm that the final distribution is a product of consensus, not compromise.ResultsWe recently conducted two SHELF elicitations as part of an ongoing project aiming to streamline the assessment of positron emission tomography (PET) in Australia. These elicitations provided insight into the usefulness of SHELF within the HTA setting. Given the constraints imposed by the COVID-19 pandemic, the elicitation sessions were conducted online rather than in the ideal face-to-face manner. In collaboration with one of the developers, we successfully adapted the method by making use of video conferencing technology to provide an online environment that mimicked the face-to-face setup as much as possible.ConclusionsSHELF provides a rigorous and scientific method by which to elicit the knowledge of multiple experts in the form of a probability distribution. However, the method is resource intensive and may be best reserved for when data on key drivers are lacking.

A semi-supervised algorithm for improving the consistency of crowdsourced datasets: The COVID-19 case study on respiratory disorder classification

Background and Objective: Cough audio signal classification is a potentially useful tool in screening for respiratory disorders, such as COVID-19. Since it is dangerous to collect data from patients with contagious diseases, many research teams have turned to crowdsourcing to quickly gather cough sound data. The COUGHVID dataset enlisted expert physicians to annotate and diagnose the underlying diseases present in a limited number of recordings. However, this approach suffers from potential cough mislabeling, as well as disagreement between experts.Methods: In this work, we use a semi-supervised learning (SSL) approach – based on audio signal processing tools and interpretable machine learning models – to improve the labeling consistency of the COUGHVID dataset for 1) COVID-19 versus healthy cough sound classification 2) distinguishing wet from dry coughs, and 3) assessing cough severity. First, we leverage SSL expert knowledge aggregation techniques to overcome the labeling inconsistencies and label sparsity in the dataset. Next, our SSL approach is used to identify a subsample of re-labeled COUGHVID audio samples that can be used to train or augment future cough classifiers.Results: The consistency of the re-labeled COVID-19 and healthy data is demonstrated in that it exhibits a high degree of inter-class feature separability: 3x higher than that of the user-labeled data. Similarly, the SSL method increases this separability by 11.3x for cough type and 5.1x for severity classifications. Furthermore, the spectral differences in the user-labeled audio segments are amplified in the re-labeled data, resulting in significantly different power spectral densities between healthy and COVID-19 coughs in the 1-1.5 kHz range (p=1.2×10−64), which demonstrates both the increased consistency of the new dataset and its explainability from an acoustic perspective. Finally, we demonstrate how the re-labeled dataset can be used to train a COVID-19 classifier, achieving an AUC of 0.797.Conclusions: We propose a SSL expert knowledge aggregation technique for the field of cough sound classification for the first time, and demonstrate how it can be used to combine the medical knowledge of multiple experts in an explainable fashion, thus providing abundant, consistent data for cough classification tasks.

Fuzzy Cognitive Maps Based on D-Number Theory

In real life, there will be a lot of uncertainty problems, one of which is due to the vagueness of the concept of things, that is, it is difficult to determine whether an object conforms to this concept. This situation widely exists in some states, phenomena, parameters and interrelationships between things. For such uncertain events with heavy subjective influencing factors and incomplete data, it is suitable to use fuzzy methods to deal with them. In present our work, we firstly introduce the notion of D-number cognitive maps (DCMs), which are intelligent framework models based on D-number theory and cognitive maps. Compared with Evidential Cognitive Maps (ECMs) and Fuzzy Cognitive Maps (FCMs), DCMs can fuse multiple sources of information with uncertainty and construct a cognitive map model with incomplete and conflicting information. To better solve the problem of knowledge combination, D-number fuzzy cognitive maps (DFCMs) are also constructed based on D-number theory and fuzzy cognitive maps. In many practical applications, the establishment of fuzzy cognitive maps is usually completed by using a simple arithmetic average method for multiple experts to obtain comprehensive fuzzy cognitive maps. This simple processing method may lead to the loss of some important information so that the synthesized results do not reflect reality. To overcome this challenge, we synthesize the knowledge of multiple experts by using D-number theory and the characteristics of the representation of expert knowledge in FCMs. It is based on expert knowledge, and the synthesized reliability distribution function is used as the basis of final weight synthesis.

Prototype of an Expert System for the Selection of Aircraft Structural Materials

Expert systems (ES) are widely used for engineering and manufacturing applications nowadays. In order to solve the problems in material selection in the aviation field, an aircraft material expert system (AMES) was developed. This expert system is based on the knowledge of multiple experts and aviation data. A decision support system (DSS) was designed for selecting the materials used in aviation fields. The flows of material selection were designed, and the models of AMES were established. A data base system was also designed to implement AMES.

Identifying the implied: Findings from three differentiated replications on the use of security requirements templates

Identifying security requirements early on can lay the foundation for secure software development. Security requirements are often implied by existing functional requirements but are mostly left unspecified. The Security Discoverer (SD) process automatically identifies security implications of individual requirements sentences and suggests applicable security requirements templates. The objective of this research is to support requirements analysts in identifying security requirements by automating the suggestion of security requirements templates that are implied by existing functional requirements. We conducted a controlled experiment in a graduate-level security class at North Carolina State University (NCSU) to evaluate the SD process in eliciting implied security requirements in 2014. We have subsequently conducted three differentiated replications to evaluate the generalizability and applicability of the initial findings. The replications were conducted across three countries at the University of Trento, NCSU, and the University of Costa Rica. We evaluated the responses of the 205 total participants in terms of quality, coverage, relevance and efficiency. We also develop shared insights regarding the impact of context factors such as time, motivation and support, on the study outcomes and provide lessons learned in conducting the replications. Treatment group, using the SD process, performed significantly better than the control group (at p-value <0.05) in terms of the coverage of the identified security requirements and efficiency of the requirements elicitation process in two of the three replications, supporting the findings of the original study. Participants in the treatment group identified 84 % more security requirements in the oracle as compared to the control group on average. Overall, 80 % of the 111 participants in the treatment group were favorable towards the use of templates in identifying security requirements. Our qualitative findings indicate that participants may be able to differentiate between relevant and extraneous templates suggestions and be more inclined to fill in the templates with additional support. Security requirements templates capture the security knowledge of multiple experts and can support the security requirements elicitation process when automatically suggested, making the implied security requirements more evident. However, individual participants may still miss out on identifying a number of security requirements due to empirical constraints as well as potential limitations on knowledge and security expertise.

Development of fuzzy cognitive map (FCM)‐based scenarios for wind energy

PurposeThe purpose of this paper is to explore a very new approach of creating scenarios with fuzzy cognitive maps (FCM); specifically to create scenarios for wind energy deployment.Design/methodology/approachIn this paper a small exercise is conducted to illustrate the usefulness of FCM‐based scenarios.FindingsScenario‐based planning is often used in technology management and scenarios are generated to cater for uncertainty and facilitate development of responsive and robust strategic plans. Scenario analysis helps to identify different alternatives of the future state of technologies, needs, policies and environment. Scenario planning helps to overcome thinking limitations by presenting multiple futures. FCMs are based on causal cognitive maps and combine the benefits of both qualitative and quantitative analysis. Domain knowledge of multiple experts can be integrated into an FCM to present collective knowledge base.Originality/valueThis is a conceptual paper exploring use of fuzzy cognitive maps to create scenarios for wind energy deployment. The paper provides scenarios and a way to integrate them into technology roadmaps.

Multi-agent knowledge integration mechanism using particle swarm optimization

Unstructured group decision-making is burdened with several central difficulties: unifying the knowledge of multiple experts in an unbiased manner and computational inefficiencies. In addition, a proper means of storing such unified knowledge for later use has not yet been established. Storage difficulties stem from of the integration of the logic underlying multiple experts' decision-making processes and the structured quantification of the impact of each opinion on the final product. To address these difficulties, this paper proposes a novel approach called the multiple agent-based knowledge integration mechanism (MAKIM), in which a fuzzy cognitive map (FCM) is used as a knowledge representation and storage vehicle. In this approach, we use particle swarm optimization (PSO) to adjust causal relationships and causality coefficients from the perspective of global optimization. Once an optimized FCM is constructed an agent based model (ABM) is applied to the inference of the FCM to solve real world problem. The final aggregate knowledge is stored in FCM form and is used to produce proper inference results for other target problems. To test the validity of our approach, we applied MAKIM to a real-world group decision-making problem, an IT project risk assessment, and found MAKIM to be statistically robust.

Handling and updating uncertain information in bow-tie analysis

Bow-tie analysis is a fairly new concept in risk assessment that can describe the relationships among different risk control parameters, such as causes, hazards and consequences to mitigate the likelihood of occurrence of unwanted events in an industrial system. It also facilitates the performance of quantitative risk analysis for an unwanted event providing a detailed investigation starting from basic causes to final consequences. The credibility of quantitative evaluation of the bow-tie is still a major concern since uncertainty, due to limited or missing data, often restricts the performance of analysis. The utilization of expert knowledge often provides an alternative for such a situation. However, it comes at the cost of possible uncertainties related to incompleteness (partial ignorance), imprecision (subjectivity), and lack of consensus (if multiple expert judgments are used). Further, if the bow-tie analysis is not flexible enough to incorporate new knowledge or evidence, it may undermine the purpose of risk assessment. Fuzzy set and evidence theory are capable of characterizing the uncertainty associated with expert knowledge. To minimize the overall uncertainty, fusing the knowledge of multiple experts and updating prior knowledge with new evidence are equally important in addition to addressing the uncertainties in the knowledge. This paper proposes a methodology to characterize the uncertainties, aggregate knowledge and update prior knowledge or evidence, if new data become available for the bow-tie analysis. A case study comprising a bow-tie for a typical offshore process facility has also been developed to describe the utility of this methodology in an industrial environment.

An Efficient Multi-Expert Knowledge Capture Technique

Capturing tacit knowledge from multiple experts in different location have not been exactly the easiest of activity for organizations and knowledge developers alike. In this research, the time tested and trusted Delphi technique is critically analyzed to allow for the automation of its processes. A simple prototype design is used to model the algorithm for the creation of an e- Delphi; a framework prototype and a guide to system developers . The algorithm and the framework presented in this work will help in the creation of KMSs and web based systems that can capture multiple expert knowledge in a collaborative environment.

확장된 Fuzzy AHP를 이용한 효율적인 의사결정

웹상에서 이용할 수 있는 방대한 문서의 집합인 WWW은 사용자를 위한 다양한 정보의 보고이다. 그러나 불필요한 정보의 필터링이나 사용자가 필요한 정보를 검색하는데 많은 시간이 소요되어 효율적인 의사결정을 하는데 어려움이 있다. 본 논문에서는 의사결정에 관한 요소를 계층화 구조로 나타내는 AHP나 Fuzzy AHP방법들을 데이터의 관점에서 대안, 평가기준, 주관적 속성가중치, 개념과 객체 사이에 퍼지 관계를 기반으로 웹 자원을 효과적으로 관리하고 의사결정을 할 수 있는 EFAM(Extended Fuzzy AHP Method) 모델을 제안하였다. 제안한 EFAM 모델은 웹상의 효율적인 문서검색과 특정 영역의 문제를 의사결정하기 위하여 영역의 코퍼스로부터 추출된 개념들이 가지는 의미론적 내용에 감성 기준을 고려함으로써 효율적으로 문서를 추출할 수 있어서 명확한 의사결정을 할 수가 있음을 실험을 통하여 확인한다. WWW which is an applicable massive set of document on the Web is a thesaurus of various information for users. However, Search engines spend a lot of time to retrieve necessary information and to filter out unnecessary information for user. In this paper, we propose the EFAM(the Extended Fuzzy AHP Method) model to manage the Web resource efficiently, and to make a decision in the problem of specific domain definitely. The EFAM model is concerned with the emotion analysis based on the domain corpus information, and it composed with systematic common concept grids by the knowledge of multiple experts. Therefore, The proposed the EFAM model can extract the documents by considering on the emotion criteria in the semantic context that is extracted concept from the corpus of specific domain and confirms that our model provides more efficient decision-making through an experiment than the conventional methods such as AHP and Fuzzy AHP which describe as a hierarchical structure elements about decision-making based on the alternatives, evaluation criteria, subjective attribute weight and fuzzy relation between concept and object.

Aggregated Causal Maps: An Approach To Elicit And Aggregate The Knowledge Of Multiple Experts

This paper presents a systematic procedure to elicit and aggregate the knowledge of multiple individual experts and represent it in the form of an Aggregated Causal Map (ACM). This procedure differs from existing methods in two ways. First, unlike other methods, this method does not rely on group interaction in eliciting knowledge of multiple experts, and, therefore, is not fraught with biases associated with group dynamics. Second, this method uses both the idiographic and nomothetic approaches while existing methods focus on nomothetic approaches to knowledge elicitation. We draw on the strengths of both approaches by using the idiographic approach to elicit and aggregate the knowledge of multiple experts and the nomothetic approach to validate the knowledge elicited. We illustrate the procedure by constructing the ACM of eight key decision makers about an enterprise system adoption decision.

A multi-expert system for material cutting plan generation

Construction firms specializing in large commercial buildings must often design and build steel structural elements as a part of each project. Such firms must purchase large steel plates, cut them into pieces and then weld the pieces into H-beams and other construction components. We formalize the material ordering and cutting problem faced by this industry and propose an expert system for efficiently combining steel elements into plates to control relevant costs. This expert system is based on the procedural knowledge of multiple experts rather than on the rules of a single expert as is more common. We tested the expert system using data supplied by Lien-Kang Heavy Industrial Company, Ltd. (LK) as well as using simulated data with known optimal solutions. Comparison to LK's solutions indicates that the expert system solution is less costly in every historical project. Testing on 40 simulated projects reveals how problem parameters affect performance. The expert system has been embedded into a decision support system (DSS) that LK is currently using. The qualitative benefits of using this computerized system include a significant reduction in the time and effort required for generating a solution and increased report accuracy.

Elicitation of knowledge from multiple experts using network inference

Eliciting knowledge from multiple experts usually entails the use of groups, and thus is subject to the problems inherent in group dynamics. We present a technique for multiple expert knowledge acquisition that does not rely upon the use of groups and can take advantage of technological advances in communications and computing, i.e., the Internet. The approach uses influence diagrams to represent the individual expert's understanding of the problem situation and develops a Multiple Expert Influence Diagram (MEID), a composite representation of the experts' knowledge. Following a review of present methods for multiple expert knowledge elicitation, we formally define the MEID, describe its manner of construction, and discuss its interpretation. We continue with a review of the issues to be faced in implementation of the technique, and give an illustrative example. Finally, we emphasize the need to provide users of decision aids with defensible measures of the quality of the rules produced by these aids. The MEID-approach is intended to serve as a first step in this direction.

Ordered logic: defeasible reasoning for multiple agents

Our goal is to provide a theoretical foundation for knowledge based applications which support nonmonotonic or defeasible reasoning and which incorporate the knowledge of multiple experts in a principled way. We present a generalized proof theory for defeasible reasoning and briefly explain the relationship of this system to other nonmonotonic formalisms. Then we present a proof theory and semantics for a logic that allows us to explicitly model internal perspectives or multiple agents. This ordered logic properly extends defeasible logic by allowing more complex precedence structure on rules. It provides a mechanism for resolving conflicts between competing perspectives without obscuring the opinions of those perspectives. We show which defeasible theories can be transformed into an equivalent theory in the new logic for multiagent reasoning and how a theory in the logic for multiagent reasoning can be transformed into an equivalent set of defeasible theories, one for each perspective or agent. We present a proof theory for quantified ordered logic and introduce the concept of a curtain which shields the cognitive resources of one agent from another agent. Finally, we consider some examples and discuss how it is possible to specify the relative superiority of some experts over others in a natural way using ordered logic.

IDSSs opportunities and problems: Steps to development of an IDSS

IDSSs should contribute to the enhancement of human performance, but their effectiveness can be guaranteed only in the case of certain decision types. The issues explored in this paper show that they can help to overcome some human limitations, especially in complex data and information processes, in uncertainty management, and in coherent reasoning. Integrating human and machine expertise is clearly beneficial, nevertheless with the aim of building intelligent solutions we should not ignore the role of human factors and the problems deriving from the integration of knowledge of multiple experts. The risk is that the systems will become clumsy and vulnerable to embarrassing failures.

Computerized conferencing for knowledge acquisition from multiple experts

Abstract The increasing complecity of many expert system application areas calls for the integration of the knowledge of multiple experts. The use of multiple experts introduces some interesting new problems during the process of knowledge acquisition. The problems are further complicated when the experts are geographically dispersed or unavailable for face-to-face interactions. This article discusses the motivations for acquiring the knowledge of multiple experts, the problems related to knowledge acquisition, new issues that arise whens multiple experts interact, solutions that can be brought to bear in building multiple expert systems (particularly when experts are geographically dispersed), and new tools for knowledge engineers to use when dealing with multiple experts.

A new class of intelligent knowledge-based systems with an optimisation-based inference engine

In this paper we describe a new class of intelligent knowledge-based system (IKBS) which can be used principally for managerial decision making applications. This class of applications often requires a framework for knowledge acquisition which allows the system to use the knowledge of several experts. In addition, since in most business decision making the objective is maximise profits, there is a need for an inference engine which allows optimisation to be carried out. The new class of IKBS which is described in this paper has both these properties, i.e., the ability to use the knowledge of multiple experts in a convenient way and an inference engine which by performing optimisations is able to pick out the profit maximising decisions. As an illustration of these concepts, a system for allocation decision making is described. The system ‘Retail-opt’ allows the user to solve problems like allocation of space in retail outlets, allocation of space in mail order catalogues, pricing policy decisions for discounted airline tickets, etc. In the paper, the basic concepts behind ‘Retail-opt’ are described and an application of ‘Retail-opt’ to the problem of retail space allocation in a Scandinavian Department Store is given. A number of other systems which use these concepts for more complicated competitive decision making situations are also described.