Knowledge Elicitation Research Articles

Statement on the active substance acetamiprid.

Acetamiprid is a pesticide active substance with insecticidal action currently under the third renewal (AIR3) of the Commission implementing regulation (EU) No 844/2012. Following concerns that this substance may pose high risks to humans and the environment, the French authorities asked the Commission to restrict its uses under Article 69 of Regulation (EC) No 1107/2009. To support this request, competent Authorities from France cited a series of literature papers investigating its hazards and/or exposure to humans and the environment. Consequently, the EFSA PPR Panel was mandated to advise on the likelihood that body of evidence would constitute proof of serious risks to humans or the environment. Therefore, the EFSA PPR Panel evaluated the likelihood of these studies indicating new or higher hazards and exposure to humans and the environment compared to previous EU assessments.A stepwise methodology was designed, including: (i) the initial screening; (ii) the data extraction and critical appraisal based on the principles of OHAT/NTP; (iii) the weight of evidence, including consideration of the previous EU assessments; (iv) the uncertainty analysis, followed, whenever relevant, by an expert knowledge elicitation process. For human health, no conclusive evidence of higher hazards compared to previous assessment was found for genotoxicity, developmental toxicity, neurotoxicity including developmental neurotoxicity and immunotoxicity. However, due to the lack of adequate assessment of the current data set, the PPR Panel recommends conducting an assessment of endocrine disrupting properties for acetamiprid in line with EFSA/ECHA guidance document for the identification of endocrine disruptors. For environment, no conclusive, robust evidence of higher hazards compared to the previous assessment was found for birds, aquatic organisms, bees and soil organisms. However, the potential of high inter‐species sensitivity of birds and bees towards acetamiprid requires further consideration.

Integrating deep learning and rule-based systems into a smart devices decision support system for visual inspection in production

Due to the increasing implementation of cyber physical systems and the internet of things in industry, there is a trend towards flexible, multi-variant production with a very high degree of automation to strengthen the resilience of the company. Each variant is subject to high quality requirements that must be checked to ensure quality. A common means for this is quality assurance through visual inspections, which are carried out manually. Visual inspections are time-consuming yet prone to wrong decisions, since they are subjective, inconsistent, and susceptible to uncertainties. The quality of the inspection relies heavily on the experience of the personnel. This work addresses this issue through the concept and design of a system for objective decision making in visual inspection by integrating Deep Learning (DL) models with a Belief Rule Based Expert System (BRBES) inside a smart devices application. Smart devices like tablets and smartphones serve to generate information by recording and evaluating image material of the components being inspected. Based on this data, DL models are trained and used to classify defects on new image material to automate part of the inspection process. Furthermore, smart devices serve to provide context-dependent decision recommendations in the visual inspection process, which were calculated by the BRBES with the inclusion of uncertainties. The knowledge base of the BRBES is fed by the expert knowledge of experienced visual inspectors using knowledge elicitation techniques. In this way, the system can enable optimized and objectified visual inspection based on the data-driven and knowledge-driven approaches used. This paper outlines the concept of integrating DL and BRBES into a smart devices decision support system.

Commodity risk assessment of specified species of Lonicera potted plants from Turkey.

The European Commission requested the EFSA Panel on Plant Health to prepare and deliver risk assessments for commodities listed in the Commission Implementing Regulation (EU) 2018/2019 as ‘High‐risk plants, plant products and other objects’. This Scientific Opinion covers plant health risks posed by potted plants (2–4 years old) of specified Lonicera species produced in nurseries and that are imported from Turkey, taking into account the available scientific information, including the technical information provided by the NPPO of Turkey. The relevance of any pest for this Opinion was based on evidence following defined criteria listed in Section 4.1. Three species, the EU‐quarantine pests Lopholeucaspis japonica and Meloidogyne chitwoodi and the protected zone quarantine pest Bemisia tabaci, fulfilled these criteria and were selected for further evaluation. For these pests, the risk mitigation measures proposed in the technical dossier from Turkey were evaluated taking into account the possible limiting factors. For these pests, an expert judgement is given on the likelihood of pest freedom taking into consideration the risk mitigation measures acting on the pest, including uncertainties associated with the assessment. The estimated degree of pest freedom varies among the pests evaluated, with B. tabaci on evergreen species of Lonicera spp. being the pest most frequently expected on the imported plants. The Expert Knowledge Elicitation indicated, with 95% certainty, that between 9,293 and 10,000 plants per 10,000 would be free of B. tabaci.

Functional Resonance Analysis in an Overtaking Situation in Road Traffic: Comparing the Performance Variability Mechanisms between Human and Automation

Automated driving promises great possibilities in traffic safety advancement, frequently assuming that human error is the main cause of accidents, and promising a significant decrease in road accidents through automation. However, this assumption is too simplistic and does not consider potential side effects and adaptations in the socio-technical system that traffic represents. Thus, a differentiated analysis, including the understanding of road system mechanisms regarding accident development and accident avoidance, is required to avoid adverse automation surprises, which is currently lacking. This paper, therefore, argues in favour of Resilience Engineering using the functional resonance analysis method (FRAM) to reveal these mechanisms in an overtaking scenario on a rural road to compare the contributions between the human driver and potential automation, in order to derive system design recommendations. Finally, this serves to demonstrate how FRAM can be used for a systemic function allocation for the driving task between humans and automation. Thus, an in-depth FRAM model was developed for both agents based on document knowledge elicitation and observations and interviews in a driving simulator, which was validated by a focus group with peers. Further, the performance variabilities were identified by structured interviews with human drivers as well as automation experts and observations in the driving simulator. Then, the aggregation and propagation of variability were analysed focusing on the interaction and complexity in the system by a semi-quantitative approach combined with a Space-Time/Agency framework. Finally, design recommendations for managing performance variability were proposed in order to enhance system safety. The outcomes show that the current automation strategy should focus on adaptive automation based on a human-automation collaboration, rather than full automation. In conclusion, the FRAM analysis supports decision-makers in enhancing safety enriched by the identification of non-linear and complex risks.

INFLUENCE FACTORS OF THE TACIT KNOWLEDGE ELICITATION PROCESS: SYSTEMATIC LITERATURE REVIEW

Goal: This systematic review aimed at highlighting and performing an integrated discussion of the factors of the tacit knowledge elicitation process. Design/Methodology/Approach: The research method adopted in this study was (PRISMA) Preferred Reporting Items for Systematic Reviews and Meta-Analyses. The databases, Science Direct, Web of Science, Scopus, and Emerald Insight were chosen based on the possibility of finding articles from collections such as Elsevier, Springer e Taylor & Francis. Search terms related to “knowledge Elicitation” are the following: "knowledge Elicitation" OR "knowledge acquisition" AND technique AND “Tacit Knowledge". Results: The main research findings listed by the article were the inclusion of a knowledge engineer considering the SECI model, the indication of predictability in the perception of episodic knowledge in a tacit knowledge elicitation process, and the hybrid adoption of knowledge elicitation techniques. Limitations of the research: The selection criteria were based only on articles written in the English language and taking into consideration the period from 2008 to 2020. Originality/value: The structure of this article was based on the indication of theoretical gaps and the need to deepen the themes underlying the process of eliciting tacit knowledge, which allowed a systematic exposure of a broad scenario that represents the scope and complexity.

Review of Prior knowledge elicitation: The past, present, and future

Review of Prior knowledge elicitation: The past, present, and future

Inactivation of indicator microorganisms and biological hazards by standard and/or alternative processing methods in Category 2 and 3 animal by-products and derived products to be used as organic fertilisers and/or soil improvers.

The European Commission requested EFSA to assess if different thermal processes achieve a 5 log10 reduction in Enterococcus faecalis or Salmonella Senftenberg (775W) and (if relevant) a 3 log10 reduction in thermoresistant viruses (e.g. Parvovirus) as well as if different chemical processes achieve a 3 log10 reduction of eggs of Ascaris sp., in eight groups of Category 2 and 3 derived products and animal by‐products (ABP). These included (1) ash derived from incineration, co‐incineration and combustion; (2) glycerine derived from the production of biodiesel and renewable fuels; (3) other materials derived from the production of biodiesel and renewable fuels; (4) hides and skins; (5) wool and hair; (6) feathers and down; (7) pig bristles; and (8) horns, horn products, hooves and hoof products. Data on the presence of viral hazards and on thermal and chemical inactivation of the targeted indicator microorganisms and biological hazards under relevant processing conditions were extracted via extensive literature searches. The evidence was assessed via expert knowledge elicitation. The certainty that the required log10 reductions in the most resistant indicator microorganisms or biological hazards will be achieved for each of the eight groups of materials mentioned above by the thermal and/or chemical processes was (1) 99–100% for the two processes assessed; (2) 98–100% in Category 2 ABP, at least 90–99% in Category 3 ABP; (3) 90–99% in Category 2 ABP; at least 66–90% in Category 3 ABP; (4) 10–66% and 33–66%; (5) 1–33% and 10–50%; (6) 66–90%; (7) 33–66% and 50–95%; (8) 66–95%, respectively. Data generation on the occurrence and reduction of biological hazards by thermal and/or chemical methods in these materials and on the characterisation of the usage pathways of ABP as organic fertilisers/soil improvers is recommended.

Expert Knowledge Elicitation to assess the ability of matrices to transmit African swine fever virus

An Expert Knowledge Elicitation (EKE) was carried out regarding the possible contamination with African swine fever virus (ASFV) of products used as pig feed, their traded/imported volumes and their use on pig farms. In addition, the EKE also concerned empty vehicles returning to ASF-unaffected areas of the EU after delivering pigs to ASF-affected areas. The EKE was carried out by three independent groups of six to eight experts each. It was carried out in three steps: assessing the likelihood of contamination of a product at origin; assessing the likelihood of the contaminated product having enough viable virus to infect a pig (the infectious dose); and assessing the volume of trade or imports of each product from an affected area in either the EU or Eurasia which would be delivered to either a small-scale or large-scale pig farm. This report presents the results of the three elicitations. The results of the EKE have been used by the AHAW Panel in a pathway model to determine the likelihood of each product to introduce ASFV into non-affected areas of the EU based on relative risk ranking.

An analytic network process model for risk quantification of mega construction projects

Risk, complexity, and uncertainty are inherent components of megaprojects due to their unique features. However, existing project management practices lack a structured synthesis of these concepts, which leads to unrealistic risk assessments, ineffective management strategies, and poor project performance. In order to fill this gap, this research aims to develop a holistic risk quantification approach incorporating risk-related concepts. For this purpose, a conceptual risk assessment process developed for mega construction projects was operationalized with an Analytic Network Process (ANP) model. The weights of the risk sources in the ANP model were determined by the domain experts through a two-round Delphi study. With the purpose of improving the efficiency and reliability of the knowledge elicitation process, the Delphi study was supported by an interactive data collection tool capable of ANP calculations. The resulting model helped to prioritize the risk sources in mega construction projects. The validity of the findings was tested through the data of 11 mega construction projects. Validation studies revealed the potential of the ANP-based model in quantifying the project risks. Hence, the novel approach proposed in this study is expected to contribute both to the literature by unveiling the interactions between risk-related concepts and to the practitioners by assisting them in assessing the project risks more realistically. Although the risk quantification model has been developed for mega construction projects, it can also be implemented in other project-based industries with minor modifications.

A Survey of Domain Knowledge Elicitation in Applied Machine Learning

Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields.

Cluster-based WSA method to elicit expert knowledge for Bayesian reasoning—Case of parcel delivery with drone

The elicitation of expert knowledge in a Bayesian model is a big challenge. Elicitation requirements increase exponentially with the size of the model, making the expert’s task more difficult and generating problems of quality or coherence of the elicited result. In this paper, we develop an extended Cluster-WSA method using several clusters as an elicitation method. A general formula to calculate the number of information to elicit and a generic algorithm for this method are developed. A selection procedure of some elicitation methods is established and the usefulness of the Cluster-WSA method is verified. The case study of parcel delivery with a drone is used to apply our method, to build a decision support system for diagnosis and prediction, and to make a performance indicator system.

Fusion-Learning of Bayesian Network Models for Fault Diagnostics.

Bayesian Network (BN) models are being successfully applied to improve fault diagnosis, which in turn can improve equipment uptime and customer service. Most of these BN models are essentially trained using quantitative data obtained from sensors. However, sensors may not be able to cover all faults and therefore such BN models would be incomplete. Furthermore, many systems have maintenance logs that can serve as qualitative data, potentially containing historic causation information in unstructured natural language replete with technical terms. The motivation of this paper is to leverage all of the data available to improve BN learning. Specifically, we propose a method for fusion-learning of BNs: for quantitative data obtained from sensors, metrology data and qualitative data from maintenance logs, corrective and preventive action reports, and then follow by fusing these two BNs. Furthermore, we propose a human-in-the-loop approach for expert knowledge elicitation of the BN structure aided by logged natural language data instead of relying exclusively on their anecdotal memory. The resulting fused BN model can be expected to provide improved diagnostics as it has a wider fault coverage than the individual BNs. We demonstrate the efficacy of our proposed method using real world data from uninterruptible power supply (UPS) fault diagnostics.

Application of collaborative serious gaming for the elicitation of expert knowledge and towards creating Situation Awareness in the field of infrastructure resilience

Increasing ageing of transport infrastructures requires a systematic resilience-based maintenance strategy. However, the assessment of risk and resilience is problematic due to the manifold influencing factors and mechanisms within the complex socio-technical system of infrastructure maintenance. This paper provides an analysis of the intricate processes and interdependencies, which influence the potential for resilient infrastructure maintenance. The presented iterative framework incorporates the elicitation of expert knowledge and a serious gaming application. Providing a valid environment for a systematic and guided dialogue among decision makers, serious gaming is highlighted as a valuable research tool that can effectively increase stakeholders' Situation Awareness and thus improve infrastructure resilience itself and its assessment.The authors developed a collaborative board game which emulates real-life decisions of stakeholders in the context of German waterways. In particular, the interaction of the players who take specific stakeholder roles contribute to a deeper understanding of the system, including the impacts of the players' decisions in the real-world system. The paper shows the contribution and potential of serious gaming, including a proposed integration into a Bayesian network as resilience assessment tool.It becomes obvious that the exploitation of expert knowledge to the end of improved systemic decision making requires that the application is closely aligned with the real-world system and the research method in all phases of gamification. On overall, the insights of this study suggest that serious collaborative gaming should be considered more in both research and practice, thus creating Situation Awareness among stakeholders of critical infrastructures (CI).

Commodity risk assessment of Malus domestica plants from Ukraine.

The European Commission requested the EFSA Panel on Plant Health to prepare and deliver risk assessments for commodities listed in Commission Implementing Regulation (EU) 2018/2019 as ‘High risk plants, plant products and other objects’). This Scientific Opinion covers plant health risks posed by 1‐ to 3‐year‐old dormant grafted plants and rootstocks of Malus domestica imported from Ukraine, taking into account the available scientific information, including the technical information provided by Ukraine. All pests associated with the commodity were evaluated against specific criteria for their relevance for this opinion. Two quarantine pests (Lopholeucaspis japonica and Tobacco ringspot virus), one protected zone quarantine pest (Erwinia amylovora) and one non‐regulated pest (Eotetranychus prunicola) that fulfilled all relevant criteria were selected for further evaluation. For Erwinia amylovora, for which special requirements are specified in Commission Implementing Regulation (EU) 2019/2072, Annex X, item 9, the fulfilment of these requirements was evaluated. Based on the information provided in the dossier, the specific requirements for Erwinia amylovora were not met. For the three remaining selected pests, the risk mitigation measures proposed in the technical dossier from Ukraine were evaluated taking into account the possible limiting factors. For the selected pests, an expert judgement is given on the likelihood of pest freedom taking into consideration the risk mitigation measures acting on the pest, including uncertainties associated with the assessment. The degree of pest freedom varies among the pests evaluated, with Eotetranychus prunicola being the pest most frequently expected on the imported plants. The Expert Knowledge Elicitation indicated with 95% certainty that between 9,912 and 10,000 bundles (consisting of 50 plants each) per 10,000 would be free from Eotetranychus prunicola.

Semantic Fusion with Deep Learning and Formal Ontologies for Evaluation of Policies and Initiatives in the Smart City Domain

Decision makers and policy analysts at different administrative levels often lack a holistic view of the problem as there are semantic variations in policy documents due to domain-specific content. For example, smart city initiatives are derived from national and international initiatives which may influence the incentives for local participants, but local initiatives reflect the local contextual elements of the city. Balanced assessment of smart city initiatives should include a systemic evaluation of the initiatives at multiple levels including the city, the country in which the city resides as well as at international level. In this paper, a knowledge elicitation methodology is presented for multi-granularity evaluation of policies and initiatives. The methodology is demonstrated on the evaluation of smart city initiatives generated at different administrative levels. Semantic networks are constructed using formal ontologies and deep learning methods for automatic semantic evaluation of initiatives to abstract knowledge found in text. Three smart city initiatives published by different administrative levels including international, national, and city level are evaluated in terms of relevance, coherence, and alignment of multi-level smart city initiatives. Experiments and analysis ultimately provide a holistic view of the problem which is necessary for decision makers and policy analysts of smart cities.

One Coin, Two Sides: Eliciting Expert Knowledge From Training Participants in a Capacity-Building Program for Veterinary Professionals.

Scientific research may include the elicitation of judgment from non-academic subject-matter experts in order to improve the quality and/or impact of research studies. Elicitation of expert knowledge or judgment is used when data are missing, incomplete, or not representative for the specific setting and processes being studied. Rigorous methods are crucial to ensure robust study results, and yet the quality of the elicitation can be affected by a number of practical constraints, including the understanding that subject-matter experts have of the elicitation process itself. In this paper, we present a case of expert elicitation embedded within an extended training course for veterinary professionals as an example of overcoming these constraints. The coupling of the two activities enabled extended opportunities for training and a relationship of mutual respect to be the foundation for the elicitation process. In addition, the participatory research activities reinforced knowledge synthesis objectives of the educational program. Finally, the synergy between the two concurrent objectives may produce benefits which transcend either independent activity: solutions and ideas built by local professionals, evolving collaborative research and training approaches, and a network of diverse academic and practicing professionals. This approach has the versatility to be adapted to many training and research opportunities.

Application of Methods to Assess Animal Welfare and Suffering Caused by Infectious Diseases in Cattle and Swine Populations.

Simple SummaryInfectious disease control in livestock is often motivated by food safety concerns and economic impact. However, most diseases also affect animal welfare. We established an approach to quantify the welfare effect of infectious diseases in cattle (three diseases) and pigs (two diseases). A “suffering score” was established based on the aggregation of severity, duration, and frequency of clinical entities of the diseases. The resulting suffering scores were then used to compare the welfare impact of the different diseases and for comparison to other common welfare hazards. For example, the approach suggested that bovine viral diarrhoea and paratuberculosis are more severe for cattle than infectious bovine rhinotracheitis. In pigs, porcine reproductive and respiratory syndrome has a much bigger welfare impact than Aujeszky’s disease, assuming all diseases remain endemic.Control of infectious diseases in livestock has often been motivated by food safety concerns and the economic impact on livestock production. However, diseases may also affect animal welfare. We present an approach to quantify the effect of five infectious diseases on animal welfare in cattle (three diseases) and pigs (two diseases). We grouped clinical manifestations that often occur together into lists of clinical entities for each disease based on literature reviews, and subsequently estimated “suffering scores” based on an aggregation of duration, frequency, and severity. The duration and severity were based on literature reviews and expert knowledge elicitation, while frequency was based mainly on estimates from the literature. The resulting suffering scores were compared to scores from common welfare hazards found under Danish conditions. Most notably, the suffering scores for cattle diseases were ranked as: bovine viral diarrhoea and infection with Mycobacterium avium subsp. paratuberculosis > infectious bovine rhinotracheitis, and for pigs as: porcine respiratory and reproductive syndrome > Aujeszky’s disease. The approach has limitations due to the limited data available in literature and uncertainties associated with expert knowledge, but it can provide decision makers with a tool to quantify the impact of infections on animal welfare given these uncertainties.

Knowledge Elicitation with Aboriginal Australian communities

This paper provides techniques for engagement and data collection in researching with Aboriginal Australian cultures, acknowledging significant differences in forms of communication and usage of Information Systems to the mainstream Australian culture. An approach to trust in relationships is developed by interpreting cultural aspects arising from the diverse relationships to technology developed by Aboriginal users. This work uses the existing Honeycomb model for social media development as a base framework for collaborative web systems and online knowledge sharing in the Indigenous domain. We present a series of product development research projects based in universities in NSW Australia, in particular user experience studies, to explain the relationship between the researchers and users and the products that are created in terms of the model. Some concepts and processes fundamental to engagement with Aboriginal Australian communities in the supply of appropriate information sharing technology are discussed in this context. For if Aboriginal people are to engage in IS development, we are sharing the knowledge or the culture that is embedded in the technology which can have detrimental effects. Either we are asking them to enter the culture that created the technology, that is assimilate, or we use engagement in design to change that technology to suit the culture.

Putting Gonuke Into Practice: Considerations for Human Factors Evaluations

The Guideline for Operational Nuclear Usability and Knowledge Elicitation (GONUKE) framework was introduced in 2015 to support human factors evaluations needed for control room upgrades at nuclear power plants. NUREG-0711, the Human Factors Engineering Program Review Model, is used by the U.S. Nuclear Regulatory Commission to review human factors activities associated with human-system interfaces at nuclear power plants, and GONUKE is anchored to the phases of development and design in NUREG-0711. This paper addresses five considerations to help users of GONUKE better apply the framework to evaluations for NUREG-0711 and beyond. These five considerations are: (1) GONUKE only specifies evaluation, not design; (2) GONUKE is a framework, not a method or process; (3) GONUKE goes beyond NUREG-0711 requirements; (4) GONUKE application shouldfollow a graded approach; (5) different evaluations are required fo r formative vs. summative phases.

Investigating the role of Non-Technical Skills in Railway Traffic Operations through Expert Knowledge Elicitation

Many high-risk industries identify non-technical skills as safety-critical abilities of the operational staff that have a protective function against human fallibility. Based on an established non-technical skills classification system, methods for expert knowledge elicitation were used to describe non-technical skills in the specific context of train traffic control in the Netherlands. The findings offer insights regarding the skill importance for good operational outcomes, skill difficulty, categorization, and attitudes based on subject matter experts’ opinions. Substantial overlap between the employed non-technical skills framework and the observed expert classification was found, which might indicate that the experts utilize a mental model of nontechnical skills similar to the one used. Furthermore, considerations concerning the organizational culture and the attitudes towards change provide a promising outlook when introducing novel solutions to non-technical skill training and assessment.