Expert Elicitation Methods Research Articles

OP141 Expert Knowledge Elicitation in Health Technology Assessment: Our Experience Using the Sheffield Elicitation Framework

IntroductionExpert judgement has an important role in health technology assessment (HTA), including as a source of evidence to inform economic modeling when published data are lacking. Quantitative information may be elicited from experts to inform model inputs and associated uncertainty using one of many expert elicitation methodologies. Here, the feasibility and potential benefits of one expert elicitation method, the Sheffield Elicitation Framework (SHELF), to the HTA process is examined.MethodsThe SHELF method seeks to express the knowledge of multiple experts in the form of a subjective probability distribution. Eliciting a subjective probability distribution allows the uncertainty of experts to be included in probabilistic sensitivity analysis, which is becoming an increasingly prominent feature of HTAs. The individual knowledge of participating experts is combined through behavioral aggregation, where experts participate in a discussion before being asked to provide judgments from the perspective of a rational impartial observer. The whole process is led by a facilitator who ensures all participants contribute and confirm that the final distribution is a product of consensus, not compromise.ResultsWe recently conducted two SHELF elicitations as part of an ongoing project aiming to streamline the assessment of positron emission tomography (PET) in Australia. These elicitations provided insight into the usefulness of SHELF within the HTA setting. Given the constraints imposed by the COVID-19 pandemic, the elicitation sessions were conducted online rather than in the ideal face-to-face manner. In collaboration with one of the developers, we successfully adapted the method by making use of video conferencing technology to provide an online environment that mimicked the face-to-face setup as much as possible.ConclusionsSHELF provides a rigorous and scientific method by which to elicit the knowledge of multiple experts in the form of a probability distribution. However, the method is resource intensive and may be best reserved for when data on key drivers are lacking.

Using an experiment among clinical experts to determine the cost and clinical impact of rapid whole exome sequencing in acute pediatric settings.

Evaluate the cost and clinical impacts of rapid whole-exome sequencing (rWES) for managing pediatric patients with unknown etiologies of critical illnesses through an expert elicitation experiment. Physicians in the intervention group (n = 10) could order rWES to complete three real-world case studies, while physicians in the control group (n = 8) could not. Costs and health outcomes between and within groups were compared. The cost incurred in the intervention group was consistently higher than the control by 60,000-70,000 THB. Fewer other investigation costs were incurred when rWES could provide a diagnosis. Less cost was incurred when an rWES that could lead to a change in management was ordered earlier. Diagnostic accuracy and the quality of non-pharmaceutical interventions were superior when rWES was available. In acute pediatric settings, rWES offered clinical benefits at the average cost of 60,000-70,000 THB. Whether this test is cost-effective warrants further investigations. Several challenges, including cost and ethical concerns for assessing high-cost technology for rare diseases in resource-limited settings, were potentially overcome by our study design using expert elicitation methods.

Characterization Factors to Assess Land Use Impacts on Pollinator Abundance in Life Cycle Assessment.

While wild pollinators play a key role in global food production, their assessment is currently missing from the most commonly used environmental impact assessment method, Life Cycle Assessment (LCA). This is mainly due to constraints in data availability and compatibility with LCA inventories. To target this gap, relative pollinator abundance estimates were obtained with the use of a Delphi assessment, during which 25 experts, covering 16 nationalities and 45 countries of expertise, provided scores for low, typical, and high expected abundance associated with 24 land use categories. Based on these estimates, this study presents a set of globally generic characterization factors (CFs) that allows translating land use into relative impacts to wild pollinator abundance. The associated uncertainty of the CFs is presented along with an illustrative case to demonstrate the applicability in LCA studies. The CFs based on estimates that reached consensus during the Delphi assessment are recommended as readily applicable and allow key differences among land use types to be distinguished. The resulting CFs are proposed as the first step for incorporating pollinator impacts in LCA studies, exemplifying the use of expert elicitation methods as a useful tool to fill data gaps that constrain the characterization of key environmental impacts.

How to measure the social sustainability of the circular economy? Developing and piloting social circular economy indicators in Finland

The purpose of the study was to research the possibility and methods of measuring the social sustainability of the circular economy (CE) transition nationally and sub-nationally in Finland based on secondary and regularly updated data. The aim was to assess both the social impacts of, and socio-cultural changes needed for, the transition. Expert elicitation method was used to develop experimental pilot indicators for Finland that would cover different types of social implications and CE principles and be based on secondary data sources. 12 indicators proceeded into piloting and data collection. The study confirmed the difficulty of monitoring the social sustainability for two main reasons. First, social outcomes are often very contextual and the result of complex personal and inter-personal relations, and they are difficult to connect to only the CE development. Second, the conventional data sources are based on a linear logic and are not fit to measure the circular processes. The monitoring results show weak signs for a socially just development and socio-cultural change towards the CE. The study answers to the research gap between CE practices and conceptual frameworks for monitoring the social sustainability of the CE. The real-world national and sub-national monitoring based on data readily available in regularly updated open databases was piloted to allow continuous monitoring to support strategic regional development. The study presents a model for an iterative and systemic process to develop social indicators for the CE that can also be applied outside of Finland.

Quantitative Evaluation Method for Industrial Control System Vulnerability Based on Improved Expert Elicitation and Fuzzy Set Method

Quantitative Evaluation Method for Industrial Control System Vulnerability Based on Improved Expert Elicitation and Fuzzy Set Method

Estimating the Cost of 3 Risk Prediction Strategies for Potential Use in the United Kingdom National Breast Screening Program.

Economic evaluations have suggested that risk-stratified breast cancer screening may be cost-effective but have used assumptions to estimate the cost of risk prediction. The aim of this study was to identify and quantify the resource use and associated costs required to introduce a breast cancer risk-stratification approach into the English national breast screening program. A micro-costing study, conducted alongside a cohort-based prospective trial (BC-PREDICT), identified the resource use and cost per individual (£; 2021 price year) of providing a risk-stratification strategy at a woman's first mammography. Costs were calculated for 3 risk-stratification approaches: Tyrer-Cuzick survey, Tyrer-Cuzick with Volpara breast-density measurement, and Tyrer-Cuzick with Volpara breast-density measurement and testing for 142 single nucleotide polymorphisms (SNP). Costs were determined for the intervention as implemented in the trial and in the health service. The cost of providing the risk-stratification strategy was calculated to be £16.45 for the Tyrer-Cuzick survey approach, £21.82 for the Tyrer-Cuzick with Volpara breast-density measurement, and £102.22 for the Tyrer-Cuzick with Volpara breast-density measurement and SNP testing. This study did not use formal expert elicitation methods to synthesize estimates. The costs of risk prediction using a survey and breast density measurement were low, but adding SNP testing substantially increases costs. Implementation issues present in the trial may also significantly increase the cost of risk prediction. This is the first study to robustly estimate the cost of risk-stratification for breast cancer screening. The cost of risk prediction using questionnaires and automated breast density measurement was low, but full economic evaluations including accurate costs are required to provide evidence of the cost-effectiveness of risk-stratified breast cancer screening. Economic evaluations have suggested that risk-stratified breast cancer screening may be a cost-effective use of resources in the United Kingdom.Current estimates of the cost of risk stratification are based on pragmatic assumptions.This study provides estimates of the cost of risk stratification using 3 strategies and when these strategies are implemented perfectly and imperfectly in the health system.The cost of risk stratification is relatively low unless single nucleotide polymorphisms are included in the strategy.

Estimating the major replacement rates in next-generation offshore wind turbines using structured expert elicitation.

With offshore wind turbines continuing to increase in size and move further offshore and into harsher environments, the complexity of carrying out the major replacement of large components is expected to pose a significant challenge for future offshore wind farms. However, the rate of major replacement operations that will be required in these next generation offshore wind turbines is currently unknown. Using a structured expert elicitation method, based on the Classical Model and implemented using EFSA guidance for the practical application of structured expert elicitation, major replacement rates of large components (generator, gearbox, and rotor) were systematically estimated for four next generation offshore wind turbine configurations, based on the knowledge of six wind energy experts. The results presented in this paper are based on an equal-weighting aggregation approach. The major replacement rate values found using this approach are presented and compared between different turbine configurations. Based on these results, it is expected that a larger number of major replacement operations are more likely to be required in medium-speed turbine configurations, in comparison to direct- drive, and in floating turbines, in comparison to fixed-foundation turbines.

A systematic review into expert knowledge elicitation methods for emerging food and feed risk identification

An emerging risk results “… from a newly identified hazard to which a significant exposure may occur, or from an unexpected new or increased significant exposure and/or susceptibility to a known hazard”. In the field of emerging food and feed risks, expert knowledge elicitation methodologies represent important tools for identifying and addressing data gaps associated with emerging risk identification, particularly under conditions of risk uncertainty and/or ambiguity. A systematic review was conducted to identify expert knowledge elicitation methods which have been used in the context of emerging food and feed risks. The primary research questions were which existing expert knowledge elicitation methodologies have been used to study emerging food and feed risks? and what contexts or situations have been studied using expert knowledge elicitation methodologies in relation to emerging food and feed-related risks? Three databases were searched: Web of Science Core Collection, Scopus, and Google Scholar. Searches covered all studies published from 1998 onwards. A total of 59 studies were included in the review following the application of the inclusion and exclusion criteria. Given the heterogeneity of the included studies, a thematic analysis was applied to assess these papers in relation to the research questions. There was no one expert knowledge elicitation method consistently adopted for the identification of a specific type of emerging food and feed risk. Method triangulation was observed in 23 studies and was reported to improve the validity of results in 15 of these. Most published research considered emerging risks associated with societal controversy (“ambiguous risks”). Although the use of methodological triangulation appears to be helpful in relation to understanding emerging food and feed risks, future research into the development of a harmonised framework will enable the identification, processing, and evaluation of emerging food risks in a systemic way which can facilitate comparative analysis and harmonise mitigation strategies to address emerging risks and their drivers.

An application of fault tree analysis for computing the bounds on system failure probability through qualitative data in intuitionistic fuzzy environment

AbstractFor the safety assessment of complex engineering systems, fault tree analysis (FTA) is a widely used method and can be performed only when all its basic events (BEs) have their corresponding failure data. However, in many applications, obtaining precise quantitative failure data of all the components of the system is not feasible due to changing environment, insufficient data or new components. The present research develops a novel FTA to identify the best estimate and bounds of system failure probability through qualitative data processing. The failure data of BEs are uncertain therefore expert elicitation method is used incorporating intuitionistic fuzzy (IF) set theory to evaluate the top event failure probabilities. The novelty of the proposed work comes with the use of linguistic values as failure possibilities of the system components and utilizing triangular intuitionistic fuzzy numbers (TIFNs) to quantify the linguistic terms into mathematical form to assess the best‐estimated value of the system failure probability, and also the lower bound and the upper bound of the system failure probability. To exemplify the appropriateness of the proposed study, the case studies Combustion Engineering Reactor Protection System (CERPS) and Emergency Core Cooling System (ECCS) are carried out and a comparison between computed results and outcomes from the previous methods is demonstrated. The results are exceptionally near to the outcomes of conventional FTA, which confirms that the proposed procedure is a real substitute in the IF environment when components have imprecise quantitative failure data.

The design of a matrix linking work situations to chemical health risk at the workplace

In France, laws require each company to draw up an inventory of the risks that may threaten employees’ health in order to prioritize the preventive actions to be implemented. Focusing on chemical risk, databases on hazards or exposures are widely available but they lack information regarding chemical risks resulting from combining the hazards of chemicals with their conditions of use, thus generating exposures. Our objective is to build a matrix of French work situations associated with their chemical risk. Eighty-eight work situations were collected from reports written by professionals from the French public health insurance service. Each work situation is defined by descriptive parameters of the task, the exposure, and the hazard. According to an expert elicitation method (Delphi, n = 21 experts), each work situation was assessed and a chemical risk score defined, taking into account all the descriptive exposure and hazard parameters. Chemical risk scores were expressed as a range of values from 0 to 100, with the size of the range chosen by the experts themselves according to their uncertainty. The experts' assessments were merged to assign one risk score for each work situation, variability, and confidence. The results showed that 50% of the work situations had a risk score between 40 and 60. The average variability and confidence were around 15% and 82%, respectively. This work situation matrix constructed from French data can be used by occupational safety and health managers that have similar work situations in their company (Western European industrial sector). In this context, it may be useful to easily determine the level of risks for similar tasks and prioritize those that are most risky. Moreover, it could be used to compare and define the differences between a risk assessment performed by "expertise" and another defined by a software.

Eliciting uncertainty for complex parameters in model-based economic evaluations: quantifying a temporal change in the treatment effect.

In model-based economic evaluations, the effectiveness parameter is often informed by studies with a limited duration of follow-up, requiring extrapolation of the treatment effect over a longer time horizon. Extrapolation from short-term data alone may not adequately capture uncertainty in that extrapolation. This study aimed to use structured expert elicitation to quantify uncertainty associated with extrapolation of the treatment effect observed in a clinical trial. A structured expert elicitation exercise was conducted for an applied study of a podiatry intervention designed to reduce the rate of falls and fractures in the elderly. A bespoke web application was used to elicit experts' beliefs about two outcomes (rate of falls and odds of fracture) as probability distributions (priors), for two treatment options (intervention and treatment as usual) at multiple time points. These priors were used to derive the temporal change in the treatment effect of the intervention, to extrapolate outcomes observed in a trial. The results were compared with extrapolation without experts' priors. The study recruited thirty-eight experts (geriatricians, general practitioners, physiotherapists, nurses, and academics) from England and Wales. The majority of experts (32/38) believed that the treatment effect would depreciate over time and expressed greater uncertainty than that extrapolated from a trial-based outcome alone. The between-expert variation in predicted outcomes was relatively small. This study suggests that uncertainty in extrapolation can be informed using structured expert elicitation methods. Using structured elicitation to attach values to complex parameters requires key assumptions and simplifications to be considered.

The risk assessment of a crude oil pipeline using fuzzy and bayesian based bow-tie analysis

Cross country pipelines are one of the major modes of transportation of petroleum products. Due to external and internal factors, the failure probability of pipeline is increasing over the past decades. The failure of pipelines can cause fires, which can result in environmental damage, loss of materials and further resulting into casualties. Effective risk assessment is vital to prevent and mitigate such potential incidents. In this work, an efficient tool that makes use of Bayesian approach along with Bow-tie analysis is used to obtain the failure frequencies of all the identified causes that may lead to failure of the selected pipeline located in Kerala, India. Bayesian networks can perform effective dynamic risk analysis by considering the conditional dependencies between various basic events leading to pipeline failure. The fuzzy logic and expert elicitation method are incorporated to determine the prior failure frequencies of all the identified causes of pipeline failure. The effectiveness of Bayesian network in performing forward analysis to determine the probabilities of pipeline failure consequences is demonstrated. This study also identifies the weak links associated with the occurrence of particular consequences, so that adequate measures can be taken to rectify them.

Effectiveness of the Preventive Intervention of Chromosomal Disorders of Iran's Community Genetics Program: Application of Bayesian Network.

Background: Evidence-based policymaking for the genetic preventive interventions at the community level requires information on the effectiveness of interventions in the operational areas taking into account the characteristics of health system and customer behaviour. These information are limited in many low- and middle-income countries. In this study, we estimated the effectiveness of preventive interventions for chromosomal disorders using the conceptual framework of Iran’s community genetics program (ICGP) using a Bayesian Network as a modeling method in limited access situation to the complete and accurate observational data. Methods: Expert elicitation method based on global and national scientific evidences was applied to determine the structure of the Bayesian Network (BN) and to quantify the probability of nodes. The nomological and face validity of the network was checked. Also, a sensitivity analysis against the sources of uncertainty of probabilities was conducted. Results: By ICGP interventions, 63% (95% CI, 0.55-0.71) of all chromosomal disorders can be prevented, which is responsible for 80% (95% CI, 0.76-0.84) and 38% (95% CI, 0.31-0.45) reduction of expected baseline birth prevalence of trisomis and other autosomal disorders, respectively. Improving the access to and the uptake of screening service can also result in a 12% and 11% increase in effectiveness, respectively. Conclusion: Effectiveness of ICGP’s intervention is between the same interventions’ effectiveness in Western Europe and the Eastern Mediterranean region. Opportunities for increasing the uptake of and the access to the interventions are strengthening the public genetic literacy and implantation of a system of laboratory sample transfer at the side of the utilization of telehealth for delivering the counseling services at remote areas.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review.

Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual. We conducted a systematic review to compare EE methods used to generate model input parameters in health research. PubMed and Web of Science. Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes. Studies were classified as "formal" EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as "indeterminate" methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods. Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods. The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16. Elicitations on nonhealth topics and those reported in the gray literature were not included. We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models. We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.There is a need to establish expert elicitation best practices and reporting guidelines.

Assessing how ecosystem-based adaptations to climate change influence community wellbeing: a Vanuatu case study

Climate change poses significant threats to wellbeing and livelihoods of people and the ecosystems in many Small Island Developing States (SIDS). Adaptation solutions must counteract these threats while also supporting development in vulnerable SIDS. Suitable options need to ensure that connections between the social, economic, and environmental dimensions of socio-economic systems are defined in a way that can support how decisions are made (and by whom) and how these can impact on other parts of these systems. This is particularly important in many Pacific SIDS, where communities practise customary natural resource management and continue to rely on local natural resources. In this study, we model the anticipated impacts of climate change and the benefits of the ecosystem-based adaptation (EbA) approaches on community wellbeing in Vanuatu. To do this, we applied participatory and expert elicitation methods to develop a Bayesian network model, which was designed to evaluate community wellbeing responses at four explicit spatial scales. The model includes both acute and chronic impacts of climate change, the impact of coral bleaching, and the potential loss of Vanuatu’s fringing coral reefs. The model predicts that all proposed EbA interventions will have a positive impact on wellbeing in all four locations to some degree, by either directly improving the integrity of Vanuatu’s ecosystems or by protecting these ecosystems as a positive spill-over of related actions. Significantly, it also predicts that if climate change exceeds 1.5 °C of warming, the costs of achieving the same level of wellbeing are increased.

Predicting species and community responses to global change using structured expert judgement: An Australian mountain ecosystems case study.

Conservation managers are under increasing pressure to make decisions about the allocation of finite resources to protect biodiversity under a changing climate. However, the impacts of climate and global change drivers on species are outpacing our capacity to collect the empirical data necessary to inform these decisions. This is particularly the case in the Australian Alps which have already undergone recent changes in climate and experienced more frequent large‐scale bushfires. In lieu of empirical data, we use a structured expert elicitation method (the IDEA protocol) to estimate the change in abundance and distribution of nine vegetation groups and 89 Australian alpine and subalpine species by the year 2050. Experts predicted that most alpine vegetation communities would decline in extent by 2050; only woodlands and heathlands are predicted to increase in extent. Predicted species‐level responses for alpine plants and animals were highly variable and uncertain. In general, alpine plants spanned the range of possible responses, with some expected to increase, decrease or not change in cover. By contrast, almost all animal species are predicted to decline or not change in abundance or elevation range; more species with water‐centric life‐cycles are expected to decline in abundance than other species. While long‐term ecological data will always be the gold standard for informing the future of biodiversity, the method and outcomes outlined here provide a pragmatic and coherent basis upon which to start informing conservation policy and management in the face of rapid change and a paucity of data.

Comparing expert elicitation and model-based probabilistic technology cost forecasts for the energy transition

We conduct a systematic comparison of technology cost forecasts produced by expert elicitation methods and model-based methods. Our focus is on energy technologies due to their importance for energy and climate policy. We assess the performance of several forecasting methods by generating probabilistic technology cost forecasts rooted at various years in the past and then comparing these with observed costs in 2019. We do this for six technologies for which both observed and elicited data are available. The model-based methods use either deployment (Wright's law) or time (Moore's law) to forecast costs. We show that, overall, model-based forecasting methods outperformed elicitation methods. Their 2019 cost forecast ranges contained the observed values much more often than elicitations, and their forecast medians were closer to observed costs. However, all methods underestimated technological progress in almost all technologies, likely as a result of structural change across the energy sector due to widespread policies and social and market forces. We also produce forecasts of 2030 costs using the two types of methods for 10 energy technologies. We find that elicitations generally yield narrower uncertainty ranges than model-based methods. Model-based 2030 forecasts are lower for more modular technologies and higher for less modular ones. Future research should focus on further method development and validation to better reflect structural changes in the market and correlations across technologies.

Defining Cyber Security and Cyber Security Risk within a Multidisciplinary Context using Expert Elicitation.

It is important to have and use standardized terminology and develop a comprehensive common understanding of what is meant by cyber security and cyber security risk given the multidisciplinary nature of cyber security and the pervasiveness of cyber security concerns throughout society. Using expert elicitation methods, collaborating cyber researchers from multiple disciplines and two sectors (academia, government–military) were individually interviewed and asked to define cyber security and cyber security risk. Data‐driven thematic analysis was used to identify the most salient themes within each definition, sector, and cyber expert group as a whole with results compared to current standards definitions. Network analysis was employed to visualize the interconnection of salient themes within and across sectors and disciplines. When examined as a whole group, “context‐driven,” “resilient system functionality,” and “maintenance of CIA (confidentiality, integrity, availability)” were the most salient themes and influential network nodes for the definition of cyber security, while “impacts of CIA vulnerabilities,” “probabilities of outcomes,” and “context‐driven” were the most salient themes for cyber security risk. We used this expert elicitation process to develop comprehensive definitions of cyber security (cybersecurity) and cyber security risk that encompass the contextual frameworks of all the disciplines represented in the collaboration and explicitly incorporates human factors as significant cyber security risk factors.

Elicitation of US and Chinese expert judgments show consistent views on solar geoengineering

Expert judgments on solar geoengineering (SG) inform policy decisions and influence public opinions. We performed face-to-face interviews using formal expert elicitation methods with 13 US and 13 Chinese climate experts randomly selected from IPCC authors or supplemented by snowball sampling. We compare their judgments on climate change, SG research, governance, and deployment. In contrast to existing literature that often stress factors that might differentiate China from western democracies on SG, we found few significant differences between quantitative judgments of US and Chinese experts. US and Chinese experts differed on topics, such as desired climate scenario and the preferred venue for international regulation of SG, providing some insight into divergent judgments that might shape future negotiations about SG policy. We also gathered closed-form survey results from 19 experts with >10 publications on SG. Both expert groups supported greatly increased research, recommending SG research funding of ~5% on average (10th–90th percentile range was 1–10%) of climate science budgets compared to actual budgets of <0.3% in 2018. Climate experts chose far less SG deployment in future climate policies than did SG experts.

Fuzzy-Based Multivariate Analysis for Input Modeling of Risk Assessment in Wind Farm Projects

Currently, input modeling for Monte Carlo simulation (MSC) is performed either by fitting a probability distribution to historical data or using expert elicitation methods when historical data are limited. These approaches, however, are not suitable for wind farm construction, where—although lacking in historical data—large amounts of subjective knowledge describing the impacts of risk factors are available. Existing approaches are also limited by their inability to consider a risk factor’s impact on cost and schedule as dependent. This paper is proposing a methodology to enhance input modeling in Monte Carlo risk assessment of wind farm projects based on fuzzy set theory and multivariate modeling. In the proposed method, subjective expert knowledge is quantified using fuzzy logic and is used to determine the parameters of a marginal generalized Beta distribution. Then, the correlation between the cost and schedule impact is determined and fit jointly into a bivariate distribution using copulas. To evaluate the feasibility of the proposed methodology and to demonstrate its main features, the method was applied to an illustrative case study, and sensitivity analysis and face validation were used to evaluate the method. The results demonstrated that the proposed approach provides a reliable method for enhancing input modeling in Monte Carlo simulation (MCS).