Expert Preferences Research Articles

Group DEMATEL Decision Method Based on Hesitant Fuzzy Linguistic Term Sets

Background: Decision-making trial and evaluation laboratory (DEMATEL) is a practical and concise method to deal with the complicated socioeconomic system problems. However, there are two defects in original DEMATEL. On the one hand the traditional expert preference expressions can’t reflect the hesitation and flexibility of expert, on the other hand the determination of group experts’ weight usually be expressed the equivalent weight which can’t reflect the scientificality of weight on the behalf of experts’ academic background, capability experience, risk preference and so on. To solve the above problems, a novel Group DEMATEL decision method based on hesitant fuzzy linguistic term sets (HFLTSs) is proposed. Method: Firstly, this paper presents that experts make their judgement on the causal relationship of factors by using a linguistic expression closed to human expression, which can be easily transformed into HFLTSs. Next the hybrid weight of experts are calculated on the base of the initial HFLTSs direct influence matrix (HDIM) according to the hesitant degree and distance between two HDIMs. And the aggregation of each expert’s information is introduced by possibility degree. Then the new group DEMATEL decision method based on HFLTSs are constructed. Finally, an illustrative example is given and analyzed to demonstrate the effectiveness and validation of the proposed approach. Results: This paper demonstrate the heterogeneity of decision experts and the hesitation degree of expert information representation must be taken into account when determining the interaction of factors in complex systems by DEMATEL method. Conclusion: This paper constructs the new amended group DEMATEL which provides a new way to deal with the integration of each expert’s information by the hybrid weight and possibility degree. The methods provides references for determining the importance of complex system factors more scientifically and objectively.

Supplier selection in the oil & gas industry: A comprehensive approach for Multi-Criteria Decision Analysis

The focus of this paper is on selecting suppliers in the Oil and Gas (O&G) industry by developing a comprehensive approach for Multi-Criteria Decision Analysis (MCDA) based on alternative methods capable of assessing different aspects of supplier selection uncertainty in terms of utility functions and criteria related to efficiency. The O&G industry has a key role in the public sector of various countries such as Iran with its revenues being of prime importance to develop infrastructure facilities such as for healthcare, education, and transportation. This comprehensive approach walks through various stages for selecting Critical Success Factors (CSFs), ranking suppliers, and for setting partial weighting alternatives. While CSFs are selected using a traditional Delphi approach, the partial supplier rankings are defined based on Complex Proportional Assessment (COPRAS) utility functions together with criteria weights derived from Step-wise Weight Assessment Ratio Analysis (SWARA) for each CSF. As it concerns information reliability of utility and efficiency functions of both methods obtained via expert preferences or perceptions, Z-numbers are used to address the intrinsic fuzziness level inherent to each analytical stage. Iran's economy depends on revenues from oil and other related production, which means that by earning more income from this industry, most of its economic indicators such as GDP and employment rate should increase significantly, thus leading to economic growth. Various countries put plans in place related to production for increasing their social economics. One of these plans is focused on suppliers since they have a high impact on providing essential items such as equipment, HR, and transportation, so by choosing the best suppliers in all fields, costs will decrease and consequently revenue will increase. This research points out how to rank O&G industry suppliers using MCDA methods in an uncertain environment. An example based on actual data from an Iranian O&G company is provided to show the applicability of the approach proposed. Results suggest that the complexity of O&G operations on selecting suppliers can be adequately handled by information reliability techniques applied to traditional economic concepts such as utility- and efficiency-related factors, particularly in business environments characterized by a trade embargo.

Evidential reasoning for preprocessing uncertain categorical data for trustworthy decisions: An application on healthcare and finance

The uncertainty attributed by discrepant data in AI-enabled decisions is a critical challenge in highly regulated domains such as health care and finance. Ambiguity and incompleteness due to missing values in output and input attributes, respectively, is ubiquitous in these domains. It could have an adverse impact on a certain unrepresented set of people in the training data without a developer’s intention to discriminate. The inherently non-numerical nature of categorical attributes than numerical attributes and the presence of incomplete and ambiguous categorical attributes in a dataset increases the uncertainty in decision-making. This paper addresses the challenges in handling categorical attributes as it is not addressed comprehensively in previous research. Three sources of uncertainties in categorical attributes are recognised in this research. The informational uncertainty, unforeseeable uncertainty in the decision task environment, and the uncertainty due to lack of pre-modelling explainability in categorical attributes are addressed in the proposed methodology on maximum likelihood evidential reasoning (MAKER). It can transform and impute incomplete and ambiguous categorical attributes into interpretable numerical features. It utilises a notion of weight and reliability to include subjective expert preference over a piece of evidence and the quality of evidence in a categorical attribute, respectively. The MAKER framework strives to integrate the recognised uncertainties in the transformed input data that allow a model to perceive data limitations during the training regime and acknowledge doubtful predictions by supporting trustworthy pre-modelling and post modelling explainability. The ability to handle uncertainty and its impact on explainability is demonstrated on a real-world healthcare and finance data for different missing data scenarios in three types of AI algorithms: deep-learning, tree-based, and rule-based model.

Extension of safety and critical effect analysis to neutrosophic sets for the evaluation of occupational risks

Occupational health and safety applications require a comprehensive risk analysis to protect employees from accidents caused by various hazard sources in the workplace. Therefore, in the literature, there are various risk assessment methods utilizing various parameters. One of them is safety and critical effect analysis (SCEA) which is the method that calculates risk magnitude based on the highest number of parameters that are probability, severity, frequency, and detectability. However, the risk assessment phase includes uncertainties and inconsistencies caused by both evaluations of experts and the complex structure of the handled process. Therefore, we proposed a new risk assessment approach to cope with these uncertainties and inconsistencies by extending SCEA with the Neutrosophic sets. In this study, each parameter, probability, severity, frequency, and detectability, is defined by using truth membership function, falsity membership function, and indeterminacy membership function in Neutrosophic sets (NSs) to deal with incomplete, indeterminate, and inconsistent information existing in the expert preferences and complex structure of the handled process. It is the first time, the indeterminacy of NSs is defined based on truth and falsity definitions by using a rule-based system based on probability, severity, frequency, and detectability parameters. Furthermore, Mamdani fuzzy inference system used in the SCEA method is adapted to NSs. The proposed approach is applied to the metalworking and woodworking workshop of a university. The obtained results are compared with the results of the SCEA method. It is concluded that the proposed approach is better than SCEA method to consider inconsistencies in the risk evaluation.

Using a Modified VIKOR Technique for Evaluating and Improving the National Healthcare System Quality

The effectiveness of the national/regional healthcare system is one of the keys to prevent the spread of COVID-19. In the face of this unknown pandemic, where the healthcare system should continue to be promoted and improved are crucial decision issues. In the past, most studies have used the subjective opinions of experts for analysis and decision-making processes when investigating complicated decision-making problems. However, such decision-making processes are easily influenced by experts’ preferences. Therefore, this research proposes a soft computing technology that integrates CRiteria Importance Through Intercriteria Correlation (CRITIC) with the modified VlseKriterijumska Optimizacija I Kompromisno Resenje in Serbian, meaning multicriteria optimization and compromise solution (modified VIKOR) technique to reduce the impact of expert preference. In order to cope with the fact that COVID-19 has spread globally and to discover problems quickly and effectively, this study uses the global health security (GHS) index as the evaluation framework and conducts overall discussions in 195 countries/regions around the world. It is verified that the technology of soft computing can be used for continuous promotion and improvement of the national/regional healthcare system. This technology facilitates decision makers to know the gap of performance between the current healthcare system and the aspiration level. Finally, based on these gaps, we provide management advice to help improve these systems.

A Fuzzy Analytic Hierarchical Method to Reduce Imprecision and Uncertainty in Drilling Operation’s Factor Selection Process for Unidirectional Carbon Fibre Reinforced Plastic Composite Plates

Parametric selection in machining processes is recently understood as a route to reducing waste generation in drilling activities and achieving a robust resource distribution in drilling activities. However, the selection methods dominant in the literature lack competence in reducing uncertainties and imprecision associated with the drilling process. The purpose of this research is to reduce the uncertainty and imprecision in previously analyzed data that used the analytic hierarchy process (AHP) method. This paper adjusts the uncertainty and imprecision by introducing a geometric mean-based fuzzy analytic hierarchy process. The selection method influences the drilling expert's preferences by imposing the fuzzy theory in a triangular member function that converts the crisp numerical values into fuzzy members and adequately suppresses the imprecision and uncertainty in the elements. The thrust force was positioned first in ranking with a FAHP method's weight of 0.415, which matched the literature value of 0.413 for the AHP method. It was found that the use of the FAHP method has corrected the imprecision and uncertainty introduced by the AHP method. It was found that the thrust force and torque were overestimated by or 0.48% and 3.95%, respectively and was accordingly corrected. Besides, no errors were found with the measurement of eccentricity response. Furthermore, the entry delamination, exit delamination and surface roughness were underestimated by -8.11%, -3.33% and -6.96%, respectively, and therefore corrected by the FAHP method. The usefulness of this effort is to enhance cost-effective decisions and the effectiveness in the distribution of scarce drilling resources.

Using argumentation in expert’s debate to analyze multi-criteria group decision making method results

Recent multi-criteria group decision making methods focus their analysis on the experts preferences. They do not take into account the reasons why each expert has provided a specific set of preferences. In this paper, a method that introduces novel measures capable of explaining the reasons behind experts decisions is presented. A novel concept, the arguments are presented. They represent the experts have for maintaining a certain position in the debate. Several measures related to the arguments are proposed. These new argumentation measures, along with consensus measures, help us to get a clear idea about how and why a specific resolution has been reached. They help us to determine which is the most influential expert, that is, the expert whose contributions to the debate have inspired the rest. Also, the proposed method allows us to determine which are the arguments that most of the experts have followed. A clear overview about how the debate is evolving in terms of arguments is also provided. The novel presented analysis indicate how the experts change their opinions in every round and what was the reason for it, which changes have occurred between rounds and they also provide global analysis results.

Experts’ multiple criteria evaluations of fuel management options to reduce wildfire susceptibility. The role of closer knowledge of the local socioeconomic context

Expert opinion can be a valuable tool for informed decision making. Concerning wildfire susceptibility reduction at the landscape scale, forest ecosystem experts play a key role in offering advice about appropriate fuel management practices to be applied by forest owners or their organizations, and in shaping public policies. A literature review aimed at identifying fuel management interventions and techniques found multiple and even opposing strategies. Recognizing the interdisciplinary and multi-dimensional nature of fuel management, we go beyond existing studies on forest experts’ opinions by comparing evaluations across forest experts with diverse training and experience, and by considering different evaluation criteria such as technical effectiveness, impact on soil or biodiversity, socioeconomic impact, and preference. Following an online survey to a sample of Portuguese experts, distinct socio-professional clusters were established and experts’ evaluations associated with their views on fire, forests, owners’ coordination, and rural development. Results show that experts rank their preferences by weighing effectiveness and impacts in different ways. Closer knowledge of the local context distinguishes expert preference, favouring more active fuels reduction strategies. Since experts with a closer knowledge of socioeconomic context tend to be further from policy-making processes, we urge their more balanced participation in those processes.

New Dynamic Group DEMATEL Decision-Making Method Based on Hesitant Fuzzy Linguistic Term Sets

Decision-making trial and evaluation laboratory (DEMATEL) method is comprehensively applied in the complicated socioeconomic system. Existing literatures on DEMATEL focus on static decision environment at single time node. However, due to the increasingly complex and rapidly changing socioeconomic environment, decision-making has become more complex and dynamic. In this paper, dynamic group DEMATEL method based on hesitant fuzzy linguistic term sets (HFLTSs) is proposed. In the view of the multi-stage, multi-factor and multi-expert complex decision-making situation, the method fully integrates the decision-making information of different periods from the perspective of dynamic evolution mechanism which can effectively link multi rounds DEMATEL decision-making process and reflect the time evolution of dynamic decision-making. Meanwhile, the TOPSIS method is applied to determine the weight of experts, which can effectively solve the influence of the best choice (positive ideal solution) and the worst choice (negative ideal solution) on the decision results in aggregation of group experts' preference information, and fully consider the influence of hesitation and fuzziness of expert preference representation on expert weight. Finally, a numerical comparison is given to verify validity and effectiveness of the proposed approach.

A Proposed Method for Multi-Criteria Group Decision Making: An Application to Site Selection

ABSTRACT In the real world, decision-making process is related to alternative evaluation with respect to multiple conflicting criteria. In this paper, a hybrid group approach to solve the site selection problem is developed by integrating two methods which are the Best Worst Method (BWM) and Evaluation based on Distance from Average (EDAS). The multi-criteria group decision-making methods aggregate expert preferences and present the best agreement. BWM method is used to do pairwise comparisons in a structured way to determine the criteria weight. EDAS is a method used for alternative ranking which is useful for the decision issues that contain conflicting criteria. In order to show the procedures and the application of the developed method, a case study for site selection of plastic manufacturing company is developed. A comparison made between the proposed method and the AHP method is developed to ensure that the proposed method offers reliable results. Also, sensitivity analysis is conveyed for robustness validation.

Uncertainty and Tourism Consumption Preferences: Evidence from the Representative Chinese City of Shenzhen

As the fastest growing city in China, Shenzhen, a pioneer of reform and opening up, seems to have exhausted its advantages of geographical convenience, cheap land, and cheap labor; thus, the tourism industry has been adopted as a new economic strategy. However, the Shenzhen tourism market has received little attention from scholars and few studies have been done to investigate the relationships between uncertainty and tourism consumption preferences in different cultures. Therefore, this paper attempts to study Shenzhen as a travel destination to explore the consumption preferences of domestic and international tourists, specify uncertainties in tourism consumption activities, and examine their impacts on preferences. The above aims are achieved using a survey method and a new theoretically proposed preference uncertainty model inspired by combining the modified virtual expert preference approach and the ordered probit model (MVEP-OPM), in which three major components of tourism consumption (food, accommodation, and shopping) are estimated. (1) The results show that there are significant differences in tourism consumption preferences between domestic tourists and international visitors. (2) Inexperience, unfamiliarity, imperfect knowledge, and policy uncertainty are major uncertainties in tourism consumption activities, which affect tourist preferences toward consumption behavior. (3) Uncertainty plays a different moderating role in product-related influential factors, such as sanitation, safety, and decoration.

Perception of Subject Matter Experts (SMEs) on Various Curation and Preservation Techniques for the Development of Natural History Gallery in Malaysia: A Qualitative Study

Preserving animal’s body or part of the animals have become an important and well-known conservation medium to ensure that the future generations are still able to enjoy the biodiversity and all the incredible species that live within it. This paper aims to present a qualitative study on perception and preference of subject matter experts (SMEs) on various curation and preservation methods that is suitable for certain animals for the development of Natural History Gallery. Interview with the SMEs were done by using semi-structured questionnaire and prior to this, permission were gained from the respondents by signing the Informed Consent Form (ICF). The findings suggest that the preservation methods of specimens ultimately depend on the purpose of specimens collected, either to be preserved as research specimens or exhibition specimens. The information gained form this study will be valuable as directive for the development of natural history gallery in Malaysia.

Interval Intuitionistic Fuzzy Decision Model with Abnormal Information and Its Application in Talent Selection

In some complex decision-making problems such as talent selection, experts often hesitate between multiple evaluation values during their decision making and can only give a range of information due to the fuzziness and imprecision of qualitative decision-making attributes. Interval intuitionistic fuzzy sets and their decision-making methods provide a useful tool to describe the fuzziness of decision attributes and decision experts’ hesitation. However, the abnormal information in the expert decision information has not been considered in the previous works; that is, some interval intuitionistic fuzzy numbers exceed the defined interval range. This kind of abnormal decision information often makes it difficult to obtain accurate decision results using the decision model. To avoid the abnormal information influence on decision-making results, the hesitancy degree-based interval intuitionistic fuzzy sets are employed to propose an adaptive correction method of abnormal information, which can correct the abnormal decision information without changing the decision preference of experts. The abnormal information correction method is utilized to construct a new interval intuitionistic fuzzy entropy by combining hesitancy and fuzziness. This provides a multiattribute decision-making method, including abnormal decision information. Finally, the effectiveness and superiority of the proposed method and decision-making model are evaluated using an application case study of talent selection.

Method of using the transitive graph of a Markovian process as part of ranking of heterogeneous items

Hierarchy analysis developed by Thomas Saaty is a closed logical structure that uses simple and well-substantiated rules that allow solving multicriterial problems that include both quantitative, and qualitative factors, whereby the quantitative factors can differ in terms of their dimensionality. The method is based on problem decomposition and its representation as a hierarchical arrangement, which allows including into such hierarchy all the knowledge the decision-maker has regarding the problem at hand and subsequent processing of decision-makers’ judgements. As the result, the relative degree of the interaction between the elements of such hierarchy can be identified and later quantified. Hierarchy analysis includes the procedure of multiple judgement synthesis, criteria priority definition and rating of the compared alternatives. The method’s significant limitation consists in the requirement of coherence of pairwise comparison matrices for correct definition of the weights of compared alternatives. The Aim of the paper is to examine a non-conventional method of solving the problem of alternative ratings estimation based on their pairwise comparisons that arises in the process of expert preference analysis in various fields of research. Approaches are discussed to the generation of pairwise comparison matrices taking into consideration the problem of coherence of such matrices and expert competence estimation. Method. The methods of hierarchy analysis, models and methods of the Markovian process theory were used. Result. The paper suggested a method of using the transitive graph of a Markovian process as part of expert ranking of items of a certain parent entity subject to the competence and qualification of the experts involved in the pairwise comparison. It is proposed to use steady-state probabilities of a Markovian process as the correlation of priorities (weights) of the compared items. The paper sets forth an algorithm for constructing the final scale of comparison taking into consideration the experts’ level of competence. Conclusion. The decision procedures, in which the experts are expected to choose the best alternatives out of the allowable set, are quite frequently used in a variety of fields for the purpose of estimation and objective priority definition, etc. The described method can be applied not only for comparing items, but also for solving more complicated problems of expert group estimation, i.e., planning and management, prediction, etc. The use of the method contributes to the objectivity of analysis, when comparing alternatives, taking into consideration various aspects of their consequences, as well as the decision-maker’s attitude to such consequences. The suggested model-based approach allows the decision-maker identifying and adjusting his/her preferences and, consequently, choosing the decisions according to such preferences, avoiding logical errors in long and complex reasoning chains. This approach can be used in group decision-making, description of the procedures that compensate a specific expert’s insufficient knowledge by using information provided by the other experts.

Managing for Stakeholders Using Multiple-Criteria Decision-Making Techniques

Managing for shareholders means maximising the market value of a company by increasing their returns. Managing for stakeholders means simultaneously creating value for multiple parties, such as employees, government, suppliers, customers, the environment and even society as a whole, which requires a multiple-criteria decision-making approach. This paper develops social indicators to measure the value that a company has distributed to stakeholders since the foundation of the company, and then calculates its relative efficiency using data envelopment analysis (DEA). The accumulated value of the investments made by the shareholders, the subsidies received and the loans borrowed are proposed as inputs. The accumulated value of taxes paid by the company, personnel payments, interest payments to creditors, and purchases from suppliers are proposed as outputs. All values are adjusted for the time value of money at the valuation date. We chose a weighted DEA model because not all stakeholders are worth the same, obtaining the weights from the preferences of social investment experts using a multi-attribute decision-making technique. A practical case is presented, obtaining the relative efficiency of a sample of companies in creating value for five years since their foundation. We found a positive relationship between efficiency in generating value for shareholders and efficiency in generating value for some non-shareholders, although the results are sensitive to the outputs chosen. The proposed method is useful for social investors when they invest or for public administrations when they provide subsidy to entrepreneurs.

Determining Importance of Many-Objective Optimisation Competitive Algorithms Evaluation Criteria Based on a Novel Fuzzy-Weighted Zero-Inconsistency Method

Along with the developments of numerous MaOO algorithms in the last decades, comparing the performance of MaOO algorithms with one another is also highly needed. Many studies have attempted to manipulate such comparison to analyze the performance quality of MaOO. In such cases, the weight of importance is critical for evaluating the performance of MaOO algorithms. All evaluation studies for MaOO algorithms have ignored to assign such weight for the target criteria during evaluation process, which plays a key role in the final decision results. Therefore, the weight value of each criterion must be determined to guarantee the accuracy of results in the evaluation process. Multicriteria decision-making (MCDM) methods are extremely preferred in solving weighting issues in the evaluation process of MaOO algorithms. Several studies in MCDM have proposed competitive weighting methods. However, these methods suffer from inconsistency issues arising from the high subjectivity of pairwise comparison. The inconsistency rate increases in an exorbitant manner when the number of criteria increases, and the final results are affected. The primary objective of this study is to propose a new method, called a Novel Fuzzy-Weighted Zero-Inconsistency (FWZIC) Method which can determine the weight coefficients of criteria with zero consistency. This method depends on differences in the preference of experts per criterion to compute its significance level in the decision-making process. The proposed FWZIC method comprises five phases for determining the weights of the evaluation criteria: (1) the set of evaluation criteria is explored and defined, (2) the structured expert judgement (SEJ) is used, (3) the expert decision matrix (EDM) is built on the basis of the crossover of criteria and SEJ, (4) a fuzzy membership function is applied to the result of the EDM and (5) the final values of the weight coefficients of the evaluation criteria are computed. The proposed method is applied to the evaluation criteria of MaOO competitive algorithms. The case study consists of more than 50 items distributed amongst the major criteria, subcriteria and indicators. The significant contribution of each item to the algorithm evaluation is determined. Results show that the criteria, subcriteria and their related indicators are weighted without inconsistency. The findings clearly show that the FWZIC method can deal with the inconsistency issue and provide accurate weight values to each criterion.

The Effects of the Qualitative Parameters of Internal Auditing Works on the Preferences of Investment Funds’ Analysts

This article documents the direct relation between the internal auditing reports that are revealed to the investors with facts of fraud in an entity and investor confidence for investing such companies. The contents of the internal auditing reports, which have been prepared specifically in response to the fraud signal in a company, provide evidence that there is a considerable correlation. This paper argues that the observed correlation fluctuates depending on the contents, independence, and objectivity of an internal auditing report. The strength of the established correlation varies according to the provided confidence to investors. With the said confidence, the investors decide to retain their investments in a company, to invest new funds in a company, not to invest any funds, or to withdraw invested funds from a company. The flow of the preferences of investment fund experts is affected: (i) If the internal auditing process and report is concluded by experienced auditors; (ii) If the internal auditing work is processed objectively; (iii) If the internal auditors act independently during the auditing process.

Identification of key sugarcane harvester features using the Analytic Hierarchy Process

The Philippine sugarcane industry is worth roughly 88 billion pesos and employs around 700,000 Filipinos. However, the competitiveness of the local industry ($0.17 per lb sugar) is currently under threat from international producers ($0.14 per lb sugar) because of the upcoming ASEAN 2015 integration. One of the major blockages to cost effectiveness is the low productivity of the local industry. This may be addressed with the use of a semi-automated mechanical harvester. As a preliminary study, the present work captures the expert preferences of various stakeholders in the selection of a semi-automated mechanical harvester. This is done using the Analytic Hierarchy Process (AHP) surveying tool. The study assumes the following important factors: a) Ease of use, b) Productivity, c) Adaptability to different terrains, d) with Lifter, and e) with Topper. Using a survey with a series of pairwise comparison questions, it is able to quantify even qualitative factors affecting decision making. The priorities of the surveyed stakeholders are averaged and discussed in the results section. Widely used in policy making and planning, the paper introduces the use of the AHP tool to commercial machinery design. The model may be expanded and improved to consider other key factors.

Evaluation of Online Teaching Quality: An Extended Linguistic MAGDM Framework Based on Risk Preferences and Unknown Weight Information

Evaluation of online teaching quality has become an important issue because many universities are turning to online classes due to the Corona Virus Disease 2019 (COVID-19) pandemic. In this paper, online teaching quality evaluation is considered as a linguistic multi-attribute group decision-making (MAGDM) problem. Generally, the evaluation sematic information can be symmetrically or asymmetrically distributed in linguistic term sets. Thus, an extended linguistic MAGDM framework is proposed for evaluating online teaching quality. As the main contribution, the proposed method takes into account the risk preferences of assessment experts (AEs) and unknown weight information of attributes and sub-attributes. To be specific, the Delphi method is employed to establish a multi-level evaluation indicator system (EIS) of online teaching quality. Then, by introducing the group generalized linguistic term set (GLTS) with two risk preference parameters, a two-stage optimization model is developed to calculate the weights of attributes and sub-attributes objectively. Subsequently, the linguistic MAGDM framework was divided into two stages. The first stage maximizes the group comprehensive rating values of teachers on different attributes to obtain partial ranking results for teachers on each attribute. The latter stage maximizes the group comprehensive rating values of teachers to evaluate the overall quality. Finally, a case study is provided to illustrate how to apply the framework to evaluate online teaching quality.

Evaluating and Comparing Ontology Alignment Systems: An Mcdm Approach

Ontology alignment is vital in Semantic Web technologies with numerous applications in diverse disciplines. Due to diversity and abundance of ontology alignment systems, a proper evaluation can portray the evolution of ontology alignment and depicts the efficiency of a system for a particular domain. Evaluation can help system designers recognize the strength and shortcomings of their systems, and aid application developers to select a proper alignment system. This article presents a new evaluation and comparison methodology based on multiple performance met-rics that accommodates experts' preferences via a multi-criteria decision-making (MCDM) method, i.e., Bayesian best-worst method (BWM). First, the importance of a performance metric for a specific task/application is determined according to experts' preferences. The alignment systems are then evaluated based on proposed expert-based collective performance (ECP) that takes into account multiple metrics as well as their calibrated importance. For comparison , the alignment systems are ranked based on a probabilistic scheme, where it includes the extent to which one alignment system is preferred over another. The proposed methodology is applied to six tracks from ontology alignment evaluation initiative (OAEI), where the importance of performance metrics are calibrated by designing a survey and eliciting the preferences of ontology alignment experts. Accordingly, the participating alignment systems in the OAEI 2018 are evaluated and ranked. While the proposed methodology is applied to six OAEI tracks to demonstrate its applicability, it can also be applied to any benchmark or application of ontology alignment.