Визначення відносної компетентності експертів під час агрегації парних порівнянь

Abstract

A method for defining the relative competence of expert group members in the process of aggregation of pair-wise comparisons is suggested. It is shown that in order to define the relative competence of experts, who provide their estimates in the form of pair-wise comparison matrices, it is not enough to consider just self-estimate, mutual estimate, and the so-called objective component of the indicator. It is assumed that judgments of an expert, who is more consistent in his estimates, and provides more complete and detailed information (choosing the scales with larger number of grades), should be assigned larger weights. The suggested approach allows us to consider consistency, compatibility and completeness of pair-wise comparisons as well as the level of detail of estimation scales (in addition to the three aforementioned components) while calculating the relative expert competence.Four conceptual levels of expert competence are described (from top to bottom): subject domain, specific problem, specific pair-wise comparison matrix, and specific pair-wise comparison. On the subject domain and the problem levels we suggest using any of the previously-developed approaches for objective estimation, self-estimation and mutual estimation of expert competence levels, including pair-wise comparison-based methods. On the level of a specific pair-wise matrix, provided by an expert, we suggest assigning it a rating, reflecting its inner consistency and compatibility with pair-wise comparison matrices, provided by other experts. Finally, on the level of each pair-wise comparison, we suggest assigning it a weight, depending on the number of grades in the scale, in which it has been provided.The suggested approach allows us to improve the aggregation procedure, used in the combinatorial pair-wise comparison aggregation method (sometimes called enumeration of all spanning trees).Obtained experimental results confirm that all different aspects of expert competence should be taken into consideration during expert examinations, particularly when completeness, consistency and compatibility levels of estimates vary across expert group members, and when experts are using scales with different numbers of grades.On the whole, obtained results allow us to improve the existing pair-wise comparison aggregation methods and increase the credibility of results of group expert examinations, using pair-wise comparisons.