Підходи до оцінки ефективності експертних методів

Abstract

It is shown that definition of accuracy of expert methods represents a significant problem due to unavailability of benchmark values of object estimates in real expert examinations. It is suggested to use the efficiency indicator based on the stability of results of aggregation methods under deviations of input data. It has been studied two approaches to expert method efficiency evaluation on the example of comparison of two modifications of combinatorial aggregation method (in one modification spanning tree weights are taken into consideration, while in the other they are not). The first approach is based on analysis of real data of expert estimation of special benchmark objects, while the second one is based on simulation of the whole expert examination cycle, including the estimates themselves. In the process of efficiency evaluation of the two specified methods, the simulation-based approach turned out to be more suitable and representative. The obtained experimental results empirically prove the advantage of combinatorial method, taking spanning tree weights into account, over the method where these weights are not considered (and, consequently, over row geometric mean and logarithmic least squares methods). They also allow us to draw several conceptual conclusions, such as:1) Accuracy of expert estimates and efficiency of expert data aggregation methods are not the same concept. Often low accuracy of expert data aggregation methods results from mistakes made by experts during estimation and not from the drawbacks of the method itself;2) Not only accuracy, but also consistency of expert estimates is important, both within the pair-wise comparison matrix provided by a single expert, and among pair-wise comparisons provided by an expert group; the more consistent the data is, the more credibility it deserves;3) Only conceptually different methods, using different input data, can be compared based on real expert estimates. If the input data is the same and only the methods of their aggregation are different, then in order to evaluate such methods, one should simulate the whole expert examination cycle.Tabl.: 5. Fig.: 6. Refs: 24 titles.