Validation in a knowledge support system: construing and consistency with multiple experts

Abstract

At the previous workshop on Knowledge Acquisition for Knowledge-Based Systems in 1986, criteria for a knowledge support system were discussed, and a preliminary version of KITTEN (Knowledge Initiation and Transfer Tools for Experts and Novices) was described and demonstrated on Apollo workstations. This study is a continuation of the validation studies done by Shaw & Gaines (1983), and investigates a framework for knowledge acquisition evaluation and validation. KITTEN has been evaluated against the first stage of the model and the results are reported in the two domains of spatial interpolation techniques to produce contour maps and in trouble-shooting and maintenance of valves for oil and gas pipelines. Some preliminary results are described on validation experiments to show the extent to which experts agree with each other, with themselves at a later date, and with the results of the processing of their knowledge. Some of the questions asked were: (1) To what extent does an expert find the generated rules meaningful? (2) Do experts agree on their terminology in talking about a topic? (3) To what extend do experts agree among themselves about the topic? (4) Does an expert always use the same terminology? (5) To what extent does each experts agree with the knowledge at a different time?