Empirical Comparisons of Knowledge Acquisition Techniques

Abstract

The degree to which a knowledge-acquisition technique (KAT) extracts useful information from a human expert varies, in large part, as a function of the type of expert system being developed. The present study employed two KATs, ARK (a structured-interview technique) and Repertory Grid (a similarity-judgment technique), to elicit knowledge from U.S. Army helicopter pilots with expertise in mission planning. The resultant knowledge bases were compared in terms of their applicability to two tasks: attack route evaluation and attack route planning. The study revealed important qualitative and quantitative differences in the knowledge bases elicited by the two KATs. The Repertory Grid method elicited dimensions with which the efficiency of attack routes could be classified and, in general, appears to be well-suited for use in development of “convergent” expert systems intended to perform tasks involving categorization or evaluation. In contrast, ARK elicited procedural information (e.g., goals, strategies, and rules) with which route planning could be performed, and appears better suited for obtaining information to be used in the development of “divergent” expert systems designed to conduct tasks involving planning or scheduling. The results have implications for the selection of KATS and the design of knowledge-elicitation sessions, and suggest that further analyses of the knowledge-acquisition process be conducted.