Validating rule-based systems: a complete methodology

Abstract

This paper describes a complete methodology for the validation of rule-based expert systems. The methodology is presented as a 5-step process that has three central themes: creation of a minimal set of test inputs that adequately cover the domain represented in the knowledge base; a Turing test-like methodology that evaluates the system's responses to the test inputs and compares them to the responses of human experts; and use of the validation results for system improvement. The development of a minimal set of test inputs takes into consideration various criteria, both user-defined and domain-specific. These criteria are used to reduce the potentially very large exhaustive set of test inputs to one that is practical. The Turing test-like evaluation methodology makes use of a panel of experts to both evaluate each set of test cases and compare the results with those of the expert system, as well as with those of the other experts in the validation panel. The hypothesis being presented is that much can be learned about the experts themselves by having them evaluate each other's responses to the same test inputs anonymously. Thus, by carefully scrutinizing the results of each expert in relation to the other experts, we are better able to judge an evaluator's expertise, and consequently, better determine the validity of an expert system.