Informed Selection of Filter Examples for Knowledge Refinement

Abstract

Refinement tools aim to incrementally modify knowledge based systems (KBSs) by identifying and repairing faults that are indicated by training examples for which the KBS gives an incorrect solution. These tools generally employ greedy hill climbing to search the space of possible refinements. Typically refinement algorithms are iterative and at each iteration chooses a fix having the best impact on the faulty KBS. This impact is ascertained by an accuracy measure taken over a subset of training examples. An informed selection of examples will help direct the search to useful areas of the refinement search space thus reducing the need to backtrack to previous refinement states. Therefore the availability of a representative set of examples is important for refinement tools. However, in real environments it is often difficult to obtain a large set of examples since each problem-solving task must be labelled with the expert’s solution. Even if a large set is available a careful selection of examples will help reduce computational costs. This paper investigates clustering and committee based approaches as a means to select a representative set of examples upon which an accuracy measure can be based. Of those selected only the subset of unlabelled examples requires to be labelled. Experiments in two domains show a reduction in the number of times previous refinements states need to be re-visited. Moreover, this reduction is possible without affecting the accuracy of the final refined KBS.