Of Brittleness and Bottlenecks: Challenges in the Creation of Pattern-Recognition and Expert-System Models

Abstract

As tools for the construction of expert systems have become commonly available, workers in artificial intelligence (AI) have begun to pay increasing attention to the problems of building and maintaining large knowledge bases. In particular, considerable discussion has concentrated on the difficulty of eliciting useful and complete knowledge about a given application task from experts—a problem widely referred to as the knowledgeacquisition bottleneck. At the same time, AI researchers have often noted that the systems that they build are brittle, showing marked degradation in reasoning performance when confronted with unusual or atypical cases. Most expert systems, like pattern-recognition systems, are concerned with the classification of entities in the world. The process of building knowledge bases for expert systems that perform classification, like the process of developing statistical classifiers, requires that someone who is familiar with the application area both determine the relevant types of objects in the domain and identify the observable features of those objects that may be germane to the classification problem. In the case of both expert systems and pattern-recognition systems, developers must create models of the application area. Although the pattern-recognition literature does not generally mention concepts such as “knowledge acquisition” or “brittleness,” these problems are also important in the construction of statistical models. This paper shows how builders of expert and patternrecognition systems face many of the same challenges, and discusses ways in which the two research communities can learn from each other's experiences in creating different types of computational models for classification tasks.