Validation of expert systems for innovation management: Issues, methodology, and empirical assessment

Abstract

Faced with the complexities of managing new product development, most of us would welcome the support of a computer-based system that captures the knowledge and the reasoning capabilities of experts in our field. Considerable effort has been focused on the design and development of expert systems for applications such as new product management. However, design and development are only two steps on the path to successful implementation of a useful expert system. A rigorous validation process is essential for ensuring that the expert system performs as intended. Using the INNOVATOR expert system as an example, Sundaresan Ram and Sudha Ram propose and test a framework for validating expert systems designed for new product management. The proposed validation framework considers three aspects of the expert system: its knowledge acquisition methodology, its performance, and its utility. Validation of an expert system's knowledge acquisition methodology involves assessment of the knowledge sources used, the criteria for selecting human experts, and the methods used for knowledge acquisition. Using multiple sources improves the likelihood that the expert system will capture the necessary core knowledge. Similarly, selection of the experts who are to supply the knowledge used by the expert system should be based on reliable measures of new product expertise rather than ad hoc measures. The system's performance is evaluated through formal tests of the accuracy and the completeness of the knowledge base, the consistency and the accuracy of the decisions made by the system, and the reasoning process by which the system reaches its decisions. Such tests may involve direct examination of the system by experts, and Turing tests, which compare both the recommendations and the reasoning process of the system with those of selected experts. Both types of tests may involve experts from whom knowledge was acquired during the development of the system as well as experts who were not involved in the design and development of the system. Assessment of an expert system's utility focuses on user perceptions of system performance and utility as well as the design of the user interface. First, end-users must evaluate the relevance of the chosen problem domain. In other words, the validation process must verify that the expert system addresses an important problem that requires decision support tools. Second, the expert system must provide a logical, systematic approach to solving the problem. Finally, the expert system must provide a consistent, intuitive user interface.