A Case Study and Field Evaluation of EDP-XPERT

Abstract

This paper reports on the continued evaluation of EDP-XPERT, and expert system for assisting Computer Audit Specialists CASs in evaluating the reliability of EDP controls in advanced computer environments. The current evaluation of the system involved four phases. First, Senior CASs used the system on two case studies. Second, it was then used by the Senior CASs to evaluate recently completed audits. Third, a sensitivity analysis of the system's output was conducted. Fourth, the knowledge base was reprogrammed into another expert system shell and the Senior CASs' responses on the two cases and the audits were re-run. The results indicated that EDP-XPERT performed well on the case studies but that performance declined on the audits. The sensitivity analysis showed that EDP-XPERT's conclusions were affected by moderate perturbations of the CASs' input judgments. Finally, reprogramming the knowledge base into a more flexible expert system shell substantially improved the performance of the system's judgements.