Evaluation of the ACAP analysis method for process-based validation of textual and graphical design methods

Abstract

The targeted improvement of design methods requires validation studies to record and evaluate difficulties in the application of the investigated methods. Current analysis methods for the validation of design methods are limited to the collection and evaluation of the design outcome and do not consider the applicability of the design method. In this paper, the authors evaluate an analysis method which, in addition to the benefits of a design method, also detects difficulties in its applicability. The Attention-Comprehension-Application-Performance analysis (ACAP analysis) method has been newly developed for this purpose. It investigates the applicability and captures metrics attention, comprehension, and correctness of application as well as the benefit of the design method. To evaluate the ACAP analysis method, a method for sheet metal design was examined in a laboratory study with 25 university students with mechanical engineering as their major subject. The results of the evaluation showed that the ACAP analysis method (1) identifies difficulties in the applicability of the design method; (2) determines the benefit of the design method; and (3) captures the impact of the identified difficulties on the benefit. Thus, the ACAP analysis method can evaluate causes of existing difficulties using objective metrics.