An Experimental Study on the Influence That Failure Number, Specialization, and Controls Have on Confidence in Predicting System Failures1

Abstract

Design reviews are typically used for three types of design activities: (1) identifying errors, (2) assessing the impact of the errors, and (3) suggesting solutions for the errors. This experimental study focuses on understanding the second issue as it relates to the number of errors considered, the existence of controls, and the level of domain familiarity of the assessor. A set of design failures and associated controls developed for a completed industry sponsored project is used as the experimental design problem. Nondomain generalists (students from an undergraduate psychology class), domain generalists (first year engineering students), and domain specialists (graduate mechanical engineering students) are provided a set of failure modes and asked to provide their own opinion or confidence on whether the system would still successfully achieve the stated objectives. The confidence level for all domain populations decreased significantly as the number of design errors increased (largest p-value = 0.0793), and this decrease in confidence is more significant as the number of design errors increases. The impact on confidence is lower when solutions (controls) are provided to prevent the errors (largest p-value = 0.0334) as the confidence decreased faster for domain general engineers as compared to domain specialists (p = < 0.0001). The domain specialists showed higher confidence in making decisions than domain generalists and nondomain generalists as the design errors increase.