User Study: Influence of Number of Design Errors on Ability to Predict Performance With and Without Controls

Abstract

Design reviews are typically used for three types of design activities: 1) identifying errors, 2) assessing the impact of the errors, and 3) suggesting solutions for the errors. This experimental study focuses on understanding the second issue as it relates to the number of errors considered, the existence of controls, and the level of domain familiarity of the assessor. A set of design failures and associated controls developed for a completed industry sponsored project is used as the experimental design problem. Non-domain individuals (students from an undergraduate psychology class), domain generalists (first year engineering students), and domain-specialists (graduate mechanical engineering students) are provided a set of failure modes and asked to estimate the likelihood that the system would still successfully achieve the stated objectives. Primary results from the study include the following: the confidence level for all domain population decreased significantly as the number of design errors increased (largest p-value = 0.0793) and this decrease in confidence is more significant as the design errors increase. The impact on confidence is less when solutions (controls) are provided to prevent the errors (largest p-value = 0.0334), the confidence decreased faster for domain general engineers as compared to domain specialists (p = <0.0001). The domain specialists showed higher confidence in making decisions than domain generals and non-domain generalists as the design errors increase.