In Search of Effective Design Problems for Design Research

Abstract

Much design theory research seeks to create, evaluate, improve or optimize design methods. Whether that research focuses on design thinking, tools, methods, or education, short design problems are often provided to participants in order to evaluate the effects of the variables being tested. When designing and creating such problems, certain characteristics may influence design outcomes: experience and exposure to the design problems vary between participants, and each problem may be more or less favorable to the controlled variable. In this paper we conjecture a small set of design problem characteristics that may influence experimental outcomes, and we discuss two experiments targeted at uncovering this influence. In our first experiment we examine differences in evaluation metrics between two design problems. In a follow up experiment we correlate the hypothesized characteristics to the variances in experiment outcome. These early results assist to further compare and contrast the empirical differences in common evaluation metrics, as well as show how familiarity and extent of the subjects’ knowledge of a design problem influence these metrics. We also expose the potential for interaction between the design method and the design problem.