Comparing Expert, Quasi-Expert, and Novice Evaluations of Award-Winning Design Products Using the Consensual Assessment Technique

Abstract

ABSTRACT The consensus-based assessment has long been a prevalent methodology employed in “panel crit” settings in design education and professional design awards. Acknowledging the subjective nature of design evaluation and its importance to the design development process, we report two studies investigating the effect of three levels of design expertise (expert, quasi-expert, and novice) on the evaluation of award-winning lamp designs using the consensual assessment technique (CAT). The pilot study (N = 30) preliminarily examined expertise-related differences in evaluation outcome (i.e., ratings and rankings) and the use of the CAT instrument. In a follow-up study (N = 135), we tested four hypotheses using the refined CAT instrument. Results showed a significant interaction effect between expertise group and design on evaluation. Correlations between groups showed that quasi-experts were more similar to novices than experts. This finding contributes to the discussion on whether quasi-experts should be considered proper substitutes for experts. Furthermore, the reliability of ratings on individual CAT scale items varied among groups, suggesting that the degree to which raters were consistent within a group depends upon which aspect of design quality they were assessing. Taken together, the findings provide valuable insights into the influence of design education and experience on judging design quality.