A CAT with caveats: is the Consensual Assessment Technique a reliable measure of graphic design creativity?

Abstract

The Consensual Assessment Technique (CAT) is considered one of the gold standards for creativity assessment, and graphic design, arguably, is the most ubiquitous domain within the creative industries. For the first time, this study tests two tasks to measure graphic design creativity, and by extension, the reliability of the CAT as a measure of graphic design creativity. Initial research suggested the level of consensus amongst judges (often referred to as inter-rater reliability) was too low to be reliable, and may be unduly influenced by a judge’s preference for technical execution. In this study, 16 professional graphic designers were randomly assigned instructions to discount technical execution from creativity ratings, or given instruction that gave no stipulation, for 60 artworks. Inter-rater reliability scores were acceptable for each task and experimental condition, but were higher for judges that received instructions to discount technical execution. These and other results are discussed, and the argument presented that, for future CAT studies in this domain, specific instructions to discount technical execution offers a more reliable measure of graphic design creativity.