Is the Reliability of Objective Originality Scores Confounded by Elaboration?

Abstract

ABSTRACT In recognition of the capability of text-mining models to quantify aspects of language use, some creativity researchers have adopted text-mining models as a mechanism to objectively and efficiently score the Originality of open-ended responses to verbal divergent thinking tasks. With the increasing use of text-mining models in divergent thinking research, concerns have been raised about how text-mining-based Originality estimates can be confounded by other dimensions of divergent thinking, especially Elaboration. Since automated Originality estimates can be influenced by varying amounts of Elaboration, or the number of words a participant uses in a response, some researchers question whether Originality scores are psychometrically valid or if the reliability of Originality scores is dependent on the variance in Elaboration. Using partial correlation procedures, we investigate whether Originality scores generated by a freely available text-mining system are significantly influenced by the amount of Elaboration a participant exhibits in their response to a divergent thinking task. Then, the reliability of Originality scores, before and after the variance accounted for by Elaboration is partialled out, are compared. Results from this brief analysis reveal that, when recent methodological recommendations for automatic Originality scoring are applied, Originality scores generated via the GloVe 840B text-mining system are not meaningfully confounded by Elaboration. We conclude that, even when the variance attributed to Elaboration is partialled out, this scoring method is capable of producing reliable Originality scores.