Subjectivity of novelty metrics based on idea decomposition

Abstract

ABSTRACT The novelty metric suggested by Shah and colleagues is one of the most widespread among the suggestions made by scholars, and it is based on the subjective identification of attributes and/or functions underpinning analyzed ideas. If not correctly managed, this subjectivity can lead to non-negligible ambiguity of assessments, which could potentially invalidate the research results. Several variants to this metric have been proposed in the last two decades, with some of them claiming to have improved the original metric. However, the related benefits and drawbacks are still unclear, especially in terms of subjectivity. The aim of this study is to estimate the potential misalignment between research teams that independently perform the assessment of the same set of ideas. To this purpose, the considered metrics have been applied to a set of 100 ideas by utilizing the assessment results from three independent evaluators. It was revealed that the obtained novelty scores can be extremely different owing to the plethora of different possible interpretations of the analyzed ideas. Accordingly, the results highlight that for the same set of ideas, very different novelty assessment rationales can be followed by the evaluators.