Measurement issues in the evaluation of projects in a project portfolio

Abstract

A common problem arising in many domains is how value the benefits of projects in a project portfolio. Recently, there has been some attention given to a decision analysis practice whereby analysts define a value function on some criterion by setting 0 as the value of the worst project. In particular, Clemen and Smith have argued this practice is not sound as it gives different results from the case where projects are “priced out”, and makes a strong implicit assumption about the value of not doing a project. In this paper we underscore the criticism of this way of using value functions by showing it can lead to a rank reversal. We provide a measurement theoretic account of the phenomenon, showing that the problem arises from using evaluating projects on an interval scale (such as a value scale) whereas to guard against such rank reversals, benefits must be measured on at least a ratio scale. Seen from this perspective, we discuss how the solution proposed by Clemen and Smith, of reformulating the underlying optimisation problem to allow for explicitly non-zero values, addresses the issue, and explore in what sense it may be open to similar problems. In closing we discuss what lessons from practice may be drawn from this analysis, focussing on settings where the Clemen and Smith proposal may not be the most natural way of modelling.