When is the probability ranking principle suboptimal?

Abstract

The probability ranking principle retrieves documents in decreasing order of their predictive probabilities of relevance. Gordon and Lenk (1991) demonstrated that this principal is optimal within a signal detection—decision theory framework, and it maximizes the inquirer's expected utility for relevant documents. These results hold under three conditions: calibration, independent assessment of relevance by the inquirer, and certainty about the computed probabilities of relevance. We demonstrate that the probability ranking principle can be suboptimal with respect to expected utility when one of these conditions fails to hold. © 1992 John Wiley & Sons, Inc.