A study of probabilistic information retrieval systems in the case of inconsistent expert judgments

Abstract

The maximum entropy principle may be applied to the design of probabilistic retrieval systems. When there are inconsistent expert judgments, the resulting optimization problem cannot be solved. The inconsistency of the expert judgments can be revealed by solving a linear programming formulation. In the case of inconsistent judgment, four plausible schemes are proposed in order to find revised judgments which are consistent with the true data structure but still reflect the original expert judgment. These schemes are the Interactive, Minimum Distance, Minimum Cross-Entropy, and Path methods. Background and Purpose of the Study The maximum entropy principle (MEP) based on Shannon’s measure (Shannon, 1928) has been used with great success in many areas. Cooper and Huizinga (1982) and Cooper (1983) have applied the MEP to the design of probabilistic information retrieval systems. Specifically, we consider a collection of documents which are categorized into Boolean components by attributes. The MEP estimates the probability of “relevance” of each Boolean component by integrating expert judgments about the “relevance” of attributes with the observed distribution of the Boolean components. The MEP retrieval system, in response to a user’s request, provides an ordering of the Boolean components using this estimated probability of “relevance.” Cooper (1983) has noted the potential of the MEP retrieval system as follows