Studies on intrinsic summary evaluation

Abstract

Summary evaluation has gained importance in the research community and seems to be a quite challenging task. Most of the work till date focus on summary generation, while this paper focus on intrinsic summary evaluation. Studies made for preparing the 'Gold Standard' reference summary present some of the key issues in preparing a reference summary. Spearman, Kendall's W, and Kendall's rank correlation coefficients have been used for measuring inter-judge rank correlation. Existing drawbacks in Precision-Recall based evaluation and utility-based evaluation schemes were analysed. A new measure, called 'Effectiveness', to measure the system performance at given compression rate has been introduced.