Entailment-based Fully Automatic Technique for Evaluation of Summaries

Abstract

We propose a fully automatic technique for evaluating text summaries without the need to prepare the gold standard summaries man- ually. A standard and popular summary evaluation techniques or tools are not fully automatic; they all need some manual process or manual reference summary. Using recognizing textual entailment (TE), automat- ically generated summaries can be evaluated completely automatically without any manual preparation process. We use a TE system based on a combination of lexical entailment module, lexical distance module, Chunk module, Named Entity module and syntactic text entailment (TE) module. The documents are used as text (T) and summary of these doc- uments are taken as hypothesis (H). Therefore, the more information of the document is entailed by its summary the better the summary. Com- paring with the ROUGE 1.5.5 evaluation scores over TAC 2008 (for- merly DUC, conducted by NIST) dataset, the proposed evaluation tech- nique predicts the ROUGE scores with a accuracy of 98.25% with respect to ROUGE-2 and 95.65% with respect to ROUGE-SU4.