ATSSC: Development of an approach based on soft computing for text summarization

Abstract

Natural Language Processing (NLP) is a field of computer science and linguistics concerned with the unique conversation between computers and human languages. It processes data through Lexical analysis, Syntax analysis, Semantic analysis, Discourse processing and Pragmatic analysis. An intelligent text summarization is one of the most challenging tasks in Natural language processing. It can be further used for applications like storytelling and question answering. This paper presents an automatic text summarizer for text documents using soft computing approach, consisting of SVO (Subject, Verb, and Object) Rules and Tag based training. This approach processes data through POS Tagger, NLP Parser, ambiguity removal, Semantic Representation, Sentence Reduction and Sentence Combination. At first, this paper defines the theme (title) of the document. After this operation, it preprocesses text document to perform pronominal reference resolution and text clustering. After these preprocessing operations, it identifies and removes ambiguity from the language using parser. And then, it calculates the score for the sentences using the title of the document, Semantic Sentence Similarity utility and n-gram Co-Occurrence relations of the words in a particular sentence. At last, sentences are combined with the SVO Rules after providing tag based training for simple and complex sentences. The summarizer was tested on the standard DUC 2007 dataset as well as a corpus of hundred text documents of different domains created by us. DUC 2007 Update Task produced accuracy F-scores of 0.13523 (ROUGE-2) and 0.112561 (ROUGE-SU4) for DUC 2007 documents and 0.4036 (ROUGE-2) and 0.3129 (ROUGE-SU4) for our corpus. Subjective evaluation was carried out by five language experts and twenty random individuals for system generated sample summaries.