Abstractive Summarization Model Research Articles

A novel abstractive summarization model based on topic-aware and contrastive learning

A novel abstractive summarization model based on topic-aware and contrastive learning

Abstractive text summarization model combining a hierarchical attention mechanism and multiobjective reinforcement learning

Text summarization research is significant and challenging in the domain of natural language processing. Abstractive text summarization mainly uses the encoder-decoder framework, wherein the encoder component does not have a sufficient semantic comprehension of the input text, and there are exposure biases and semantic inconsistencies between the reference and generated summaries during the training process. We propose an improved encoder-decoder model that incorporates a hierarchical attention mechanism and multiobjective reinforcement learning. The encoder introduces a multihead self-attention mechanism to allow for the acquisition of more comprehensive semantic information from multiple angles and dimensions, while the decoder introduces a pointer-generator network to solve the out-of-vocabulary problem. Multiobjective reinforcement learning methods are constructed throughout the training process to optimize the model in terms of addressing exposure bias, maintaining semantic consistency, and enhancing readability. The results of the comparative experiments demonstrate that the proposed model significantly improved in terms of the ROUGE evaluation metric, and the generated summaries were semantically similar to the reference summaries.

Methodical Systematic Review of Abstractive Summarization and Natural Language Processing Models for Biomedical Health Informatics: Approaches, Metrics and Challenges

Text summarization tasks are primarily very useful for decision support systems and provide a source for useful data for training of bots as they can reduce and retain the useful information from the large corpus. This review article is for studying the literature that already exists in context of abstractive summarization and application of NLP language models in biomedical and associated healthcare applications. In past decade with trends like bigdata, IOT, enormous amount of data is getting processed in all structured, unstructured and semi structured formats. This review provides a comprehensive literature survey in research trends for abstractive summarization, foundations of machine translation and evolution of language models. This review identifies the potential of language model to provide a possible methodology for improving the performance and accuracy of various tasks in summarization. Deep neural network-based language models have now been the widely accepted state of art for various abstractive summarization and there exists an enormous scope to improvise and tune the language models for domain specific use case. This study shows current systems lack in faithfulness to original content and control of degree of hallucination. This review also details on the evaluation criteria and need for automated metrics and attempts to provide guideline for evaluation for abstractive summarization for health informatics.

Joint learning of text alignment and abstractive summarization for long documents via unbalanced optimal transport

Abstract Recently, neural abstractive text summarization (NATS) models based on sequence-to-sequence architecture have drawn a lot of attention. Real-world texts that need to be summarized range from short news with dozens of words to long reports with thousands of words. However, most existing NATS models are not good at summarizing long documents, due to the inherent limitations of their underlying neural architectures. In this paper, we focus on the task of long document summarization (LDS). Based on the inherent section structures of source documents, we divide an abstractive LDS problem into several smaller-sized problems. In this circumstance, how to provide a less-biased target summary as the supervision for each section is vital for the model’s performance. As a preliminary, we formally describe the section-to-summary-sentence (S2SS) alignment for LDS. Based on this, we propose a novel NATS framework for the LDS task. Our framework is built based on the theory of unbalanced optimal transport (UOT), and it is named as UOTSumm. It jointly learns three targets in a unified training objective, including the optimal S2SS alignment, a section-level NATS summarizer, and the number of aligned summary sentences for each section. In this way, UOTSumm directly learns the text alignment from summarization data, without resorting to any biased tool such as ROUGE. UOTSumm can be easily adapted to most existing NATS models. And we implement two versions of UOTSumm, with and without the pretrain-finetune technique. We evaluate UOTSumm on three publicly available LDS benchmarks: PubMed, arXiv, and GovReport. UOTSumm obviously outperforms its counterparts that use ROUGE for the text alignment. When combined with UOTSumm, the performance of two vanilla NATS models improves by a large margin. Besides, UOTSumm achieves better or comparable performance when compared with some recent strong baselines.

An Abstractive Summarization Model Based on Joint-Attention Mechanism and a Priori Knowledge

An abstractive summarization model based on the joint-attention mechanism and a priori knowledge is proposed to address the problems of the inadequate semantic understanding of text and summaries that do not conform to human language habits in abstractive summary models. Word vectors that are most relevant to the original text should be selected first. Second, the original text is represented in two dimensions—word-level and sentence-level, as word vectors and sentence vectors, respectively. After this processing, there will be not only a relationship between word-level vectors but also a relationship between sentence-level vectors, and the decoder discriminates between word-level and sentence-level vectors based on their relationship with the hidden state of the decoder. Then, the pointer generation network is improved using a priori knowledge. Finally, reinforcement learning is used to improve the quality of the generated summaries. Experiments on two classical datasets, CNN/DailyMail and DUC 2004, show that the model has good performance and effectively improves the quality of generated summaries.

Hierarchical Sliding Inference Generator for Question-driven Abstractive Answer Summarization

Text summarization on non-factoid question answering (NQA) aims at identifying the core information of redundant answer guidance using questions, which can dramatically improve answer readability and comprehensibility. Most existing approaches focus on extracting query-related sentences to construct a summary, where the logical connection of natural language and the hierarchical interpretable semantic association are often neglected, thus degrading performance. To address these issues, we propose a novel question-driven abstractive answer summarization model, called the H ierarchical S liding I nference G enerator (HSIG), to form inferable and interpretable summaries by explicitly introducing hierarchical information reasoning between questions and corresponding answers. Specifically, we first apply an elaborately designed hierarchical sliding fusion inference model to determine the most relevant question sentence-level representation that provides a deeper interpretable basis for sentence selection in summarization, which further increases computational performance on the premise of following the semantic inheritance structure. Additionally, to improve summary fluency, we construct a double-driven selective generator to integrate various semantic information from two mutual question-and-answer perspectives. Experimental results illustrate that compared with state-of-the-art baselines, our model achieves remarkable improvement on two benchmark datasets and specifically improves the 2.46 ROUGE-1 points on PubMedQA, which demonstrates the superiority of our model on abstractive summarization with hierarchical sequential reasoning.

Attribute-Sentiment-Guided Summarization of User Opinions From Online Reviews

Abstract Eliciting informative user opinions from online reviews is a key success factor for innovative product design and development. The unstructured, noisy, and verbose nature of user reviews, however, often complicate large-scale need finding in a format useful for designers without losing important information. Recent advances in abstractive text summarization have created the opportunity to systematically generate opinion summaries from online reviews to inform the early stages of product design and development. However, two knowledge gaps hinder the applicability of opinion summarization methods in practice. First, there is a lack of formal mechanisms to guide the generative process with respect to different categories of product attributes and user sentiments. Second, the annotated training datasets needed for supervised training of abstractive summarization models are often difficult and costly to create. This article addresses these gaps by (1) devising an efficient computational framework for abstractive opinion summarization guided by specific product attributes and sentiment polarities, and (2) automatically generating a synthetic training dataset that captures various degrees of granularity and polarity. A hierarchical multi-instance attribute-sentiment inference model is developed for assembling a high-quality synthetic dataset, which is utilized to fine-tune a pretrained language model for abstractive summary generation. Numerical experiments conducted on a large dataset scraped from three major e-Commerce retail stores for apparel and footwear products indicate the performance, feasibility, and potentials of the developed framework. Several directions are provided for future exploration in the area of automated opinion summarization for user-centered design.

Exploiting Intersentence Information for Better Question-Driven Abstractive Summarization: Algorithm Development and Validation.

BackgroundQuestion-driven summarization has become a practical and accurate approach to summarizing the source document. The generated summary should be concise and consistent with the concerned question, and thus, it could be regarded as the answer to the nonfactoid question. Existing methods do not fully exploit question information over documents and dependencies across sentences. Besides, most existing summarization evaluation tools like recall-oriented understudy for gisting evaluation (ROUGE) calculate N-gram overlaps between the generated summary and the reference summary while neglecting the factual consistency problem.ObjectiveThis paper proposes a novel question-driven abstractive summarization model based on transformer, including a two-step attention mechanism and an overall integration mechanism, which can generate concise and consistent summaries for nonfactoid question answering.MethodsSpecifically, the two-step attention mechanism is proposed to exploit the mutual information both of question to context and sentence over other sentences. We further introduced an overall integration mechanism and a novel pointer network for information integration. We conducted a question-answering task to evaluate the factual consistency between the generated summary and the reference summary.ResultsThe experimental results of question-driven summarization on the PubMedQA data set showed that our model achieved ROUGE-1, ROUGE-2, and ROUGE-L measures of 36.01, 15.59, and 30.22, respectively, which is superior to the state-of-the-art methods with a gain of 0.79 (absolute) in the ROUGE-2 score. The question-answering task demonstrates that the generated summaries of our model have better factual constancy. Our method achieved 94.2% accuracy and a 77.57% F1 score.ConclusionsOur proposed question-driven summarization model effectively exploits the mutual information among the question, document, and summary to generate concise and consistent summaries.

Abstractive Text Summarization Model with Coherence Reinforcement and No Ground Truth Dependency

Abstractive Text Summarization Model with Coherence Reinforcement and No Ground Truth Dependency

CATS: Customizable Abstractive Topic-based Summarization

Neural sequence-to-sequence models are the state-of-the-art approach used in abstractive summarization of textual documents, useful for producing condensed versions of source text narratives without being restricted to using only words from the original text. Despite the advances in abstractive summarization, custom generation of summaries (e.g., towards a user’s preference) remains unexplored. In this article, we present CATS, an abstractive neural summarization model that summarizes content in a sequence-to-sequence fashion while also introducing a new mechanism to control the underlying latent topic distribution of the produced summaries. We empirically illustrate the efficacy of our model in producing customized summaries and present findings that facilitate the design of such systems. We use the well-known CNN/DailyMail dataset to evaluate our model. Furthermore, we present a transfer-learning method and demonstrate the effectiveness of our approach in a low resource setting, i.e., abstractive summarization of meetings minutes, where combining the main available meetings’ transcripts datasets, AMI and International Computer Science Institute(ICSI) , results in merely a few hundred training documents.

NASca and NASes: Two Monolingual Pre-Trained Models for Abstractive Summarization in Catalan and Spanish

Most of the models proposed in the literature for abstractive summarization are generally suitable for the English language but not for other languages. Multilingual models were introduced to address that language constraint, but despite their applicability being broader than that of the monolingual models, their performance is typically lower, especially for minority languages like Catalan. In this paper, we present a monolingual model for abstractive summarization of textual content in the Catalan language. The model is a Transformer encoder-decoder which is pretrained and fine-tuned specifically for the Catalan language using a corpus of newspaper articles. In the pretraining phase, we introduced several self-supervised tasks to specialize the model on the summarization task and to increase the abstractivity of the generated summaries. To study the performance of our proposal in languages with higher resources than Catalan, we replicate the model and the experimentation for the Spanish language. The usual evaluation metrics, not only the most used ROUGE measure but also other more semantic ones such as BertScore, do not allow to correctly evaluate the abstractivity of the generated summaries. In this work, we also present a new metric, called content reordering, to evaluate one of the most common characteristics of abstractive summaries, the rearrangement of the original content. We carried out an exhaustive experimentation to compare the performance of the monolingual models proposed in this work with two of the most widely used multilingual models in text summarization, mBART and mT5. The experimentation results support the quality of our monolingual models, especially considering that the multilingual models were pretrained with many more resources than those used in our models. Likewise, it is shown that the pretraining tasks helped to increase the degree of abstractivity of the generated summaries. To our knowledge, this is the first work that explores a monolingual approach for abstractive summarization both in Catalan and Spanish.

An Automatic Abstractive Text Summarization Model based on Hybrid Attention Mechanism

Attentional sequence-to-sequence models based on RNN have achieved promising performances in the automatic abstractive summarization technology. However, there are still some shortcomings, including the inaccuracies and lack of key information. In this paper, an abstractive text summarization model based on the hybrid attention mechanism is presented. The model aims at adopting sentence-level attention mechanism to guide the word-level attention distribution. Moreover, this model modulates the weight of sentence-level attention value to alleviate the adverse effect of high variance on word-level attention distribution for short documents. The experimental results on the LCSTS dataset show that the presented model can effectively improve the ROUGE scores and can better summarize the source document while retaining the important information. An example in this paper shows that the model can generate summaries that are similar to human-written ones.

문장 랭킹 스코어와 그래프 기법을 사용한 질의 기반 생성 요약 모델

기본적인 생성 요약 모델은 문서 내 모든 중요 내용을 포함하는 짧은 요약문을 생성하는 것을 목표로 한다. 반면, 질의(Query) 기반 생성 요약 모델의 경우 문서 내에서 질의와 관련된 정보를 요약해야 한다. 기존의 질의 기반 요약 모델은 문서 내 단어들과 질의문 간의 어텐션(Attention) 메커니즘을 통해 단어의 가중치를 계산하고 이를 기반으로 문장의 중요도를 계산한다. 이러한 방식은 문서의 전체적인 문맥정보를 반영하기 어렵다는 단점이 있다. 본 논문에서는 이러한 문제를 문장 랭킹 스코어와 문장 단위 그래프 구조를 만들어 문장의 중요도뿐만 아니라 문맥 정보를 반영하여 생성 요약의 성능을 향상시킬 수 있는 새로운 생성 요약 기법을 제안한다. 실험으로 살펴본 최종 제안 모델의 성능은 같은 데이터를 사용하는 선행 모델 대비 ROUGE-1 1.44%p, ROUGE-L 0.52%p의 향상된 성능을 보인다.

A two-step abstractive summarization model with asynchronous and enriched-information decoding

Most sequence-to-sequence abstractive summarization models generate the summaries based on the source article and the generated words, but they often neglect the future information implied in the un-generated words, which means that they lack the ability of “lookahead.” In this paper, we present a novel summarization model with “lookahead” ability to fully employ the implied future information. Our model takes two steps: (1) in the first step, an asynchronous decoder model with a no ground truth guiding backward decoder that explicitly produces and exploits the future information is trained. (2) in the inference process, in addition to the joint probability of the generated sequence, an enriched-information decoding method is proposed to further take future ROUGE reward of the un-generated words into account. Furthermore, the future ROUGE reward is predicted by a novel reward-predict model, and it takes the hidden states of the pre-trained asynchronous decoder model as input. Experimental results show that our two-step summarization model achieves new state-of-the-art results on CNN/Daily Mail dataset and the generalization of our model on test-only DUC-2002 datasets achieves higher scores than the state-of-the-art model.

An Optimized Abstractive Text Summarization Model Using Peephole Convolutional LSTM

Abstractive text summarization that generates a summary by paraphrasing a long text remains an open significant problem for natural language processing. In this paper, we present an abstractive text summarization model, multi-layered attentional peephole convolutional LSTM (long short-term memory) (MAPCoL) that automatically generates a summary from a long text. We optimize parameters of MAPCoL using central composite design (CCD) in combination with the response surface methodology (RSM), which gives the highest accuracy in terms of summary generation. We record the accuracy of our model (MAPCoL) on a CNN/DailyMail dataset. We perform a comparative analysis of the accuracy of MAPCoL with that of the state-of-the-art models in different experimental settings. The MAPCoL also outperforms the traditional LSTM-based models in respect of semantic coherence in the output summary.