CBAG: Conditional biomedical abstract generation.

Justin Sybrandt,Ilya Safro

doi:10.1371/journal.pone.0253905

Abstract

Biomedical research papers often combine disjoint concepts in novel ways, such as when describing a newly discovered relationship between an understudied gene with an important disease. These concepts are often explicitly encoded as metadata keywords, such as the author-provided terms included with many documents in the MEDLINE database. While substantial recent work has addressed the problem of text generation in a more general context, applications, such as scientific writing assistants, or hypothesis generation systems, could benefit from the capacity to select the specific set of concepts that underpin a generated biomedical text. We propose a conditional language model following the transformer architecture. This model uses the “encoder stack” to encode concepts that a user wishes to discuss in the generated text. The “decoder stack” then follows the masked self-attention pattern to perform text generation, using both prior tokens as well as the encoded condition. We demonstrate that this approach provides significant control, while still producing reasonable biomedical text.

Highlights

Scientific papers often combine a range of disconnected concepts in novel patterns, following the typical research strategies of many scientists [1]
In Multi-Conditional Language Model we describe the methodology behind the Conditional Biomedical Abstract Generation (CBAG) model, which specializes the transformer architecture for generating biomedical abstracts
While Natural Language Processing (NLP) benchmarks such as GLUE [24] and its biomedical counterpart BLUE [22] help researchers compare performance across a range tasks, we are unaware of a benchmark for the generation of biomedical abstracts

Summary

Introduction

Scientific papers often combine a range of disconnected concepts in novel patterns, following the typical research strategies of many scientists [1]. The CBAG model is a transformer featuring a shallow encoder stack to encode qualities of the condition and a deep decoder stack to produce a high quality language model We train this model using semi-supervised multi-task generative pre-training, wherein to minimize our proposed objective function, the model must predict successive tokens, parts of speech, dependency tags, as well as entity labels. Trained using MEDLINE records and informed by semi-supervised domain-specific annotations, this model captures biomedical jargon, entities, and pattern of scientific discussion. We compare this model to two instances of GPT-2, both original and finetuned, and find competitive quantitative results. We discuss these concerns further in Future Challenges and Ethical Considerations

Background

Results

Related work

Conclusions