Evidence Sentences Research Articles

Document-level multiple relations extraction method via evidence guidance and relation correlation

Document-level Relation Extraction (DocRE) aims to extract semantic relations between entity pairs, spanning multiple sentences, paragraphs or even the entire document. These relations can often be predicted by partial sentences within the document, the evidence sentence. However, the relation derived only from sentence information is incomplete, because it ignores the case of multiple relations between entity pairs. Therefore, how to select effective evidence sentences and how to predict multiple relations more accurately have become challenges for the existing DocRE models. In response to these challenges, we introduce Reinforcement Learning (RL) to select more effective evidence sentences, while using heuristic rules to narrow down the search space of RL. Secondly, we utilize GAT to acquire the features of co-occurrence relations, which can greatly improve multiple relations prediction performance. Moreover, the combination of the features of co-occurrence relations and the evidence sentence information enables our method to achieve both high effectiveness and precision. The experimental results show that, compared with other advanced methods, our method achieves an F1 score of 66.56 and the EviF1 score of 56.69, which attains the state-of-the-art performance on public datasets.

Pipeline and dataset generation for automated fact-checking in almost any language

This article presents a pipeline for automated fact-checking leveraging publicly available language models and data. The objective is to assess the accuracy of textual claims using evidence from a ground-truth evidence corpus. The pipeline consists of two main modules—the evidence retrieval and the claim veracity evaluation. Our primary focus is on the ease of deployment in various languages that remain unexplored in the field of automated fact-checking. Unlike most similar pipelines, which work with evidence sentences, our pipeline processes data on a paragraph level, simplifying the overall architecture and data requirements. Given the high cost of annotating language-specific fact-checking training data, our solution builds on the question answering for claim generation method, which we adapt and use to generate the data for all models of the pipeline. Our strategy enables the introduction of new languages through machine translation of only two fixed datasets of moderate size. Subsequently, any number of training samples can be generated based on an evidence corpus in the target language. We provide open access to all data and fine-tuned models for Czech, English, Polish, and Slovak pipelines, as well as to our codebase that may be used to reproduce the results. We comprehensively evaluate the pipelines for all four languages, including human annotations and per-sample difficulty assessment using Pointwise V\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {V}}$$\\end{document}-information. The presented experiments are based on full Wikipedia snapshots to promote reproducibility. To facilitate implementation and user interaction, we develop the FactSearch application featuring the proposed pipeline and the preliminary feedback on its performance.

Assessing citation integrity in biomedical publications: corpus annotation and NLP models.

Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model-which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification-yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy. We make the corpus and the best-performing NLP model publicly available at https://github.com/ScienceNLP-Lab/Citation-Integrity/.

Enhancing cross-evidence reasoning graph for document-level relation extraction.

The objective of document-level relation extraction (RE) is to identify the semantic connections that exist between named entities present within a document. However, most entities are distributed among different sentences, there is a need for inter-entity relation prediction across sentences. Existing research has focused on framing sentences throughout documents to predict relationships between entities. However, not all sentences play a substantial role in relation extraction, which inevitably introduces noisy information. Based on this phenomenon, we believe that we can extract evidence sentences in advance and use these evidence sentences to construct graphs to mine semantic information between entities. Thus, we present a document-level RE model that leverages an Enhancing Cross-evidence Reasoning Graph (ECRG) for improved performance. Specifically, we design an evidence extraction rule based on center-sentence to pre-extract higher-quality evidence. Then, this evidence is constructed into evidence graphs to mine the connections between mentions within the same evidence. In addition, we construct entity-level graphs by aggregating mentions from the same entities within the evidence graphs, aiming to capture distant interactions between entities. Experiments result on both DocRED and RE-DocRED datasets demonstrate that our model improves entity RE performance compared to existing work.

A syntactic evidence network model for fact verification

In natural language processing, fact verification is a very challenging task, which requires retrieving multiple evidence sentences from a reliable corpus to verify the authenticity of a claim. Although most of the current deep learning methods use the attention mechanism for fact verification, they have not considered imposing attentional constraints on important related words in the claim and evidence sentences, resulting in inaccurate attention for some irrelevant words. In this paper, we propose a syntactic evidence network (SENet) model which incorporates entity keywords, syntactic information and sentence attention for fact verification. The SENet model extracts entity keywords from claim and evidence sentences, and uses a pre-trained syntactic dependency parser to extract the corresponding syntactic sentence structures and incorporates the extracted syntactic information into the attention mechanism for language-driven word representation. In addition, the sentence attention mechanism is applied to obtain a richer semantic representation. We have conducted experiments on the FEVER and UKP Snopes datasets for performance evaluation. Our SENet model has achieved 78.69% in Label Accuracy and 75.63% in FEVER Score on the FEVER dataset. In addition, our SENet model also has achieved 65.0% in precision and 61.2% in macro F1 on the UKP Snopes dataset. The experimental results have shown that our proposed SENet model has outperformed the baseline models and achieved the state-of-the-art performance for fact verification.

IDPpub: Illuminating the Dark Phosphoproteome Through PubMed Mining

Global phosphoproteomics experiments quantify tens of thousands of phosphorylation sites. However, data interpretation is hampered by our limited knowledge on functions, biological contexts, or precipitating enzymes of the phosphosites. This study establishes a repository of phosphosites with associated evidence in biomedical abstracts, using deep learning-based natural language processing techniques. Our model for illuminating the dark phosphoproteome through PubMed mining (IDPpub) was generated by fine-tuning BioBERT, a deep learning tool for biomedical text mining. Trained using sentences containing protein substrates and phosphorylation site positions from 3000 abstracts, the IDPpub model was then used to extract phosphorylation sites from all MEDLINE abstracts. The extracted proteins were normalized to gene symbols using the National Center for Biotechnology Information gene query, and sites were mapped to human UniProt sequences using ProtMapper and mouse UniProt sequences by direct match. Precision and recall were calculated using 150 curated abstracts, and utility was assessed by analyzing the CPTAC (Clinical Proteomics Tumor Analysis Consortium) pan-cancer phosphoproteomics datasets and the PhosphoSitePlus database. Using 10-fold cross validation, pairs of correct substrates and phosphosite positions were extracted with an average precision of 0.93 and recall of 0.94. After entity normalization and site mapping to human reference sequences, an independent validation achieved a precision of 0.91 and recall of 0.77. The IDPpub repository contains 18,458 unique human phosphorylation sites with evidence sentences from 58,227 abstracts and 5918 mouse sites in 14,610 abstracts. This included evidence sentences for 1803 sites identified in CPTAC studies that are not covered by manually curated functional information in PhosphoSitePlus. Evaluation results demonstrate the potential of IDPpub as an effective biomedical text mining tool for collecting phosphosites. Moreover, the repository (http://idppub.ptmax.org), which can be automatically updated, can serve as a powerful complement to existing resources.

Document-level Relation Extraction via Separate Relation Representation and Logical Reasoning

Document-level relation extraction (RE) extends the identification of entity/mentions’ relation from the single sentence to the long document. It is more realistic and poses new challenges to relation representation and reasoning skills. In this article, we propose a novel model, SRLR , using S eparate Relation R epresentation and L ogical R easoning considering the indirect relation representation and complex reasoning of evidence sentence problems. Specifically, we first expand the judgment of relational facts from the entity-level to the mention-level, highlighting fine-grained information to capture the relation representation for the entity pair. Second, we propose a logical reasoning module to identify evidence sentences and conduct relational reasoning. Extensive experiments on two publicly available benchmark datasets demonstrate the effectiveness of our proposed SRLR as compared to 19 baseline models. Further ablation study also verifies the effects of the key components.

Mining heuristic evidence sentences for more interpretable document-level relation extraction

Current research on evidence sentences is aimed at developing document-level relational extraction models with improved interpretability. Evidence sentences extracted using existing methods are often incomplete, leading to poor relationship prediction accuracy. To address this problem, we developed a novel efficient heuristic rule and entity representation method. First, a heuristic rule is constructed according to the interactions between different mentions of the head and tail entities of the target entity pair, and evidence sentences are subsequently extracted. Second, pseudo documents, constructed according to the original document order, are used as input text to remove noisy statements. Finally, different representations of the same entity in different entity pairs are learned to represent it more accurately through the interactive mention of head and tail entities. Experiments on the document-level general domain dataset DocRED indicated that our heuristic rules improved sentence extraction by 6.01% compared to that achieved by the baseline model Paths-BiLSTM. In terms of relation prediction, the accuracy of the proposed method was comparable to those of existing models that use the entire document as input text; however, the input text used by the proposed method was shorter and more interpretable.

A Feedback-Enhanced Two-Stage Framework for judicial machine reading comprehension

Machine Reading Comprehension (MRC) is the task of teaching machines to understand and answer questions based on a given text. In the judicial domain, MRC has garnered attention as a novel strategy for addressing a variety of legal issues. Judicial MRC generally requires interpretability, meaning that the model should not only answer the question correctly but also provide supporting evidence sentences. A straightforward approach is to handle the two tasks separately, which ignores the relation between the two tasks and leads to the loss of annotated information. To make better use of the limited judicial MRC annotation data, we simulate the strategies used by humans in solving MRC problems and propose the Feedback-Enhanced Two-Stage Framework for Machine Reading Comprehension (FETSF-MRC). This framework consists of two cascaded modules: (a) a scanning module that identifies evidence sentences and (b) a detailed reading module that focuses on the evidence to identify candidate answers and provides feedback to the scanning module. Experiments on two Chinese judicial reading comprehension datasets (CJRC, CAIL2020) and an open-domain English dataset HotpotQA show that FETSF-MRC achieves superior performance compared to several baselines. FETSF-MRC outperforms the best baseline by 1.12% and 1.49% in joint F1 score on the CJRC and CAIL2020 datasets, respectively, and achieves 73.74% joint F1 score on the HotpotQA.

Dual-Channel Reasoning Model for Complex Question Answering

Multihop question answering has attracted extensive studies in recent years because of the emergence of human annotated datasets and associated leaderboards. Recent studies have revealed that question answering systems learn to exploit annotation artifacts and other biases in current datasets. Therefore, a model with strong interpretability should not only predict the final answer, but more importantly find the supporting facts’ sentences necessary to answer complex questions, also known as evidence sentences. Most existing methods predict the final answer and evidence sentences in sequence or simultaneously, which inhibits the ability of models to predict the path of reasoning. In this paper, we propose a dual-channel reasoning architecture, where two reasoning channels predict the final answer and supporting facts’ sentences, respectively, while sharing the contextual embedding layer. The two reasoning channels can simply use the same reasoning structure without additional network designs. Through experimental analysis based on public question answering datasets, we demonstrate the effectiveness of our proposed method

The Complexities of Conscience: Reconciling Death Penalty Law with Capital Jurors’ Concerns

Jurors exercise unique legal power when they are called upon to decide whether to sentence someone to death. The Supreme Court emphasizes the central role of the jury’s moral judgment in making this sentencing decision, noting that it is the jurors who are ‘best able to express the conscience of the community on the ultimate question of life or death.’” Many lower courts nevertheless narrow the range of admissible evidence at the mitigation phase of a capital trial, insisting on a standard of legal relevance that interferes with the jury’s ability to exercise the very moral judgment the Supreme Court has deemed essential. Combining moral theory and original empirical evidence, this Article breaks new ground by linking these to a legal framework that gives full effect to the Supreme Court’s vision of the jury. Aided by a novel dataset of federal capital jury verdict forms, the Article focuses on three types of evidence frequently excluded in state and federal courts: the impact of the defendant’s execution on loved ones, co-participant sentences, and the government’s negligent facilitation of the murder. The data show that jurors consistently find all three forms of evidence highly salient in their mitigation deliberations. Further, two of these—execution impact evidence and co-participant sentences—have a statistically significant correlation with the jurors’ sentencing decision. This Article’s empirical and moral account of juror behavior strongly supports expanding the admissibility of this evidence to reflect the Supreme Court’s evolution in defining the relevance of mitigating evidence as a moral, rather than legalistic, question, appropriately recognizing the jury’s normative role.

DigChem: Identification of disease-gene-chemical relationships from Medline abstracts.

Chemicals interact with genes in the process of disease development and treatment. Although much biomedical research has been performed to understand relationships among genes, chemicals, and diseases, which have been reported in biomedical articles in Medline, there are few studies that extract disease–gene–chemical relationships from biomedical literature at a PubMed scale. In this study, we propose a deep learning model based on bidirectional long short-term memory to identify the evidence sentences of relationships among genes, chemicals, and diseases from Medline abstracts. Then, we develop the search engine DigChem to enable disease–gene–chemical relationship searches for 35,124 genes, 56,382 chemicals, and 5,675 diseases. We show that the identified relationships are reliable by comparing them with manual curation and existing databases. DigChem is available at http://gcancer.org/digchem.

MAPK8 mediates resistance to temozolomide and apoptosis of glioblastoma cells through MAPK signaling pathway

ObjectiveIn this study, we aimed to evaluate the expression and functions of MAPK8 in temozolomide (TMZ) -resistant glioblastoma cells as well as to explore the mechanism of TMZ resistance in glioblastoma cells. MethodsGene Expression Omnibus (GEO) database was used for identifying the differentially expressed genes (DEGs) in TMZ resistant samples. The functional partner genes of TMZ were screened out by Gene-drug interaction network (STITCH) and the glioblastoma-related genes were selected by gene search engine with evidence sentences (Digsee). The interactions among identified DEGs and glioblastoma-related genes were detected by Search Tool for the Retrieval of Interacting Genes (STRING). The dysregulated pathways were identified by Gene set enrichment analysis (GSEA). qRT-PCR was performed to detect the expression level of MAPK8 in glioblastoma cells. Western blot was used to detect the expressions of MAPK8 and MAPK signaling pathway-related proteins. MTT assay was utilized to measure the cell viability of TMZ sensitive and resistant cells. Colony formation assay was performed to detect the clone ability and flow cytometry (FCM) assay was applied to identify the apoptosis rate of TMZ resistant glioblastoma cells. ResultsMAPK8 was one of the DEGs and was up-regulated in TMZ resistant glioblastoma cells. The MAPK signaling pathway was activated in TMZ resistant glioblastoma cells under the condition of over-expression of MAPK8. The inhibition of MAPK8 restrained the colony formation, inducing apoptosis of TMZ resistant glioblastoma cells and suppressed the MAPK signaling pathway. ConclusionMAPK8 promoted the resistance to TMZ, accelerated cell proliferation and inhibited the apoptosis of glioblastoma cells via activating MAPK signaling pathway.

BELMiner: adapting a rule-based relation extraction system to extract biological expression language statements from bio-medical literature evidence sentences.

Extracting meaningful relationships with semantic significance from biomedical literature is often a challenging task. BioCreative V track4 challenge for the first time has organized a comprehensive shared task to test the robustness of the text-mining algorithms in extracting semantically meaningful assertions from the evidence statement in biomedical text. In this work, we tested the ability of a rule-based semantic parser to extract Biological Expression Language (BEL) statements from evidence sentences culled out of biomedical literature as part of BioCreative V Track4 challenge. The system achieved an overall best F-measure of 21.29% in extracting the complete BEL statement. For relation extraction, the system achieved an F-measure of 65.13% on test data set. Our system achieved the best performance in five of the six criteria that was adopted for evaluation by the task organizers. Lack of ability to derive semantic inferences, limitation in the rule sets to map the textual extractions to BEL function were some of the reasons for low performance in extracting the complete BEL statement. Post shared task we also evaluated the impact of differential NER components on the ability to extract BEL statements on the test data sets besides making a single change in the rule sets that translate relation extractions into a BEL statement. There is a marked improvement by over 20% in the overall performance of the BELMiner’s capability to extract BEL statement on the test set. The system is available as a REST-API at http://54.146.11.205:8484/BELXtractor/finder/ Database URL: http://54.146.11.205:8484/BELXtractor/finder/

BELTracker: evidence sentence retrieval for BEL statements.

Biological expression language (BEL) is one of the main formal representation models of biological networks. The primary source of information for curating biological networks in BEL representation has been literature. It remains a challenge to identify relevant articles and the corresponding evidence statements for curating and validating BEL statements. In this paper, we describe BELTracker, a tool used to retrieve and rank evidence sentences from PubMed abstracts and full-text articles for a given BEL statement (per the 2015 task requirements of BioCreative V BEL Task). The system is comprised of three main components, (i) translation of a given BEL statement to an information retrieval (IR) query, (ii) retrieval of relevant PubMed citations and (iii) finding and ranking the evidence sentences in those citations. BELTracker uses a combination of multiple approaches based on traditional IR, machine learning, and heuristics to accomplish the task. The system identified and ranked at least one fully relevant evidence sentence in the top 10 retrieved sentences for 72 out of 97 BEL statements in the test set. BELTracker achieved a precision of 0.392, 0.532 and 0.615 when evaluated with three criteria, namely full, relaxed and context criteria, respectively, by the task organizers. Our team at Mayo Clinic was the only participant in this task. BELTracker is available as a RESTful API and is available for public use.Database URL: http://www.openbionlp.org:8080/BelTracker/finder/Given_BEL_Statement

Extraction of pharmacokinetic evidence of drug-drug interactions from the literature.

Drug-drug interaction (DDI) is a major cause of morbidity and mortality and a subject of intense scientific interest. Biomedical literature mining can aid DDI research by extracting evidence for large numbers of potential interactions from published literature and clinical databases. Though DDI is investigated in domains ranging in scale from intracellular biochemistry to human populations, literature mining has not been used to extract specific types of experimental evidence, which are reported differently for distinct experimental goals. We focus on pharmacokinetic evidence for DDI, essential for identifying causal mechanisms of putative interactions and as input for further pharmacological and pharmacoepidemiology investigations. We used manually curated corpora of PubMed abstracts and annotated sentences to evaluate the efficacy of literature mining on two tasks: first, identifying PubMed abstracts containing pharmacokinetic evidence of DDIs; second, extracting sentences containing such evidence from abstracts. We implemented a text mining pipeline and evaluated it using several linear classifiers and a variety of feature transforms. The most important textual features in the abstract and sentence classification tasks were analyzed. We also investigated the performance benefits of using features derived from PubMed metadata fields, various publicly available named entity recognizers, and pharmacokinetic dictionaries. Several classifiers performed very well in distinguishing relevant and irrelevant abstracts (reaching F1≈0.93, MCC≈0.74, iAUC≈0.99) and sentences (F1≈0.76, MCC≈0.65, iAUC≈0.83). We found that word bigram features were important for achieving optimal classifier performance and that features derived from Medical Subject Headings (MeSH) terms significantly improved abstract classification. We also found that some drug-related named entity recognition tools and dictionaries led to slight but significant improvements, especially in classification of evidence sentences. Based on our thorough analysis of classifiers and feature transforms and the high classification performance achieved, we demonstrate that literature mining can aid DDI discovery by supporting automatic extraction of specific types of experimental evidence.

A robust data-driven approach for gene ontology annotation.

Gene ontology (GO) and GO annotation are important resources for biological information management and knowledge discovery, but the speed of manual annotation became a major bottleneck of database curation. BioCreative IV GO annotation task aims to evaluate the performance of system that automatically assigns GO terms to genes based on the narrative sentences in biomedical literature. This article presents our work in this task as well as the experimental results after the competition. For the evidence sentence extraction subtask, we built a binary classifier to identify evidence sentences using reference distance estimator (RDE), a recently proposed semi-supervised learning method that learns new features from around 10 million unlabeled sentences, achieving an F1 of 19.3% in exact match and 32.5% in relaxed match. In the post-submission experiment, we obtained 22.1% and 35.7% F1 performance by incorporating bigram features in RDE learning. In both development and test sets, RDE-based method achieved over 20% relative improvement on F1 and AUC performance against classical supervised learning methods, e.g. support vector machine and logistic regression. For the GO term prediction subtask, we developed an information retrieval-based method to retrieve the GO term most relevant to each evidence sentence using a ranking function that combined cosine similarity and the frequency of GO terms in documents, and a filtering method based on high-level GO classes. The best performance of our submitted runs was 7.8% F1 and 22.2% hierarchy F1. We found that the incorporation of frequency information and hierarchy filtering substantially improved the performance. In the post-submission evaluation, we obtained a 10.6% F1 using a simpler setting. Overall, the experimental analysis showed our approaches were robust in both the two tasks.

Closing the loop: from paper to protein annotation using supervised Gene Ontology classification.

Gene function curation of the literature with Gene Ontology (GO) concepts is one particularly time-consuming task in genomics, and the help from bioinformatics is highly requested to keep up with the flow of publications. In 2004, the first BioCreative challenge already designed a task of automatic GO concepts assignment from a full text. At this time, results were judged far from reaching the performances required by real curation workflows. In particular, supervised approaches produced the most disappointing results because of lack of training data. Ten years later, the available curation data have massively grown. In 2013, the BioCreative IV GO task revisited the automatic GO assignment task. For this issue, we investigated the power of our supervised classifier, GOCat. GOCat computes similarities between an input text and already curated instances contained in a knowledge base to infer GO concepts. The subtask A consisted in selecting GO evidence sentences for a relevant gene in a full text. For this, we designed a state-of-the-art supervised statistical approach, using a naïve Bayes classifier and the official training set, and obtained fair results. The subtask B consisted in predicting GO concepts from the previous output. For this, we applied GOCat and reached leading results, up to 65% for hierarchical recall in the top 20 outputted concepts. Contrary to previous competitions, machine learning has this time outperformed standard dictionary-based approaches. Thanks to BioCreative IV, we were able to design a complete workflow for curation: given a gene name and a full text, this system is able to select evidence sentences for curation and to deliver highly relevant GO concepts. Contrary to previous competitions, machine learning this time outperformed dictionary-based systems. Observed performances are sufficient for being used in a real semiautomatic curation workflow. GOCat is available at http://eagl.unige.ch/GOCat/.Database URL: http://eagl.unige.ch/GOCat4FT/

Integrating information retrieval with distant supervision for gene ontology annotation.

This article describes our participation of the Gene Ontology Curation task (GO task) in BioCreative IV where we participated in both subtasks: A) identification of GO evidence sentences (GOESs) for relevant genes in full-text articles and B) prediction of GO terms for relevant genes in full-text articles. For subtask A, we trained a logistic regression model to detect GOES based on annotations in the training data supplemented with more noisy negatives from an external resource. Then, a greedy approach was applied to associate genes with sentences. For subtask B, we designed two types of systems: (i) search-based systems, which predict GO terms based on existing annotations for GOESs that are of different textual granularities (i.e., full-text articles, abstracts, and sentences) using state-of-the-art information retrieval techniques (i.e., a novel application of the idea of distant supervision) and (ii) a similarity-based system, which assigns GO terms based on the distance between words in sentences and GO terms/synonyms. Our best performing system for subtask A achieves an F1 score of 0.27 based on exact match and 0.387 allowing relaxed overlap match. Our best performing system for subtask B, a search-based system, achieves an F1 score of 0.075 based on exact match and 0.301 considering hierarchical matches. Our search-based systems for subtask B significantly outperformed the similarity-based system.Database URL: https://github.com/noname2020/Bioc

BC4GO: a full-text corpus for the BioCreative IV GO task.

Gene function curation via Gene Ontology (GO) annotation is a common task among Model Organism Database groups. Owing to its manual nature, this task is considered one of the bottlenecks in literature curation. There have been many previous attempts at automatic identification of GO terms and supporting information from full text. However, few systems have delivered an accuracy that is comparable with humans. One recognized challenge in developing such systems is the lack of marked sentence-level evidence text that provides the basis for making GO annotations. We aim to create a corpus that includes the GO evidence text along with the three core elements of GO annotations: (i) a gene or gene product, (ii) a GO term and (iii) a GO evidence code. To ensure our results are consistent with real-life GO data, we recruited eight professional GO curators and asked them to follow their routine GO annotation protocols. Our annotators marked up more than 5000 text passages in 200 articles for 1356 distinct GO terms. For evidence sentence selection, the inter-annotator agreement (IAA) results are 9.3% (strict) and 42.7% (relaxed) in F1-measures. For GO term selection, the IAAs are 47% (strict) and 62.9% (hierarchical). Our corpus analysis further shows that abstracts contain ∼10% of relevant evidence sentences and 30% distinct GO terms, while the Results/Experiment section has nearly 60% relevant sentences and >70% GO terms. Further, of those evidence sentences found in abstracts, less than one-third contain enough experimental detail to fulfill the three core criteria of a GO annotation. This result demonstrates the need of using full-text articles for text mining GO annotations. Through its use at the BioCreative IV GO (BC4GO) task, we expect our corpus to become a valuable resource for the BioNLP research community.Database URL: http://www.biocreative.org/resources/corpora/bc-iv-go-task-corpus/.