Relation Extraction Task Research Articles

Vulcan: Automatic extraction and analysis of cyber threat intelligence from unstructured text

To counteract the rapidly evolving cyber threats, many research efforts have been made to design cyber threat intelligence (CTI) systems that extract CTI data from publicly available sources. Specifically, indicators of compromise (IOC), such as file hash and IP address, receives the most attention among security researchers. However, the ability of IOC-centric CTI systems to understand and detect threats remains questionable for two reasons. First, IOCs are forensic artifacts that indicate that an endpoint or network has been compromised. They cannot depict the technical details of threats. Second, attackers frequently change infrastructure and static indicators, which makes IOCs have a very short lifespan. Therefore, when designing a CTI system, we should turn our attention to other types of CTI data that are helpful in threat understanding and detection (e.g., attack vector, tool).In this work, we propose Vulcan, a novel CTI system that extracts descriptive or static CTI data from unstructured text and determines their semantic relationships. To do this, we design a neural language model-based named entity recognition (NER) and relation extraction (RE) models tailored for cybersecurity domain. The experimental results confirm that Vulcan is highly accurate with an average F1-score of 0.972 and 0.985 for NER and RE tasks, respectively. Vulcan also provides an environment where security practitioners can develop applications for threat analysis. To prove the applicability of Vulcan, we introduce two applications, evolution identification and threat profiling. The applications save time and labor costs to analyze cyber threats and show the detailed characteristics of the threats.

Knowledge guided distance supervision for biomedical relation extraction in Chinese electronic medical records

The goal of biomedical relation extraction is to obtain structured information from electronic medical records by identifying relations among clinical entities. By integrating the advantages of unsupervised and semi-supervised learning, the distant supervision approach has achieved significant success for a relation extraction task without a large amount of labeled corpora. However, in many cases, the recognized entities from the Chinese clinical text are not defined in semantic knowledge base, which limits the application of distant supervision for biomedical relation extraction. This work proposes a Knowledge Guided Distance Supervision (KGDS) model for handling the biomedical relation extraction task in Chinese electronic medical records. To handle the unknown entities, entity-type alignment (instead of entity alignment in traditional distant supervision) is employed for extracting coarse-grained relations. Then, by learning the relation embeddings both from semantic knowledge base and electronic medical record dataset as knowledge-enhanced features, this work presents a knowledge-enhanced bootstrapping learning process for fine-grained relation disambiguation. The empirical experiments on the real-world dataset of electronic medical records illustrate that our KGDS model achieves the best performance comparing to other state-of-the-art models, thereby advancing the field of biomedical relation extraction from Chinese electronic medical records.

Distantly Supervised Relation Extraction via Recursive Hierarchy-Interactive Attention and Entity-Order Perception

Wrong-labeling problem and long-tail relations severely affect the performance of distantly supervised relation extraction task. Many studies mitigate the effect of wrong-labeling through selective attention mechanism and handle long-tail relations by introducing relation hierarchies to share knowledge. However, almost all existing studies ignore the fact that, in a sentence, the appearance order of two entities contributes to the understanding of its semantics. Furthermore, they only utilize each relation level of relation hierarchies separately, but do not exploit the heuristic effect between relation levels, i.e., higher-level relations can give useful information to the lower ones. Based on the above, in this paper, we design a novel Recursive Hierarchy-Interactive Attention network (RHIA) to further handle long-tail relations, which models the heuristic effect between relation levels. From the top down, it passes relation-related information layer by layer, which is the most significant difference from existing models, and generates relation-augmented sentence representations for each relation level in a recursive structure. Besides, we introduce a newfangled training objective, called Entity-Order Perception (EOP), to make the sentence encoder retain more entity appearance information. Substantial experiments on the popular New York Times (NYT) dataset are conducted. Compared to prior baselines, our RHIA-EOP achieves state-of-the-art performance in terms of precision–recall (P–R) curves, AUC, Top-N precision and other evaluation metrics. Insightful analysis also demonstrates the necessity and effectiveness of each component of RHIA-EOP.

Contextual Semantic-Guided Entity-Centric GCN for Relation Extraction

Relation extraction tasks aim to predict potential relations between entities in a target sentence. As entity mentions have ambiguity in sentences, some important contextual information can guide the semantic representation of entity mentions to improve the accuracy of relation extraction. However, most existing relation extraction models ignore the semantic guidance of contextual information to entity mentions and treat entity mentions in and the textual context of a sentence equally. This results in low-accuracy relation extractions. To address this problem, we propose a contextual semantic-guided entity-centric graph convolutional network (CEGCN) model that enables entity mentions to obtain semantic-guided contextual information for more accurate relational representations. This model develops a self-attention enhanced neural network to concentrate on the importance and relevance of different words to obtain semantic-guided contextual information. Then, we employ a dependency tree with entities as global nodes and add virtual edges to construct an entity-centric logical adjacency matrix (ELAM). This matrix can enable entities to aggregate the semantic-guided contextual information with a one-layer GCN calculation. The experimental results on the TACRED and SemEval-2010 Task 8 datasets show that our model can efficiently use semantic-guided contextual information to enrich semantic entity representations and outperform previous models.

Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction

BackgroundRecently, automatically extracting biomedical relations has been a significant subject in biomedical research due to the rapid growth of biomedical literature. Since the adaptation to the biomedical domain, the transformer-based BERT models have produced leading results on many biomedical natural language processing tasks. In this work, we will explore the approaches to improve the BERT model for relation extraction tasks in both the pre-training and fine-tuning stages of its applications. In the pre-training stage, we add another level of BERT adaptation on sub-domain data to bridge the gap between domain knowledge and task-specific knowledge. Also, we propose methods to incorporate the ignored knowledge in the last layer of BERT to improve its fine-tuning.ResultsThe experiment results demonstrate that our approaches for pre-training and fine-tuning can improve the BERT model performance. After combining the two proposed techniques, our approach outperforms the original BERT models with averaged F1 score improvement of 2.1% on relation extraction tasks. Moreover, our approach achieves state-of-the-art performance on three relation extraction benchmark datasets.ConclusionsThe extra pre-training step on sub-domain data can help the BERT model generalization on specific tasks, and our proposed fine-tuning mechanism could utilize the knowledge in the last layer of BERT to boost the model performance. Furthermore, the combination of these two approaches further improves the performance of BERT model on the relation extraction tasks.

Multi‐head attention graph convolutional network model: End‐to‐end entity and relation joint extraction based on multi‐head attention graph convolutional network

AbstractAt present, the entity and relation joint extraction task has attracted more and more scholars' attention in the field of natural language processing (NLP). However, most of their methods rely on NLP tools to construct dependency trees to obtain sentence structure information. The adjacency matrix constructed by the dependency tree can convey syntactic information. Dependency trees obtained through NLP tools are too dependent on the tools and may not be very accurate in contextual semantic description. At the same time, a large amount of irrelevant information will cause redundancy. This paper presents a novel end‐to‐end entity and relation joint extraction based on the multi‐head attention graph convolutional network model (MAGCN), which does not rely on external tools. MAGCN generates an adjacency matrix through a multi‐head attention mechanism to form an attention graph convolutional network model, uses head selection to identify multiple relations, and effectively improve the prediction result of overlapping relations. The authors extensively experiment and prove the method's effectiveness on three public datasets: NYT, WebNLG, and CoNLL04. The results show that the authors’ method outperforms the state‐of‐the‐art research results for the task of entities and relation extraction.

Hybrid neural tagging model for open relation extraction

Open Relation Extraction (ORE) task remains a challenge to obtain a semantic representation by discovering arbitrary relations from the unstructured text. Conventional methods heavily depend on feature engineering or syntactic parsing, which are inefficient or error-cascading. Recently, leveraging supervised deep learning methods to address the ORE task is a promising way. However, there are two main challenges: (1) The lack of enough labeled corpus to support supervised training; (2) The exploration of specific neural architecture that adapts to the characteristics of open relation extracting. In this paper, we build a large-scale, high-quality training corpus in a fully automated way. And wedesign a tagging scheme to assist in transforming the ORE task into a sequence tagging processing. Furthermore, we propose a hybrid neural network model (HNN4ORT) for open relation tagging. The model employs the Ordered Neurons LSTM to encode potential syntactic information to capture the associations among the arguments and relations. It also emerges a novel Dual Aware Mechanism, including Local-aware Attention and Global-aware Convolution. The dual awarenesses complement each other. Takes the sentence-level semantics as a global perspective, and at the same time, the model implements salient local features to achieve sparse annotation. Experiment results on various testing sets show that our model achieves state-of-the-art performance compared toconventional methods or other neural models.

Short Review on srBERT: Automatic Article Classification Model for Systematic Review Using BERT

Systematic reviews (SRs) have been recognized as the most rigorous and reliable approach to enable evidence-based medicine. However, the considerable workload required to create SRs prevents reflecting the latest knowledge. This study automated the classification of included articles by adopting the Bidirectional Encoder Representations from Transformers (BERT) algorithm. By pretraining with abstracts of articles and a generated vocabulary fine-tuned with titles of articles, the proposed srBERTmy overcomes the training data insufficiency while improving performance in both classification and relation-extraction tasks. Despite the limitation of model vulnerabilities based on training dataset quality, the results demonstrated the feasibility of automatic article classification using machine-learning (ML) approaches to support SR tasks Keywords: Systematic review, process automation, deep learning, text mining

DocR-BERT: Document-Level R-BERT for Chemical-Induced Disease Relation Extraction via Gaussian Probability Distribution.

Chemical-induced disease (CID) relation extraction from biomedical articles plays an important role in disease treatment and drug development. Existing methods are insufficient for capturing complete document level semantic information due to ignoring semantic information of entities in different sentences. In this work, we proposed an effective document-level relation extraction model to automatically extract intra-/inter-sentential CID relations from articles. Firstly, our model employed BERT to generate contextual semantic representations of the title, abstract and shortest dependency paths (SDPs). Secondly, to enhance the semantic representation of the whole document, cross attention with self-attention (named cross2self-attention) between abstract, title and SDPs was proposed to learn the mutual semantic information. Thirdly, to distinguish the importance of the target entity in different sentences, the Gaussian probability distribution was utilized to compute the weights of the co-occurrence sentence and its adjacent entity sentences. More complete semantic information of the target entity is collected from all entities occurring in the document via our presented document-level R-BERT (DocR-BERT). Finally, the related representations were concatenated and fed into the softmax function to extract CIDs. We evaluated the model on the CDR corpus provided by BioCreative V. The proposed model without external resources is superior in performance as compared with other state-of-the-art models (our model achieves 53.5%, 70%, and 63.7% of the F1-score on inter-/intra-sentential and overall CDR dataset). The experimental results indicate that cross2self-attention, the Gaussian probability distribution and DocR-BERT can effectively improve the CID extraction performance. Furthermore, the mutual semantic information learned by the cross self-attention from abstract towards title can significantly influence the extraction performance of document-level biomedical relation extraction tasks.

Entity and relation collaborative extraction approach based on multi-head attention and gated mechanism

Entity and relation extraction has been widely studied in natural language processing, and some joint methods have been proposed in recent years. However, existing studies still suffer from two problems. Firstly, the token space information has been fully utilized in those studies, while the label space information is underutilized. However, a few preliminary works have proven that the label space information could contribute to this task. Secondly, the performance of relevant entities detection is still unsatisfactory in entity and relation extraction tasks. In this paper, a new model GANCE (Gated and Attentive Network Collaborative Extracting) is proposed to address these problems. Firstly, GANCE exploits the label space information by applying a gating mechanism, which could improve the performance of the relation extraction. Then, two multi-head attention modules are designed to update the token and token-label fusion representation. In this way, the relevant entities detection could be solved. Experimental results demonstrate that GANCE has better accuracy than several competitive approaches in terms of entity recognition and relation extraction on the CoNLL04 dataset at 90.32% and 73.59%, respectively. Moreover, the F1 score of relation extraction increased by 1.24% over existing approaches in the ADE dataset.

A deep learning relation extraction approach to support a biomedical semi-automatic curation task: The case of the gluten bibliome

Discover relevant biomedical interactions in the literature is crucial for enhancing biology research. This curation process has an essential role in studying the different processes and interactions reported that affect the biological process (e.g., genome, metabolome, and transcriptome). In this sense, the objective of this work is twofold: reduce the manual effort required to curate and review the existing biochemical interactions reported in the gluten-related bibliome, while proposing a novel vector-space integrated into a deep learning model to assists manual curators in a real curation task by learning from their previous decisions. With this objective, the present work proposes a novel vector-space that combine (i) high-level lexical and syntactic inference features as Wordnets and Health-related domain ontologies, (ii) unsupervised semantic resources as word embedding, (iii) semantic and syntactic sentence knowledge, (iv) abbreviation resolution support, (v) several state-of-the-art Named-entity recognition methods, and, finally, (vi) different feature construction and optimization techniques to support a semi-automatic curation workflow. Therefore, the application of the proposed workflow over a classified set of 2,451 relevant gluten-related documents produces a total of 8,349 relevant and 471,813 irrelevant relations distributed in thirteen domain health-related categories. Experimental results showed that the proposed workflow is a valuable approach for a semi-automatic relation extraction task. It was able to obtain satisfactory results in the early stages of a real-world curation task and saved manual annotation efforts by learning from the decisions made by manual curators in iterative annotation rounds. The average F.score for the proposed relation categories was 0.731, being the lowest F.score at 0.47 and the highest F.score at 0.929. The different resources used in this work as well as the manually curated corpus are public available on our GitHub repository.

Biomedical relation extraction via knowledge-enhanced reading comprehension

BackgroundIn biomedical research, chemical and disease relation extraction from unstructured biomedical literature is an essential task. Effective context understanding and knowledge integration are two main research problems in this task. Most work of relation extraction focuses on classification for entity mention pairs. Inspired by the effectiveness of machine reading comprehension (RC) in the respect of context understanding, solving biomedical relation extraction with the RC framework at both intra-sentential and inter-sentential levels is a new topic worthy to be explored. Except for the unstructured biomedical text, many structured knowledge bases (KBs) provide valuable guidance for biomedical relation extraction. Utilizing knowledge in the RC framework is also worthy to be investigated. We propose a knowledge-enhanced reading comprehension (KRC) framework to leverage reading comprehension and prior knowledge for biomedical relation extraction. First, we generate questions for each relation, which reformulates the relation extraction task to a question answering task. Second, based on the RC framework, we integrate knowledge representation through an efficient knowledge-enhanced attention interaction mechanism to guide the biomedical relation extraction.ResultsThe proposed model was evaluated on the BioCreative V CDR dataset and CHR dataset. Experiments show that our model achieved a competitive document-level F1 of 71.18% and 93.3%, respectively, compared with other methods.ConclusionResult analysis reveals that open-domain reading comprehension data and knowledge representation can help improve biomedical relation extraction in our proposed KRC framework. Our work can encourage more research on bridging reading comprehension and biomedical relation extraction and promote the biomedical relation extraction.

Nessy: A Neuro-Symbolic System for Label Noise Reduction

Noisy labels represent one of the key issues in supervised machine learning. Existing work for label noise reduction mainly takes a probabilistic approach that infers true labels from data distributions in low-level feature spaces. Such an approach is not only limited by its capability to learn high-quality data representations, but also by the low predictive power of data distributions in inferring true classes. To address those problems, we introduce Nessy, a neuro-symbolic system that integrates deep probabilistic modeling and symbolic knowledge for label noise reduction. Our deep probabilistic model infers the true classes of data instances with noisy labels by exploiting data distributions in an underlying latent feature representation space. For data instances where inference is not reliable enough, Nessy extracts symbolic rules and ranks them according to several utility metrics. Top-ranking rules are injected into the deep probabilistic model via expectation regularization, i.e., via a posterior regularization term constraining the class distribution in the objective function. In a real deployment over multiple relation extraction tasks, we demonstrate that Nessy is able to significantly improve the state of the art, by 7% accuracy and 10.7% AUC on average.

Spatially oriented convolutional neural network for spatial relation extraction from natural language texts

AbstractSpatial relation extraction (e.g., topological relations, directional relations, and distance relations) from natural language descriptions is a fundamental but challenging task in several practical applications. Current state‐of‐the‐art methods rely on rule‐based metrics, either those specifically developed for extracting spatial relations or those integrated in methods that combine multiple metrics. However, these methods all rely on developed rules and do not effectively capture the characteristics of natural language spatial relations because the descriptions may be heterogeneous and vague and may be context sparse. In this article, we present a spatially oriented piecewise convolutional neural network (SP‐CNN) that is specifically designed with these linguistic issues in mind. Our method extends a general piecewise convolutional neural network with a set of improvements designed to tackle the task of spatial relation extraction. We also propose an automated workflow for generating training datasets by integrating new sentences with those in a knowledge base, based on string similarity and semantic similarity, and then transforming the sentences into training data. We exploit a spatially oriented channel that uses prior human knowledge to automatically match words and understand the linguistic clues to spatial relations, finally leading to an extraction decision. We present both the qualitative and quantitative performance of the proposed methodology using a large dataset collected from Wikipedia. The experimental results demonstrate that the SP‐CNN, with its supervised machine learning, can significantly outperform current state‐of‐the‐art methods on constructed datasets.

An attentive joint model with transformer-based weighted graph convolutional network for extracting adverse drug event relation

Adverse drug event (ADE) relation extraction is a crucial task for drug safety surveillance which aims to discover potential relations between ADE mentions from unstructured medical texts. To date, the graph convolutional networks (GCN) have been the state-of-the-art solutions for improving the ability of relation extraction task. However, there are many challenging issues that should be addressed. Among these, the syntactic information is not fully exploited by GCN-based methods, especially the diversified dependency edges. Still, these methods fail to effectively extract complex relations that include nested, discontinuous and overlapping mentions. Besides, the task is primarily regarded as a classification problem where each candidate relation is treated independently which neglects the interaction between other relations. To deal with these issues, in this paper, we propose an attentive joint model with transformer-based weighted GCN for extracting ADE Relations, called ADERel. Firstly, the ADERel system formulates the ADE relation extraction task as an N-level sequence labelling so as to model the complex relations in different levels and capture greater interaction between relations. Then, it exploits our neural joint model to process the N-level sequences jointly. The joint model leverages the contextual and structural information by adopting a shared representation that combines a bidirectional encoder representation from transformers (BERT) and our proposed weighted GCN (WGCN). The latter assigns a score to each dependency edge within a sentence so as to capture rich syntactic features and determine the most influential edges for extracting ADE relations. Finally, the system employs a multi-head attention to exchange boundary knowledge across levels. We evaluate ADERel on two benchmark datasets from TAC 2017 and n2c2 2018 shared tasks. The experimental results show that ADERel is superior in performance compared with several state-of-the-art methods. The results also demonstrate that incorporating a transformer model with WGCN makes the proposed system more effective for extracting various types of ADE relations. The evaluations further highlight that ADERel takes advantage of joint learning, showing its effectiveness in recognizing complex relations.

NLIRE: A Natural Language Inference method for Relation Extraction

Relation extraction task aims at detecting the semantic relation between a pair of entities in a given target sentence. However, previous methods lack the description of the relation definition, thus needing to model the implication of relations during training. To tackle this issue, we propose a natural language inference method for relation extraction. Given a premise and a hypothesis, the natural language inference task refers to predicting whether the facts in the premise necessarily imply the facts in the hypothesis. Specifically, for each relation type, we construct a relation description. These relation descriptions are the definition of relation, containing prior knowledge that helps model understand the meaning of relation. The given target sentence is viewed as the premise, and these descriptions are viewed as the hypotheses. Then model infers whether these hypotheses can be concluded from the premise. Based on the inference results, our model selects the relation corresponding to the most confident hypothesis as the prediction. Substantial experiments on SemEval2010 Task8 dataset demonstrate that the proposed method achieves state-of-the-art performance.

Drug knowledge discovery via multi-task learning and pre-trained models

BackgroundDrug repurposing is to find new indications of approved drugs, which is essential for investigating new uses for approved or investigational drug efficiency. The active gene annotation corpus (named AGAC) is annotated by human experts, which was developed to support knowledge discovery for drug repurposing. The AGAC track of the BioNLP Open Shared Tasks using this corpus is organized by EMNLP-BioNLP 2019, where the “Selective annotation” attribution makes AGAC track more challenging than other traditional sequence labeling tasks. In this work, we show our methods for trigger word detection (Task 1) and its thematic role identification (Task 2) in the AGAC track. As a step forward to drug repurposing research, our work can also be applied to large-scale automatic extraction of medical text knowledge.MethodsTo meet the challenges of the two tasks, we consider Task 1 as the medical name entity recognition (NER), which cultivates molecular phenomena related to gene mutation. And we regard Task 2 as a relation extraction task, which captures the thematic roles between entities. In this work, we exploit pre-trained biomedical language representation models (e.g., BioBERT) in the information extraction pipeline for mutation-disease knowledge collection from PubMed. Moreover, we design the fine-tuning framework by using a multi-task learning technique and extra features. We further investigate different approaches to consolidate and transfer the knowledge from varying sources and illustrate the performance of our model on the AGAC corpus. Our approach is based on fine-tuned BERT, BioBERT, NCBI BERT, and ClinicalBERT using multi-task learning. Further experiments show the effectiveness of knowledge transformation and the ensemble integration of models of two tasks. We conduct a performance comparison of various algorithms. We also do an ablation study on the development set of Task 1 to examine the effectiveness of each component of our method.ResultsCompared with competitor methods, our model obtained the highest Precision (0.63), Recall (0.56), and F-score value (0.60) in Task 1, which ranks first place. It outperformed the baseline method provided by the organizers by 0.10 in F-score. The model shared the same encoding layers for the named entity recognition and relation extraction parts. And we obtained a second high F-score (0.25) in Task 2 with a simple but effective framework.ConclusionsExperimental results on the benchmark annotation of genes with active mutation-centric function changes corpus show that integrating pre-trained biomedical language representation models (i.e., BERT, NCBI BERT, ClinicalBERT, BioBERT) into a pipe of information extraction methods with multi-task learning can improve the ability to collect mutation-disease knowledge from PubMed.

SrBERT: automatic article classification model for systematic review using BERT

BackgroundSystematic reviews (SRs) are recognized as reliable evidence, which enables evidence-based medicine to be applied to clinical practice. However, owing to the significant efforts required for an SR, its creation is time-consuming, which often leads to out-of-date results. To support SR tasks, tools for automating these SR tasks have been considered; however, applying a general natural language processing model to domain-specific articles and insufficient text data for training poses challenges.MethodsThe research objective is to automate the classification of included articles using the Bidirectional Encoder Representations from Transformers (BERT) algorithm. In particular, srBERT models based on the BERT algorithm are pre-trained using abstracts of articles from two types of datasets, and the resulting model is then fine-tuned using the article titles. The performances of our proposed models are compared with those of existing general machine-learning models.ResultsOur results indicate that the proposed srBERTmy model, pre-trained with abstracts of articles and a generated vocabulary, achieved state-of-the-art performance in both classification and relation-extraction tasks; for the first task, it achieved an accuracy of 94.35% (89.38%), F1 score of 66.12 (78.64), and area under the receiver operating characteristic curve of 0.77 (0.9) on the original and (generated) datasets, respectively. In the second task, the model achieved an accuracy of 93.5% with a loss of 27%, thereby outperforming the other evaluated models, including the original BERT model.ConclusionsOur research shows the possibility of automatic article classification using machine-learning approaches to support SR tasks and its broad applicability. However, because the performance of our model depends on the size and class ratio of the training dataset, it is important to secure a dataset of sufficient quality, which may pose challenges.

Extracting Entity Synonymous Relations via Context-Aware Permutation Invariance

Discovering entity synonymous relations is an important work for many entity-based applications. Existing entity synonymous relation extraction approaches are mainly based on lexical patterns or distributional corpus-level statistics, ignoring the context semantics between entities. For example, the contexts around ''apple'' determine whether ''apple'' is a kind of fruit or Apple Inc. In this paper, an entity synonymous relation extraction approach is proposed using context-aware permutation invariance. Specifically, a triplet network is used to obtain the permutation invariance between the entities to learn whether two given entities possess synonymous relation. To track more synonymous features, the relational context semantics and entity representations are integrated into the triplet network, which can improve the performance of extracting entity synonymous relations. The proposed approach is implemented on three real-world datasets. Experimental results demonstrate that the approach performs better than the other compared approaches on entity synonymous relation extraction task.

Deep learning models in detection of dietary supplement adverse event signals from Twitter.

ObjectiveThe objective of this study is to develop a deep learning pipeline to detect signals on dietary supplement-related adverse events (DS AEs) from Twitter.Materials and MethodsWe obtained 247 807 tweets ranging from 2012 to 2018 that mentioned both DS and AE. We designed a tailor-made annotation guideline for DS AEs and annotated biomedical entities and relations on 2000 tweets. For the concept extraction task, we fine-tuned and compared the performance of BioClinical-BERT, PubMedBERT, ELECTRA, RoBERTa, and DeBERTa models with a CRF classifier. For the relation extraction task, we fine-tuned and compared BERT models to BioClinical-BERT, PubMedBERT, RoBERTa, and DeBERTa models. We chose the best-performing models in each task to assemble an end-to-end deep learning pipeline to detect DS AE signals and compared the results to the known DS AEs from a DS knowledge base (ie, iDISK).ResultsDeBERTa-CRF model outperformed other models in the concept extraction task, scoring a lenient microaveraged F1 score of 0.866. RoBERTa model outperformed other models in the relation extraction task, scoring a lenient microaveraged F1 score of 0.788. The end-to-end pipeline built on these 2 models was able to extract DS indication and DS AEs with a lenient microaveraged F1 score of 0.666.ConclusionWe have developed a deep learning pipeline that can detect DS AE signals from Twitter. We have found DS AEs that were not recorded in an existing knowledge base (iDISK) and our proposed pipeline can as sist DS AE pharmacovigilance.