Text Mining Tasks Research Articles

AN IMPROVED MULTI-VERSE OPTIMIZER FOR TEXT DOCUMENTS CLUSTERING

Text document clustering (TDC) represents a key task in text mining and unsupervised machine learning, which partitions a specific documents’ collection into varied K-groups according to certain similarity/dissimilarity criterion. The multi-verse optimizer algorithm (MVO) is a stochastic population-based algorithm, which was recently introduced and successfully utilized to tackle many optimization problems that are complex. The original MVO performance is limited to the utilization of only the best solution in the exploitation phase (local search capability), which makes it suffer from entrapment in local optima and low convergence rate. This paper aims to propose a novel method of modifying the MVO algorithm called link-based multi-verse optimizer algorithm (LBMVO) to enhance the exploitation phase in the original MVO. Generally, LBMVO has outperformed or at least showed that it is profoundly competitive compared with the original MVO algorithm.

Distinguished representation of identical mentions in bio-entity coreference resolution

BackgroundBio-entity Coreference Resolution (CR) is a vital task in biomedical text mining. An important issue in CR is the differential representation of identical mentions as their similar representations may make the coreference more puzzling. However, when extracting features, existing neural network-based models may bring additional noise to the distinction of identical mentions since they tend to get similar or even identical feature representations.MethodsWe propose a context-aware feature attention model to distinguish similar or identical text units effectively for better resolving coreference. The new model can represent the identical mentions based on different contexts by adaptively exploiting features, which enables the model reduce the text noise and capture the semantic information effectively.ResultsThe experimental results show that the proposed model brings significant improvements on most of the baseline for coreference resolution and mention detection on the BioNLP dataset and CRAFT-CR dataset. The empirical studies further demonstrate its superior performance on the differential representation and coreferential link of identical mentions.ConclusionsIdentical mentions impose difficulties on the current methods of Bio-entity coreference resolution. Thus, we propose the context-aware feature attention model to better distinguish identical mentions and achieve superior performance on both coreference resolution and mention detection, which will further improve the performance of the downstream tasks.

Lexicon-Enhanced Multi-Task Convolutional Neural Network for Emotion Distribution Learning

Emotion distribution learning (EDL) handles emotion fuzziness by means of the emotion distribution, which is an emotion vector that quantitatively represents a set of emotion categories with their intensity of a given instance. Despite successful applications of EDL in many practical emotion analysis tasks, existing EDL methods have seldom considered the linguistic prior knowledge of affective words specific to the text mining task. To address the problem, this paper proposes a text emotion distribution learning model based on a lexicon-enhanced multi-task convolutional neural network (LMT-CNN) to jointly solve the tasks of text emotion distribution prediction and emotion label classification. The LMT-CNN model designs an end-to-end multi-module deep neural network to utilize both semantic information and linguistic knowledge. Specifically, the architecture of the LMT-CNN model consists of a semantic information module, an emotion knowledge module based on affective words, and a multi-task prediction module to predict emotion distributions and labels. Extensive comparative experiments on nine commonly used emotional text datasets showed that the proposed LMT-CNN model is superior to the compared EDL methods for both emotion distribution prediction and emotion recognition tasks.

MetaboListem and TABoLiSTM: Two Deep Learning Algorithms for Metabolite Named Entity Recognition.

Reviewing the metabolomics literature is becoming increasingly difficult because of the rapid expansion of relevant journal literature. Text-mining technologies are therefore needed to facilitate more efficient literature reviews. Here we contribute a standardised corpus of full-text publications from metabolomics studies and describe the development of two metabolite named entity recognition (NER) methods. These methods are based on Bidirectional Long Short-Term Memory (BiLSTM) networks and each incorporate different transfer learning techniques (for tokenisation and word embedding). Our first model (MetaboListem) follows prior methodology using GloVe word embeddings. Our second model exploits BERT and BioBERT for embedding and is named TABoLiSTM (Transformer-Affixed BiLSTM). The methods are trained on a novel corpus annotated using rule-based methods, and evaluated on manually annotated metabolomics articles. MetaboListem (F1-score 0.890, precision 0.892, recall 0.888) and TABoLiSTM (BioBERT version: F1-score 0.909, precision 0.926, recall 0.893) have achieved state-of-the-art performance on metabolite NER. A training corpus with full-text sentences from >1000 full-text Open Access metabolomics publications with 105,335 annotated metabolites was created, as well as a manually annotated test corpus (19,138 annotations). This work demonstrates that deep learning algorithms are capable of identifying metabolite names accurately and efficiently in text. The proposed corpus and NER algorithms can be used for metabolomics text-mining tasks such as information retrieval, document classification and literature-based discovery and are available from the omicsNLP GitHub repository.

Neuroimaging-ITM: A Text Mining Pipeline Combining Deep Adversarial Learning with Interaction Based Topic Modeling for Enabling the FAIR Neuroimaging Study.

Sharing various neuroimaging digital resources have received widespread attention in FAIR (Findable, Accessible, Interoperable and Reusable) neuroscience. In order to support a comprehensive understanding of brain cognition, neuroimaging provenance should be constructed to characterize both research processes and results, and integrates various digital resources for quick replication and open cooperation. This brings new challenges to neuroimaging text mining, including fragmented information, lack of labelled corpora, and vague topics. This paper proposes a text mining pipeline for enabling the FAIR neuroimaging study. In order to avoid fragmented information, the Brain Informatics provenance model is redesigned based on NIDM (Neuroimaging Data Model) and FAIR facets. It can systematically capture the provenance requests from the FAIR neuroimaging study and then transform them into a group of text mining tasks. A neuroimaging text mining pipeline combining deep adversarial learning with interaction based topic modeling, called neuroimaging interaction topic model (Neuroimaging-ITM), is proposed to automatically extract neuroimaging provenance and identify research topics in the few-shot scenario. Finally, a group of experiments is completed by using real data from the journal PloS One. The experimental results show that Neuroimaging-ITM can systematically and accurately extract provenance information and obtain high-quality research topics from the full text of neuroimaging articles. Most of the mean F1 values of provenance extraction exceed 0.9. The topic coherence and KL (Kullback-Leibler) divergence reach 9.95 and 0.96 respectively. The results are obviously better than baseline methods.

Auto-CORPus: A Natural Language Processing Tool for Standardizing and Reusing Biomedical Literature.

To analyse large corpora using machine learning and other Natural Language Processing (NLP) algorithms, the corpora need to be standardized. The BioC format is a community-driven simple data structure for sharing text and annotations, however there is limited access to biomedical literature in BioC format and a lack of bioinformatics tools to convert online publication HTML formats to BioC. We present Auto-CORPus (Automated pipeline for Consistent Outputs from Research Publications), a novel NLP tool for the standardization and conversion of publication HTML and table image files to three convenient machine-interpretable outputs to support biomedical text analytics. Firstly, Auto-CORPus can be configured to convert HTML from various publication sources to BioC. To standardize the description of heterogenous publication sections, the Information Artifact Ontology is used to annotate each section within the BioC output. Secondly, Auto-CORPus transforms publication tables to a JSON format to store, exchange and annotate table data between text analytics systems. The BioC specification does not include a data structure for representing publication table data, so we present a JSON format for sharing table content and metadata. Inline tables within full-text HTML files and linked tables within separate HTML files are processed and converted to machine-interpretable table JSON format. Finally, Auto-CORPus extracts abbreviations declared within publication text and provides an abbreviations JSON output that relates an abbreviation with the full definition. This abbreviation collection supports text mining tasks such as named entity recognition by including abbreviations unique to individual publications that are not contained within standard bio-ontologies and dictionaries. The Auto-CORPus package is freely available with detailed instructions from GitHub at: https://github.com/omicsNLP/Auto-CORPus.

Granularity-aware Area Prototypical Network with Bimargin Loss for Few Shot Relation Classification

Relation Classification is one of the most important tasks in text mining. Previous methods either require large-scale manually-annotated data or rely on distant supervision approaches which suffer from the long-tail problem. To reduce the expensive manually-annotating cost and solve the long-tail problem, prototypical networks are widely used in few-shot RC tasks. Despite their remarkable performance, current prototypical networks ignore the different granularities of relations, which degrades the classification performance dramatically. Moreover, the optimization of current prototypical networks simply relies on the cross-entropy loss, which cannot consider the intra-relation compactness and the dispersion among relations in a semantic space. It is not robust enough for current prototypical network in real-world and complicated scenarios. In this paper, we propose an area prototypical network with a granularity-aware measurement, aiming to considering the different granularities of relations. Each relation is represented as an area whose width can reflect the granularity level of relation. Moreover, to improve the robustness, bimargin loss is designed to force area prototypical network to improve the intra-relation compactness and inter-relation dispersion for the feature representation in a semantic space. Extensive experiments on two public datasets are conducted and evaluate the effectiveness of our proposed model.

Biomedical Literature Mining and Its Components.

The published biomedical articles are the best source of knowledge to understand the importance of biomedical entities such as disease, drugs, and their role in different patient population groups. The number of biomedical literature available and being published is increasing at an exponential rate with the use of large scale experimental techniques. Manual extraction of such information is becoming extremely difficult because of the huge number of biomedical literature available. Alternatively, text mining approaches receive much interest within biomedicine by providing automatic extraction of such information in more structured format from the unstructured biomedical text. Here, a text mining protocol to extract the patient population information, to identify the disease and drug mentions in PubMed titles and abstracts, and a simple information retrieval approach to retrieve a list of relevant documents for a user query are presented. The text mining protocol presented in this chapter is useful for retrieving information on drugs for patients with a specific disease. The protocol covers three major text mining tasks, namely, information retrieval, information extraction, and knowledge discovery.

BatteryDataExtractor: battery-aware text-mining software embedded with BERT models.

Due to the massive growth of scientific publications, literature mining is becoming increasingly popular for researchers to thoroughly explore scientific text and extract such data to create new databases or augment existing databases. Efforts in literature-mining software design and implementation have improved text-mining productivity, but most of the toolkits that mine text are based on traditional machine-learning-algorithms which hinder the performance of downstream text-mining tasks. Natural-language processing (NLP) and text-mining technologies have seen a rapid development since the release of transformer models, such as bidirectional encoder representations from transformers (BERT). Upgrading rule-based or machine-learning-based literature-mining toolkits by embedding transformer models into the software is therefore likely to improve their text-mining performance. To this end, we release a Python-based literature-mining toolkit for the field of battery materials, BatteryDataExtractor, which involves the embedding of BatteryBERT models in its automated data-extraction pipeline. This pipeline employs BERT models for token-classification tasks, such as abbreviation detection, part-of-speech tagging, and chemical-named-entity recognition, as well as new double-turn question-answering data-extraction models for auto-generating repositories of inter-related material and property data as well as general information. We demonstrate that BatteryDataExtractor exhibits state-of-the-art performance on the evaluation data sets for both token classification and automated data extraction. To aid the use of BatteryDataExtractor, its code is provided as open-source software, with associated documentation to serve as a user guide.

Improving deep learning method for biomedical named entity recognition by using entity definition information

BackgroundBiomedical named entity recognition (NER) is a fundamental task of biomedical text mining that finds the boundaries of entity mentions in biomedical text and determines their entity type. To accelerate the development of biomedical NER techniques in Spanish, the PharmaCoNER organizers launched a competition to recognize pharmacological substances, compounds, and proteins. Biomedical NER is usually recognized as a sequence labeling task, and almost all state-of-the-art sequence labeling methods ignore the meaning of different entity types. In this paper, we investigate some methods to introduce the meaning of entity types in deep learning methods for biomedical NER and apply them to the PharmaCoNER 2019 challenge. The meaning of each entity type is represented by its definition information.Material and methodWe investigate how to use entity definition information in the following two methods: (1) SQuad-style machine reading comprehension (MRC) methods that treat entity definition information as query and biomedical text as context and predict answer spans as entities. (2) Span-level one-pass (SOne) methods that predict entity spans of one type by one type and introduce entity type meaning, which is represented by entity definition information. All models are trained and tested on the PharmaCoNER 2019 corpus, and their performance is evaluated by strict micro-average precision, recall, and F1-score.ResultsEntity definition information brings improvements to both SQuad-style MRC and SOne methods by about 0.003 in micro-averaged F1-score. The SQuad-style MRC model using entity definition information as query achieves the best performance with a micro-averaged precision of 0.9225, a recall of 0.9050, and an F1-score of 0.9137, respectively. It outperforms the best model of the PharmaCoNER 2019 challenge by 0.0032 in F1-score. Compared with the state-of-the-art model without using manually-crafted features, our model obtains a 1% improvement in F1-score, which is significant. These results indicate that entity definition information is useful for deep learning methods on biomedical NER.ConclusionOur entity definition information enhanced models achieve the state-of-the-art micro-average F1 score of 0.9137, which implies that entity definition information has a positive impact on biomedical NER detection. In the future, we will explore more entity definition information from knowledge graph.

ChemTables: a dataset for semantic classification on tables in chemical patents

Chemical patents are a commonly used channel for disclosing novel compounds and reactions, and hence represent important resources for chemical and pharmaceutical research. Key chemical data in patents is often presented in tables. Both the number and the size of tables can be very large in patent documents. In addition, various types of information can be presented in tables in patents, including spectroscopic and physical data, or pharmacological use and effects of chemicals. Since images of Markush structures and merged cells are commonly used in these tables, their structure also shows substantial variation. This heterogeneity in content and structure of tables in chemical patents makes relevant information difficult to find. We therefore propose a new text mining task of automatically categorising tables in chemical patents based on their contents. Categorisation of tables based on the nature of their content can help to identify tables containing key information, improving the accessibility of information in patents that is highly relevant for new inventions. For developing and evaluating methods for the table classification task, we developed a new dataset, called ChemTables, which consists of 788 chemical patent tables with labels of their content type. We introduce this data set in detail. We further establish strong baselines for the table classification task in chemical patents by applying state-of-the-art neural network models developed for natural language processing, including TabNet, ResNet and Table-BERT on ChemTables. The best performing model, Table-BERT, achieves a performance of 88.66 micro-averaged F_1 score on the table classification task. The ChemTables dataset is publicly available at https://doi.org/10.17632/g7tjh7tbrj.3, subject to the CC BY NC 3.0 license. Code/models evaluated in this work are in a Github repository https://github.com/zenanz/ChemTables.

Improving legal judgment prediction through reinforced criminal element extraction

Legal text mining is targeted at automatically analyzing the texts in the legal domain by employing various natural language processing techniques and has attracted enormous attention from the NLP community. As one of the most crucial tasks of legal text mining, Legal Judgment Prediction (LJP) aims to automatically predict judgment results (e.g., applicable law articles, charges, and terms of penalty) according to fact descriptions on law cases and becomes a promising application of artificial intelligence techniques.Unfortunately, ambiguous fact descriptions and law articles often appear due to a great number of shared words and legal concepts. Prior works are proposed to partially address these problems, focusing on introducing additional attributes to distinguish similar fact descriptions, or differentiating confusing law articles by grouping and distilling law articles. However, existing works still face two severe challenges: (1) indistinguishable fact descriptions with different criminals and targets and (2) misleading law articles with highly similar TF–IDF representations, both of which lead to serious misjudgments for the LJP task. In this paper, we present a novel reinforcement learning (RL) based framework, named Criminal Element Extraction Network (CEEN), to handle above challenges simultaneously. In CEEN, we propose four types of discriminative criminal elements, including the criminal, target, intentionality, and criminal behavior. To discriminate ambiguous fact descriptions, an reinforcement learning based extractor is designed to accurately locate elements for different cases. To enhance law article predictions, distinctive element representations are constructed for each type of criminal element. Finally, with the input of element representations, a multi-task predictor is adopted for the judgment predictions. Experimental results on real-world datasets show that extracting criminal elements is highly useful for predicting the judgment results.

BioVAE: a pre-trained latent variable language model for biomedical text mining.

SummaryLarge-scale pre-trained language models (PLMs) have advanced state-of-the-art (SOTA) performance on various biomedical text mining tasks. The power of such PLMs can be combined with the advantages of deep generative models. These are examples of these combinations. However, they are trained only on general domain text, and biomedical models are still missing. In this work, we describe BioVAE, the first large-scale pre-trained latent variable language model for the biomedical domain, which uses the OPTIMUS framework to train on large volumes of biomedical text. The model shows SOTA performance on several biomedical text mining tasks when compared to existing publicly available biomedical PLMs. In addition, our model can generate more accurate biomedical sentences than the original OPTIMUS output.Availability and implementationOur source code and pre-trained models are freely available: https://github.com/aistairc/BioVAE.Supplementary information Supplementary data are available at Bioinformatics online.

Text Mining for Building Biomedical Networks Using Cancer as a Case Study.

In the assembly of biological networks it is important to provide reliable interactions in an effort to have the most possible accurate representation of real-life systems. Commonly, the data used to build a network comes from diverse high-throughput essays, however most of the interaction data is available through scientific literature. This has become a challenge with the notable increase in scientific literature being published, as it is hard for human curators to track all recent discoveries without using efficient tools to help them identify these interactions in an automatic way. This can be surpassed by using text mining approaches which are capable of extracting knowledge from scientific documents. One of the most important tasks in text mining for biological network building is relation extraction, which identifies relations between the entities of interest. Many interaction databases already use text mining systems, and the development of these tools will lead to more reliable networks, as well as the possibility to personalize the networks by selecting the desired relations. This review will focus on different approaches of automatic information extraction from biomedical text that can be used to enhance existing networks or create new ones, such as deep learning state-of-the-art approaches, focusing on cancer disease as a case-study.

МОДИФИЦИРОВАННЫЙ МЕТОД УСТРАНЕНИЯ НЕОДНОЗНАЧНОСТИ СМЫСЛА СЛОВ, ОСНОВАННЫЙ НА МЕТОДАХ РАСПРЕДЕЛЕННОГО ПРЕДСТАВЛЕНИЯ

In the text mining tasks, textual representation should be not only efficient but also interpretable,as this enables an understanding of the operational logic underlying the data miningmodels. This paper describes a modified Word Sense Disambiguation (WSD) method which extendstwo well-known variations of the Lesk WSD approach. Given a word and its context, Leskbases its calculations on the overlap between the context of a word and each definition of its senses(gloss) in order to select the proper meaning. The main contribution of the proposed method isthe adoption of the concept of “similarity” between definition and context instead of "overlap", inaddition to expanding the definition with examples provided by WordNet for each sense of thetarget word. The proposed method is also characterized by the use of text similarity measurementfunctions defined in a distributed semantic space. The proposed method has been tested on fivedifferent benchmark datasets for words sense disambiguation tasks and compared with severalbasic methods, including simple Lesk, extended Lesk, WordNet 1st sense, Babelfy and UKB. Theresults show that proposed method outperforms most basic methods with the exception of Babelfyand the WN 1st sense methods.

Novel Approach to Transform Unstructured Healthcare Data to Structured Data

With the rapid growth in number and dimension of databases and database applications in Healthcare records, it is necessary to design a system to achieve automatic extraction of facts from huge table. At the same point, there is a provocation in controlling unstructured data as it highly difficult to analyze and extract actionable intelligence. Preprocessing is an important task and critical step in Text Mining, Regular Expression and Information retrieval. The accession of key data from unstructured data is often difficult. The objective of this project is to transform the unstructured healthcare data to structured data particularly to gain perception and to generate appropriate structured data.

Deep learning methods for biomedical named entity recognition: a survey and qualitative comparison.

The biomedical literature is growing rapidly, and the extraction of meaningful information from the large amount of literature is increasingly important. Biomedical named entity (BioNE) identification is one of the critical and fundamental tasks in biomedical text mining. Accurate identification of entities in the literature facilitates the performance of other tasks. Given that an end-to-end neural network can automatically extract features, several deep learning-based methods have been proposed for BioNE recognition (BioNER), yielding state-of-the-art performance. In this review, we comprehensively summarize deep learning-based methods for BioNER and datasets used in training and testing. The deep learning methods are classified into four categories: single neural network-based, multitask learning-based, transfer learning-based and hybrid model-based methods. They can be applied to BioNER in multiple domains, and the results are determined by the dataset size and type. Lastly, we discuss the future development and opportunities of BioNER methods.

Text Mining in Cybersecurity

The growth of data volume has changed cybersecurity activities, demanding a higher level of automation. In this new cybersecurity landscape, text mining emerged as an alternative to improve the efficiency of the activities involving unstructured data. This article proposes a Systematic Literature Review ( SLR ) to present the application of text mining in the cybersecurity domain. Using a systematic protocol, we identified 2,196 studies, out of which 83 were summarized. As a contribution, we propose a taxonomy to demonstrate the different activities in the cybersecurity domain supported by text mining. We also detail the strategies evaluated in the application of text mining tasks and the use of neural networks to support activities involving unstructured data. The work also discusses text classification performance aiming its application in real-world solutions. The SLR also highlights open gaps for future research, such as the analysis of non-English content and the intensification in the usage of neural networks.

ВЕКТОРИЗАЦИЯ ТЕКСТА С ИСПОЛЬЗОВАНИЕМ МЕТОДОВ ИНТЕЛЛЕКТУАЛЬНОГО АНАЛИЗА ДАННЫХ

In the text mining tasks, textual representation should be not only efficient but also interpretable,as this enables an understanding of the operational logic underlying the data miningmodels. Traditional text vectorization methods such as TF-IDF and bag-of-words are effective andcharacterized by intuitive interpretability, but suffer from the «curse of dimensionality», and theyare unable to capture the meanings of words. On the other hand, modern distributed methods effectivelycapture the hidden semantics, but they are computationally intensive, time-consuming,and uninterpretable. This article proposes a new text vectorization method called Bag of weightedConcepts BoWC that presents a document according to the concepts’ information it contains. Theproposed method creates concepts by clustering word vectors (i.e. word embedding) then uses thefrequencies of these concept clusters to represent document vectors. To enrich the resulted documentrepresentation, a new modified weighting function is proposed for weighting concepts basedon statistics extracted from word embedding information. The generated vectors are characterizedby interpretability, low dimensionality, high accuracy, and low computational costs when used indata mining tasks. The proposed method has been tested on five different benchmark datasets intwo data mining tasks; document clustering and classification, and compared with several baselines,including Bag-of-words, TF-IDF, Averaged GloVe, Bag-of-Concepts, and VLAC. The resultsindicate that BoWC outperforms most baselines and gives 7 % better accuracy on average

MLAF: Multi-Level Attention Fused Model for Sentence Similarity Measurement

Semantic textual similarity measurement is an essential task in text mining and natural language processing, which has profound research value. Recent LSTM-based methods have achieved attractive success by modelling sentences as fixed-length vectors. However, since the LSTM network can only extract the semantic information of sentences sequentially, the longterm dependency of sentences cannot be efficiently captured when the sequence length is long. Moreover, current models treat each syntactic component in the sentence equally, while the effects of different syntactic elements to the sentence meaning expression are different. Generally, nouns, verbs, and adjectives contribute more than prepositions and conjunctions to the semantic meaning of the sentence. In this paper, we propose a multi-level attention fused (MLAF) model, which correlates self-attention and word attention with sentence representations, extracting the semantic information of sentences from different levels. The self-attention supplements sequential operations to model long-term dependency of sentences. The word attention built upon raw sentence highlights the vital syntactic components of sentences. Experiments on different datasets demonstrate that the proposed MLAF model has achieved significant improvements over baselines.