Character-level Recurrent Neural Network Research Articles

Biomedical named entity recognition based on multi-cross attention feature fusion.

Currently, in the field of biomedical named entity recognition, CharCNN (Character-level Convolutional Neural Networks) or CharRNN (Character-level Recurrent Neural Network) is typically used independently to extract character features. However, this approach does not consider the complementary capabilities between them and only concatenates word features, ignoring the feature information during the process of word integration. Based on this, this paper proposes a method of multi-cross attention feature fusion. First, DistilBioBERT and CharCNN and CharLSTM are used to perform cross-attention word-char (word features and character features) fusion separately. Then, the two feature vectors obtained from cross-attention fusion are fused again through cross-attention to obtain the final feature vector. Subsequently, a BiLSTM is introduced with a multi-head attention mechanism to enhance the model's ability to focus on key information features and further improve model performance. Finally, the output layer is used to output the final result. Experimental results show that the proposed model achieves the best F1 values of 90.76%, 89.79%, 94.98%, 80.27% and 88.84% on NCBI-Disease, BC5CDR-Disease, BC5CDR-Chem, JNLPBA and BC2GM biomedical datasets respectively. This indicates that our model can capture richer semantic features and improve the ability to recognize entities.

Natural Language Enhancement for English Teaching Using Character-Level Recurrent Neural Network with Back Propagation Neural Network based Classification by Deep Learning Architectures

Natural Language Processing (NLP) is an efficient method for enhancing educational outcomes. In educational settings, implementing NLP entails starting the learning process through natural acquisition. English teaching and learning have received increased attention from the relevant education departments as an integral aspect of the new curriculum reform. The environment of English teaching and learning is undergoing extraordinary changes as a result of the constant improvement and extension of teaching level and scale, as well as the growth of Internet information technology. As a result, the current research aims to look into techniques for efficiently using AI (artificial intelligence) apps to teach and learn English from the perspective of university students. This research can measure the levels as well as effectiveness of the employment of AI applications for teaching English based on deep learning techniques. There, the NLP based language enhancement has been carried out using Character-level recurrent neural network with back Propagation neural network (Cha_RNN_BPNN) based classification. With the help of this DL (deep learning) technique, it is possible to use AI methods to assist teachers in analysing and diagnosing students' English learning behaviour, replacing teachers in part to answer students' questions in a timely manner, and automatically grading assignments during the English teaching process. Experimental analysis shows Word Perplexity, Flesch-Kincaid (F-K) Grade Level for Readability, Cosine Similarity for Semantic Coherence, gradient change of NN, validation accuracy, and training accuracy of the proposed technique.

Categorization of free-text drug orders using character-level recurrent neural networks

Background and purposeManual annotation and categorization of non-standardized text (“free-text”) of drug orders entered into electronic health records is a labor-intensive task. However, standardization is required for drug order analyses and has implications for clinical decision support. Machine learning could help to speed up manual labelling efforts. The objective of this study was to analyze the performance of deep machine learning methods to annotate non-standardized text of drug order entries with their therapeutically active ingredients. Materials and methodsThe data consisted of drug orders entered 8/2009-4/2014 into the electronic health records of inpatients at a large tertiary care academic medical center. We manually annotated the most frequent order entry patterns with the active ingredient they contain (e.g. “Prograf”⟵“Tacrolimus”). We heuristically included additional orders by means of character sequence comparisons to augment the training dataset. Finally, we trained and employed character-level recurrent deep neural networks to classify non-standardized text of drug order entries according to their active ingredients. ResultsA total of 26,611 distinct order patterns were considered in our study, of which the top 7.6% (2028) had been annotated with one of 558 distinct ingredients, leaving 24,583 unlabeled observations. Character-level recurrent deep neural networks achieved a Mean Reciprocal Rank (MRR) of 98% and outperformed the best representative baseline, a trigram-based Support Vector Machine, by 2 percentage points. ConclusionCharacter-level recurrent deep neural networks can be used to map the active ingredient to non-standardized text of drug order entries, outperforming other representative techniques. While machine learning might help to facilitate categorization tasks, still a considerable amount of manual labelling and reviewing work is required to train such systems.

Character-level recurrent neural networks in practice: comparing training and sampling schemes

Recurrent neural networks are nowadays successfully used in an abundance of applications, going from text, speech and image processing to recommender systems. Backpropagation through time is the algorithm that is commonly used to train these networks on specific tasks. Many deep learning frameworks have their own implementation of training and sampling procedures for recurrent neural networks, while there are in fact multiple other possibilities to choose from and other parameters to tune. In existing literature this is very often overlooked or ignored. In this paper we therefore give an overview of possible training and sampling schemes for character-level recurrent neural networks to solve the task of predicting the next token in a given sequence. We test these different schemes on a variety of datasets, neural network architectures and parameter settings, and formulate a number of take-home recommendations. The choice of training and sampling scheme turns out to be subject to a number of trade-offs, such as training stability, sampling time, model performance and implementation effort, but is largely independent of the data. Perhaps the most surprising result is that transferring hidden states for correctly initializing the model on subsequences often leads to unstable training behavior depending on the dataset.

By the Community & For the Community

Community edits to questions and answers (called post edits) plays an important role in improving content quality in Stack Overflow. Our study of post edits in Stack Overflow shows that a large number of edits are about formatting, grammar and spelling. These post edits usually involve small-scale sentence edits and our survey of trusted contributors suggests that most of them care much or very much about such small sentence edits. To assist users in making small sentence edits, we develop an edit-assistance tool for identifying minor textual issues in posts and recommending sentence edits for correction. We formulate the sentence editing task as a machine translation problem, in which an original sentence is "translated" into an edited sentence. Our tool implements a character-level Recurrent Neural Network (RNN) encoder-decoder model, trained with about 6.8 millions original-edited sentence pairs from Stack Overflow post edits. We evaluate our edit assistance tool using a large-scale archival post edits, a field study of assisting a novice post editor, and a survey of trusted contributors. Our evaluation demonstrates the feasibility of training a deep learning model with post edits by the community and then using the trained model to assist post editing for the community.

Extracting Drug-Drug Interactions with Word and Character-Level Recurrent Neural Networks.

Drug-drug interactions (DDIs) are known to be responsible for nearly a third of all adverse drug reactions. Hence several current efforts focus on extracting signal from EMRs to prioritize DDIs that need further exploration. To this end, being able to extract explicit mentions of DDIs in free text narratives is an important task. In this paper, we explore recurrent neural network (RNN) architectures to detect and classify DDIs from unstructured text using the DDIExtraction dataset from the SemEval 2013 (task 9) shared task. Our methods are in line with those used in other recent deep learning efforts for relation extraction including DDI extraction. However, to our knowledge, we are the first to investigate the potential of character-level RNNs (Char-RNNs) for DDI extraction (and relation extraction in general). Furthermore, we explore a simple but effective model bootstrapping method to (a). build model averaging ensembles, (b). derive confidence intervals around mean micro-F scores (MMF), and (c). assess the average behavior of our methods. Without any rule based filtering of negative examples, a popular heuristic used by most earlier efforts, we achieve an MMF of 69.13. By adding simple replicable heuristics to filter negative instances we are able to achieve an MMF of 70.38. Furthermore, our best ensembles produce micro F-scores of 70.81 (without filtering) and 72.13 (with filtering), which are superior to metrics reported in published results. Although Char-RNNs turnout to be inferior to regular word based RNN models in overall comparisons, we find that ensembling models from both architectures results in nontrivial gains over simply using either alone, indicating that they complement each other.

Can Robots Write Treaties? Using Recurrent Neural Networks to Draft International Investment Agreements

Negotiating international investment agreements is costly, complex, and prone to power asymmetries. Would it then not make sense to let computers do part of the work? In this contribution, we train a character-level recurrent neural network (RNN) to write international investment agreements. Benefiting from the formulaic nature of treaty language, the RNN generates texts of lawyer-like quality on the article-level, but fails to compose treaties in a legally sensible manner. By embedding RNNs in a user-controlled pipeline we overcome this problem. First, users can specify the treaty content categories ex ante on which the RNN is trained. Second, the pipeline allows a filtering of output ex post by identifying output that corresponds most closely to a user-selected treaty design benchmark. The result is an improved system that produces meaningful texts with legally sensible composition. We test the pipeline by comparing predicted treaties to actually concluded ones and by verifying that our filter captures latent policy preferences by predicting the outcome of current investment treaty negotiations between China and the United States.

Towards an Automated Production of Legal Texts Using Recurrent Neural Networks

This paper constructs a legal text generation and assembly system in the domain of international investment law. We rely on a corpus of 1600 bilateral investment treaties split into 22 600 articles to train a character-level recurrent neural network (char-RNN). Prior work has shown that while char-RNNs can produce legally meaningful texts, its output tends to be repetitive. In this contribution, we remedy this shortcoming by proposing a new framework for RNN-based text production. First, we elicit priors at the training stage to give more weight to under-represented treaty practice. Second, we use q-gram distance and GloVe word embeddings as filters imposed on the generated texts to draw them closer to a target document. Third, we develop a validation routine that compares the distribution of pre-defined legal concepts in actual and generated texts. Our results indicate that the RNN produces texts that are not repetitive and convey meaningful legal concepts. We conclude by showcasing a practical application of our framework by predicting provisions of the USA-China bilateral investment treaty currently under negotiation.