Tree-structured Long Short-term Memory Research Articles

Prediction of glycopeptide fragment mass spectra by deep learning

Deep learning has achieved a notable success in mass spectrometry-based proteomics and is now emerging in glycoproteomics. While various deep learning models can predict fragment mass spectra of peptides with good accuracy, they cannot cope with the non-linear glycan structure in an intact glycopeptide. Herein, we present DeepGlyco, a deep learning-based approach for the prediction of fragment spectra of intact glycopeptides. Our model adopts tree-structured long-short term memory networks to process the glycan moiety and a graph neural network architecture to incorporate potential fragmentation pathways of a specific glycan structure. This feature is beneficial to model explainability and differentiation ability of glycan structural isomers. We further demonstrate that predicted spectral libraries can be used for data-independent acquisition glycoproteomics as a supplement for library completeness. We expect that this work will provide a valuable deep learning resource for glycoproteomics.

Learning tree-structured representation for 3D coronary artery segmentation

Extensive research has been devoted to the segmentation of the coronary artery. However, owing to its complex anatomical structure, it is extremely challenging to automatically segment the coronary artery from 3D coronary computed tomography angiography (CCTA). Inspired by recent ideas to use tree-structured long short-term memory (LSTM) to model the underlying tree structures for NLP tasks, we propose a novel tree-structured convolutional gated recurrent unit (ConvGRU) model to learn the anatomical structure of the coronary artery. However, unlike tree-structured LSTM proposed for semantic relatedness as well as sentiment classification in natural language processing, our tree-structured ConvGRU model considers the local spatial correlations in the input data as the convolutions are used for input-to-state as well as state-to-state transitions, thus more suitable for image analysis. To conduct voxel-wise segmentation, a tree-structured segmentation framework is presented. It consists of a fully convolutional network (FCN) for multi-scale discriminative feature extraction and the final prediction, and a tree-structured ConvGRU layer for anatomical structure modeling. The proposed framework is extensively evaluated on four large-scale 3D CCTA dataset (the largest to the best of our knowledge), and experiments show that our method is more accurate as well as efficient, compared with other coronary artery segmentation approaches.

Semantic relation extraction using sequential and tree-structured LSTM with attention

Semantic relation extraction is crucial to automatically constructing a knowledge graph (KG), and it supports a variety of downstream natural language processing (NLP) tasks such as query answering (QA), semantic search and textual entailment. In addition, the semantic relation extraction task is mainly responsible for identifying entity pairs from raw texts and extracting the semantic relations between the extracted entity pairs. Existing methods consider only lexical-level features and often ignore syntactic features, resulting in poor relation extraction performance. By analyzing the necessity of the syntactic dependency and the contributions of words in a sentence to relation extraction, this paper proposes an end-to-end method that uses bidirectional tree-structured long short-term memory (LSTM) to extract structural features based on the dependency tree of a sentence. To enhance the performance of the relation extraction, the bidirectional sequential LSTM with attention is used to identify word-based features including the positional information of entity pairs and the contribution of words. Then, structural features and word-based features are concatenated to optimize the relation extraction performance. Finally, the proposed method is used on the SemEval 2010 task 8 and the CoNLL04 datasets to validate its performance. The experimental results show that the proposed method achieves state-of-the-art results on the SemEval 2010 task 8 and the CoNLL04 datasets.

An architecture of deep learning in QSPR modeling for the prediction of critical properties using molecular signatures

AbstractDeep learning rapidly promotes many fields with successful stories in natural language processing. An architecture of deep neural network (DNN) combining tree‐structured long short‐term memory (Tree‐LSTM) network and back‐propagation neural network (BPNN) is developed for predicting physical properties. Inspired by the natural language processing in artificial intelligence, we first developed a strategy for data preparation including encoding molecules with canonical molecular signatures and vectorizing bond‐substrings by an embedding algorithm. Then, the dynamic neural network named Tree‐LSTM is employed to depict molecular tree data‐structures while the BPNN is used to correlate properties. To evaluate the performance of proposed DNN, the critical properties of nearly 1,800 compounds are employed for training and testing the DNN models. As compared with classical group contribution methods, it can be demonstrated that the learned DNN models are able to provide more accurate prediction and cover more diverse molecular structures without considering frequencies of substructures.

Learning document representation via topic-enhanced LSTM model

Document representation plays an important role in the fields of text mining, natural language processing, and information retrieval. Traditional approaches to document representation may suffer from the disregard of the correlations or order of words in a document, due to unrealistic assumption of word independence or exchangeability. Recently, long–short-term memory (LSTM) based recurrent neural networks have been shown effective in preserving local contextual sequential patterns of words in a document, but using the LSTM model alone may not be adequate to capture global topical semantics for learning document representation. In this work, we propose a new topic-enhanced LSTM model to deal with the document representation problem. We first employ an attention-based LSTM model to generate hidden representation of word sequence in a given document. Then, we introduce a latent topic modeling layer with similarity constraint on the local hidden representation, and build a tree-structured LSTM on top of the topic layer for generating semantic representation of the document. We evaluate our model in typical text mining applications, i.e., document classification, topic detection, information retrieval, and document clustering. Experimental results on real-world datasets show the benefit of our innovations over state-of-the-art baseline methods.

Tree-Structured Neural Networks With Topic Attention for Social Emotion Classification

Social emotion classification studies the emotion distribution evoked by an article among numerous readers. Although recently neural network-based methods can improve the classification performance compared with the previous word-emotion and topic-emotion approaches, they have not fully utilized some important sentence language features and document topic features. In this paper, we propose a new neural network architecture exploiting both the syntactic information of a sentence and topic distribution of a document. The proposed architecture first constructs a tree-structured long short-term memory (Tree-LSTM) network based on the sentence syntactic dependency tree to obtain a sentence vector representation. For a multi-sentence document, we then use a Chain-LSTM network to obtain the document representation from its sentences’ hidden states. Furthermore, we design a topic-based attention mechanism with two attention levels. The word-level attention is used for weighting words of a single-sentence document and the sentence-level attention for weighting sentences of a multi-sentence document. The experiments on three public datasets show that the proposed scheme outperforms the state-of-the-art ones in terms of higher average Pearson correlation coefficient and MicroF1 performance.

A Sequential Neural Encoder With Latent Structured Description for Modeling Sentences

In this paper, we propose a sequential neural encoder with latent structured description (SNELSD) for modeling sentences. This model introduces latent chunk-level representations into conventional sequential neural encoders, i.e., recurrent neural networks (RNNs) with long short-term memory (LSTM) units, to consider the compositionality of languages in semantic modeling. An SNELSD model has a hierarchical structure that includes a detection layer and a description layer. The detection layer predicts the boundaries of latent word chunks in an input sentence and derives a chunk-level vector for each word. The description layer utilizes modified LSTM units to process these chunk-level vectors in a recurrent manner and produces sequential encoding outputs. These output vectors are further concatenated with word vectors or the outputs of a chain LSTM encoder to obtain the final sentence representation. All the model parameters are learned in an end-to-end manner without a dependency on additional text chunking or syntax parsing. A natural language inference (NLI) task and a sentiment analysis (SA) task are adopted to evaluate the performance of our proposed model. The experimental results demonstrate the effectiveness of the proposed SNELSD model on exploring task-dependent chunking patterns during the semantic modeling of sentences. Furthermore, the proposed method achieves better performance than conventional chain LSTMs and tree-structured LSTMs on both tasks.

Encoding Syntactic Knowledge in Neural Networks for Sentiment Classification

Phrase/Sentence representation is one of the most important problems in natural language processing. Many neural network models such as Convolutional Neural Network (CNN), Recursive Neural Network (RNN), and Long Short-Term Memory (LSTM) have been proposed to learn representations of phrase/sentence, however, rich syntactic knowledge has not been fully explored when composing a longer text from its shorter constituent words. In most traditional models, only word embeddings are utilized to compose phrase/sentence representations, while the syntactic information of words is yet to be explored. In this article, we discover that encoding syntactic knowledge (part-of-speech tag) in neural networks can enhance sentence/phrase representation. Specifically, we propose to learn tag-specific composition functions and tag embeddings in recursive neural networks, and propose to utilize POS tags to control the gates of tree-structured LSTM networks. We evaluate these models on two benchmark datasets for sentiment classification, and demonstrate that improvements can be obtained with such syntactic knowledge encoded.