Abstract

Neural network-based approaches have become the driven forces for Natural Language Processing (NLP) tasks. Conventionally, there are two mainstream neural architectures for NLP tasks: the recurrent neural network (RNN) and the convolution neural network (ConvNet). RNNs are good at modeling long-term dependencies over input texts, but preclude parallel computation. ConvNets do not have memory capability and it has to model sequential data as un-ordered features. Therefore, ConvNets fail to learn sequential dependencies over the input texts, but it is able to carry out high-efficient parallel computation. As each neural architecture, such as RNN and ConvNets, has its own pro and con, integration of different architectures is assumed to be able to enrich the semantic representation of texts, thus enhance the performance of NLP tasks. However, few investigation explores the reconciliation of these seemingly incompatible architectures. To address this issue, we propose a hybrid architecture based on a novel hierarchical multi-granularity attention mechanism, named Multi-granularity Attention-based Hybrid Neural Network (MahNN). The attention mechanism is to assign different weights to different parts of the input sequence to increase the computation efficiency and performance of neural models. In MahNN, two types of attentions are introduced: the syntactical attention and the semantical attention. The syntactical attention computes the importance of the syntactic elements (such as words or sentence) at the lower symbolic level and the semantical attention is used to compute the importance of the embedded space dimension corresponding to the upper latent semantics. We adopt the text classification as an exemplifying way to illustrate the ability of MahNN to understand texts. The experimental results on a variety of datasets demonstrate that MahNN outperforms most of the state-of-the-arts for text classification.

Highlights

  • Nature language understanding plays an critical role in machine intelligence and it includes many challenging Natural Language Processing (NLP) tasks such as reading comprehension [1], machine translation [2], question answering [3] and etc

  • Employ hand-crafted features to model the statistical properties of syntactical elements, which are further fed into the classification algorithms such as k-Nearest Neighbor (k-NN), Random Forests, Support Vector Machines (SVM), or its probabilistic versions [5]–[7]

  • The different neural structure each learns a different aspect of semantic information from the linguistic structures and collectively strengthen the power of semantical understanding of texts. 2) we introduce a novel hierarchical multi-granularity attention mechanism, which includes the syntactical attention and the semantical attention

Read more

Summary

INTRODUCTION

Nature language understanding plays an critical role in machine intelligence and it includes many challenging NLP tasks such as reading comprehension [1], machine translation [2], question answering [3] and etc. Wang et al proposed the convolutional recurrent neural network [23] that stacked four types of neural layers: word embedding, Bidirectional RNN layer, convolutional layer, and max-pooling layer This approach functions very to the one in [22], but with disparate applications in sentence classification and answer selection. The main contributions of our work are listed as follows: 1) We propose a hybrid neural architecture MahNN that, for the first time, seamlessly integrate the RNN architecture and the ConvNet with an attention mechanism. In this architecture, the different neural structure each learns a different aspect of semantic information from the linguistic structures and collectively strengthen the power of semantical understanding of texts.

RELATED WORK
LONG SHORT-TERM MEMORY NETWORK
HIERARCHICAL MULTI-GRANULARITY ATTENTIONS
CONCLUSION AND FUTURE WORK
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call