Word-level Embeddings Research Articles

Cognitive structure learning model for hierarchical multi-label text classification

The human mind grows in learning new knowledge, which finally organizes and develops a basic mental pattern called cognitive structure. Hierarchical multi-label text classification (HMLTC), a fundamental but challenging task in many real-world applications, aims to classify the documents with hierarchical labels to form a resembling cognitive structure learning process. Existing approaches for HMLTC mainly focus on partial new knowledge learning or the global cognitive-structure-like label structure utilization in a cognitive view. However, the complete cognitive structure learning model is a unity that is indispensably constructed by the global label structure utilization and partial knowledge learning, which is ignored among those HMLTC approaches. To address this problem, we will imitate the cognitive structure learning process into the HMLTC learning and propose a unified framework called Hierarchical Cognitive Structure Learning Model (HCSM) in this paper. HCSM is composed of the Attentional Ordered Recurrent Neural Network (AORNN) submodule and Hierarchical Bi-Directional Capsule (HBiCaps) submodule. Both submodules utilize the partial new knowledge and global hierarchical label structure comprehensively for the HMLTC task. On the one hand, AORNN extracts the semantic vector as partial new knowledge from the original text by the word-level and hierarchy-level embedding granularities. On the other hand, AORNN builds the hierarchical text representation learning corresponding to the global label structure by the document-level neurons ordering. HBiCaps employs an iteration to form a unified label categorization process similar to cognitive-structure learning: firstly, using the probability computation of local hierarchical relationships to maintain partial knowledge learning; secondly, modifying the global hierarchical label structure based on the dynamic routing mechanism between capsules. Moreover, the experimental results on four benchmark datasets demonstrate that HCSM outperforms or matches state-of-the-art text classification methods.

Improving the state-of-the-art in Thai semantic similarity using distributional semantics and ontological information.

Research into semantic similarity has a long history in lexical semantics, and it has applications in many natural language processing (NLP) tasks like word sense disambiguation or machine translation. The task of calculating semantic similarity is usually presented in the form of datasets which contain word pairs and a human-assigned similarity score. Algorithms are then evaluated by their ability to approximate the gold standard similarity scores. Many such datasets, with different characteristics, have been created for English language. Recently, four of those were transformed to Thai language versions, namely WordSim-353, SimLex-999, SemEval-2017-500, and R&G-65. Given those four datasets, in this work we aim to improve the previous baseline evaluations for Thai semantic similarity and solve challenges of unsegmented Asian languages (particularly the high fraction of out-of-vocabulary (OOV) dataset terms). To this end we apply and integrate different strategies to compute similarity, including traditional word-level embeddings, subword-unit embeddings, and ontological or hybrid sources like WordNet and ConceptNet. With our best model, which combines self-trained fastText subword embeddings with ConceptNet Numberbatch, we managed to raise the state-of-the-art, measured with the harmonic mean of Pearson on Spearman ρ, by a large margin from 0.356 to 0.688 for TH-WordSim-353, from 0.286 to 0.769 for TH-SemEval-500, from 0.397 to 0.717 for TH-SimLex-999, and from 0.505 to 0.901 for TWS-65.

Improving Loanword Identification in Low-Resource Language with Data Augmentation and Multiple Feature Fusion.

Loanword identification is studied in recent years to alleviate data sparseness in several natural language processing (NLP) tasks, such as machine translation, cross-lingual information retrieval, and so on. However, recent studies on this topic usually put efforts on high-resource languages (such as Chinese, English, and Russian); for low-resource languages, such as Uyghur and Mongolian, due to the limitation of resources and lack of annotated data, loanword identification on these languages tends to have lower performance. To overcome this problem, we first propose a lexical constraint-based data augmentation method to generate training data for low-resource language loanword identification; then, a loanword identification model based on a log-linear RNN is introduced to improve the performance of low-resource loanword identification by incorporating features such as word-level embeddings, character-level embeddings, pronunciation similarity, and part-of-speech (POS) into one model. Experimental results on loanword identification in Uyghur (in this study, we mainly focus on Arabic, Chinese, Russian, and Turkish loanwords in Uyghur) showed that our proposed method achieves best performance compared with several strong baseline systems.

Joint entity and relation extraction model based on rich semantics

Extracting entities and relations from unstructured texts has become an important task in the natural language processing (NLP), especially knowledge graphs (KG). However, relation classification (RC) and named entity recognition (NER) tasks are usually considered separately, which lost a lot of associated contextual information. Therefore, a novel end-to-end method based on the attention mechanism integrating convolutional and recurrent neural networks is proposed for joint entity and relation extraction, which can obtain rich semantics and takes full advantage of the associated information between entities and relations without introducing external complicated features. The convolutional operation is employed to obtain character-level and word-level embeddings which are transferred to the multi-head attention mechanism. Then the multi-head attention mechanism can encode contextual semantics and embeddings to obtain efficient semantic representation. Moreover, the rich semantics are encoded to obtain final tag sequence based on recurrent neural networks. Finally, the experiments are performed on NYT10 and NYT11 benchmarks to demonstrate the proposed method. Compared with the current pipelined and joint approaches, the experimental results indicate that the proposed method can obtain state-of-the-art performance in terms of the standard F1-score.

A hybrid deep learning architecture for opinion-oriented multi-document summarization based on multi-feature fusion

Opinion summarization is a process to produce concise summaries from a large number of opinionated texts. In this paper, we present a novel deep-learning-based method for the generic opinion-oriented extractive summarization of multi-documents (also known as RDLS). The method comprises sentiment analysis embedding space (SAS), text summarization embedding spaces (TSS) and opinion summarizer module (OSM). SAS employs recurrent neural network (RNN) which is composed by long short-term memory (LSTM) to take advantage of sequential processing and overcome several flaws in traditional methods, where order and information about a word have vanished. Furthermore, it uses sentiment knowledge, sentiment shifter rules and multiple strategies to overcome the existing drawbacks. TSS exploits multiple sources of statistical and linguistic knowledge features to augment word-level embedding and extract a proper set of sentences from multiple documents. TSS also uses the Restricted Boltzmann Machine algorithm to enhance and optimize those features and improve resultant accuracy without losing any important information. OSM consists of two phases: sentence classification and sentence selection which work together to produce a useful summary. Experiment results show that RDLS outperforms other existing methods. Moreover, the ensemble of statistical and linguistic knowledge, sentiment knowledge, sentiment shifter rules and word-embedding model allows RLDS to achieve significant accuracy.

A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings

We propose a novel generative model to explore both local and global context for joint learning topics and topic-specific word embeddings. In particular, we assume that global latent topics are shared across documents, a word is generated by a hidden semantic vector encoding its contextual semantic meaning, and its context words are generated conditional on both the hidden semantic vector and global latent topics. Topics are trained jointly with the word embeddings. The trained model maps words to topic-dependent embeddings, which naturally addresses the issue of word polysemy. Experimental results show that the proposed model outperforms the word-level embedding methods in both word similarity evaluation and word sense disambiguation. Furthermore, the model also extracts more coherent topics compared with existing neural topic models or other models for joint learning of topics and word embeddings. Finally, the model can be easily integrated with existing deep contextualized word embedding learning methods to further improve the performance of downstream tasks such as sentiment classification.

Character level and word level embedding with bidirectional LSTM – Dynamic recurrent neural network for biomedical named entity recognition from literature

Named Entity Recognition is the process of identifying different entities in a given context. Biomedical Named Entity Recognition (BNER) is the task of extracting chemical names from biomedical texts to support biomedical and translational research. The aim of the system is to extract useful chemical names from biomedical literature text without a lot of handcrafted engineering features. This approach introduces a novel neural network architecture with the composition of bidirectional long short-term memory (BLSTM), dynamic recurrent neural network (RNN) and conditional random field (CRF) that uses character level and word level embedding as the only features to identify the chemical entities. Using this approach we have achieved the F1 score of 89.98 on BioCreAtIvE II GM corpus and 90.84 on NCBI corpus by outperforming the existing systems. Our system is based on the deep neural architecture that uses both character and word level embedding which captures the morphological and orthographic information eliminating the need for handcrafted engineering features. The proposed system outperforms the existing systems without a lot of handcrafted engineering features. The embedding concept along with the bidirectional LSTM network proved to be an effective method to identify most of the chemical entities.

Revisiting paraphrase question generator using pairwise discriminator

In this paper, we propose a method for obtaining sentence-level embeddings. While the problem of obtaining word-level embeddings is very well studied, we propose a novel method for obtaining sentence-level embeddings. This is obtained by a simple method in the context of solving the paraphrase generation task. If we use a sequential encoder-decoder model for generating paraphrase, we would like the generated paraphrase to be semantically close to the original sentence. One way to ensure this is by adding constraints for true paraphrase embeddings to be close and unrelated paraphrase candidate sentence embeddings to be far. This is ensured by using a sequential pair-wise discriminator that shares weights with the encoder. This discriminator is trained with a suitable loss function. Our loss function penalizes paraphrase sentence embedding distances from being too large. This loss is used in combination with a sequential encoder-decoder network. We also validate our method by evaluating the obtained embeddings for a sentiment analysis task. The proposed method results in semantic embeddings and provide competitive results on the paraphrase generation and sentiment analysis task on standard dataset. These results are also shown to be statistically significant.

Speech-Driven End-to-End Language Discrimination toward Chinese Dialects

Language discrimination among similar languages, varieties, and dialects is a challenging natural language processing task. The traditional text-driven focus leads to poor results. In this article, we explore the effectiveness of speech-driven features toward language discrimination among Chinese dialects. First, we systematically explore the appropriateness of speech-driven MFCC features toward CNN-based language discrimination. Then, we design an end-to-end speech recognition model based on HMM-DNN to predict Chinese dialect words. We adopt attention mechanism to extract the discriminative words related to different Chinese dialects. Finally, through a CNN, we combine the word-level embedding and the MFCC-based features. Evaluation of two benchmark Chinese dialect corpora shows the appropriateness and effectiveness of the proposed speech-driven approach to fine-grained Chinese dialect discrimination compared to the state-of-the-art methods.

A deep neural network-based model for named entity recognition for Hindi language

The aim of this work is to develop efficient named entity recognition from the given text that in turn improves the performance of the systems that use natural language processing (NLP). The performance of IoT-based devices such as Alexa and Cortana significantly depends upon an efficient NLP model. To increase the capability of the smart IoT devices in comprehending the natural language, named entity recognition (NER) tools play an important role in these devices. In general, the NER is a two-step process that initially the proper nouns are identified from text and then classify them into predefined categories of entities such as person, location, measure, organization and time. NER is often performed as a subtask while processing natural languages which increases the accuracy level of a NLP task. In this paper, we propose deep neural network architecture for named entity recognition for the resource-scarce language Hindi, based on convolutional neural network (CNN), bidirectional long short-term memory (Bi-LSTM) neural network and conditional random field (CRF). In the proposed approach, initially, we use skip-gram word2vec model and GloVe model to represent words in semantic vectors which are further used in different deep neural network-based architectures. In the proposed approach, we use character- and word-level embedding to represent the text that includes information at fine-grained level. Due to the use of character-level embeddings, the proposed model is robust for the out-of-vocabulary words. Experimental results show that the combination of Bi-LSTM, CNN and CRF algorithms performs better as compared to the other baseline methods such as recurrent neural network, long short-term memory and Bi-LSTM individually.

Hierarchical multimodal attention for end-to-end audio-visual scene-aware dialogue response generation

This work is extended from our participation in the 7th Dialogue System Technology Challenge (DSTC7), where we participated in the Audio Visual Scene-aware Dialogue System (AVSD) track. The AVSD track evaluates how dialogue systems understand video scenes and responds to users about the video visual and audio content. We propose a hierarchical attention approach on user queries, video caption, audio and visual features that contribute to improved evaluation results. We also apply a nonlinear feature fusion approach to combine the visual and audio features for better knowledge representation. Our proposed model shows superior performance in terms of both objective evaluation and human rating as compared to the baselines. In this extended work, we also provide a more extensive review of the related work, conduct additional experiments with word-level and context-level pretrained embeddings, and investigate different qualitative aspects of the generated responses.

Neural Named Entity Boundary Detection

In this paper, we focus on named entity boundary detection, which is to detect the start and end boundaries of an entity mention in text, without predicting its type. The detected entities are input to entity linking or fine-grained typing systems for semantic enrichment. We propose BdryBot, a recurrent neural network encoder-decoder framework with a pointer network to detect entity boundaries from a given sentence. The encoder considers both character-level representations and word-level embeddings to represent the input words. In this way, BdryBot does not require any hand-crafted features. Because of the pointer network, BdryBot overcomes the problem of variable size output vocabulary and the issue of sparse boundary tags. We conduct two sets of experiments, in-domain detection and cross-domain detection, on six datasets. Our results show that BdryBot achieves state-of-the-art performance against five baselines. In addition, our proposed approach can be further enhanced when incorporating contextualized language embeddings into token representations.

Text Classification Using Parallel Word-level and Character-level Embeddings in Convolutional Neural Networks

Deep learning techniques such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) show superior performance in text classification than traditional approaches such as Support Vector Machines (SVMs) and Naive Bayesian approaches. When using CNNs for text classification tasks, word embedding or character embedding is a step to transform words or characters to fixed size vectors before feeding them into convolutional layers. In this paper, we propose a parallel word-level and character-level embedding approach in CNNs for text classification. The proposed approach can capture word-level and character-level patterns concurrently in CNNs. To show the usefulness of proposed approach, we perform experiments with two English and three Korean text datasets. The experimental results show that character-level embedding works better in Korean and word-level embedding performs well in English. Also the experimental results reveal that the proposed approach provides better performance than traditional CNNs with word-level embedding or character-level embedding in both Korean and English documents. From more detail investigation, we find that the proposed approach tends to perform better when there is relatively small amount of data comparing to the traditional embedding approaches.

Leveraging More Fine-grained Representation to Reduce Instability within Word Embeddings

Suzi Park and Hyopil Shin. 2019. This paper investigates the notion of instability in word embeddings, primarily focused on the Korean language, by analyzing the effects of instability on several tasks using word level, character level, and sub-character level embeddings. Korean has a complex morphological system and is written at units smaller than that of a single character, making it optimal for comparing the three types of embeddings. We first carry out several intrinsic evaluations such as word similarity and analogy tests, as well as an extrinsic evaluation of the embeddings on the task of sentiment analysis. These results show that sub-character embeddings outperform the other two types, obtaining both higher and more consistent results, especially when the corpus size is large. Based on these findings, our hypothesis is that embeddings trained on smaller units and larger corpora would also be a better choice in terms of stability. To investigate this, we implemented two stability tests, looking into cosine similarity standard deviations and intersection of nearest neighbors among models. Our findings demonstrate that indeed, the smaller the embedding unit and the larger the corpus, the more stable the embedding type is. Importantly, our results illustrate the necessity of taking instability into consideration when using word embeddings within a given experiment.

Learning cross-lingual phonological and orthagraphic adaptations: a case study in improving neural machine translation between low-resource languages

Out-of-vocabulary (OOV) words can pose serious challenges for machine translation (MT) tasks, and in particular, for low-resource language (LRL) pairs, i.e., language pairs for which few or no parallel corpora exist. Our work adapts variants of seq2seq models to perform transduction of such words from Hindi to Bhojpuri (an LRL instance), learning from a set of cognate pairs built from a bilingual dictionary of Hindi - Bhojpuri words. We demonstrate that our models can be effectively used for language pairs that have limited parallel corpora; our models work at the character level to grasp phonetic and orthographic similarities across multiple types of word adaptations, whether synchronic or diachronic, loan words or cognates. We describe the training aspects of several character level NMT systems that we adapted to this task and characterize their typical errors. Our method improves BLEU score by 6.3 on the Hindi-to-Bhojpuri translation task. Further, we show that such transductions can generalize well to other languages by applying it successfully to Hindi - Bangla cognate pairs. Our work can be seen as an important step in the process of: (i) resolving the OOV words problem arising in MT tasks; (ii) creating effective parallel corpora for resource constrained languages; and (iii) leveraging the enhanced semantic knowledge captured by word-level embeddings to perform character-level tasks.

Bidirectional Encoder–Decoder Model for Arabic Named Entity Recognition

Sequence labeling models with recurrent neural network variants, such as long short-term memory (LSTM) and gated recurrent units, show promising performance in several natural language processing tasks, such as named entity recognition (NER). Most current models use unidirectional decoders, which reason only about the past and remain limited to retaining future contexts while generating predictions. Therefore, these models suffer from their generation of unbalanced outputs. Moreover, most existing NER models utilize word embeddings for capturing similarities between words but sustain when handling previously unobserved or infrequently used words. We propose a bidirectional encoder–decoder model for addressing the problem of Arabic NER on the basis of recent work in deep learning, in which the encoder and decoder are bidirectional LSTMs. In addition to word-level embeddings, character-level embeddings are adopted, and they are combined via an embedding-level attention mechanism. Our model can dynamically determine the information that must be utilized from a word- or character-level component through this attention mechanism. Experimental results on the ANERCorp and AQMAR datasets show that the model with a bi-encoder–decoder network and embedding attention layer achieves a high F-score measure of approximately 92%.

A Multi-Task Hierarchical Approach for Intent Detection and Slot Filling

Spoken language understanding (SLU) plays an integral part in every dialogue system. To understand the intention of the user and extract the necessary information to help the user achieve desired goals is a challenging task. In this work, we propose an end-to-end hierarchical multi-task model that can jointly perform both intent detection and slot filling tasks for the datasets of varying domains. The primary aim is to capture context information in a dialogue to help the SLU module in a dialogue system to correctly understand the user and assist the user in achieving the desired goals. It is vital for the SLU module to capture the past information along with the present utterance said by the user to retrieve correct information. The dependency and correlation between the two tasks, i.e. intent detection and slot filling makes the multi-task learning framework effective in capturing the desired information provided by the user. We use Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) to capture contextual information for the utterances. We employ Conditional Random Field (CRF) to model label dependency. Both character and word level embeddings are provided as input to the models. We create a benchmark corpus for the SLU tasks, on TRAINS and FRAMES dataset for capturing more realistic and natural utterances spoken by the speakers in a human/machine dialogue system. Experimental results on multiple datasets of various domains (ATIS, SNIP, TRAINS and FRAMES) show that our proposed approach is effective compared to the individual models and the state-of-the-art methods.

Extracting Resource Needs and Availabilities From Microblogs for Aiding Post-Disaster Relief Operations

Microblogging sites like Twitter are the important sources of real-time information during disaster/emergency events. During such events, the critical situational information posted is immersed in a lot of conversational content; hence, reliable methodologies are needed for extracting the meaningful information. In this paper, we focus on a particular application that is critical for efficient management of post-disaster relief operations—identifying tweets that inform about resource needs and resource availabilities. Two broad types of methodologies can be practically applied to identify such tweets during an ongoing disaster event: 1) supervised classification approaches, where the classifier models are trained on microblogs posted during prior events and applied on those posted during the ongoing event and 2) unsupervised pattern matching and information retrieval approaches that can be directly applied on the microblogs posted during the ongoing event. In this paper, we experiment with several supervised and unsupervised approaches to address the problem, including several neural network-based classification and retrieval models. We also propose two novel neural retrieval models (unsupervised) for the said application, which effectively combine word-level embeddings and character-level embeddings. We conduct experiments on tweets posted during two disaster events and observe that the two approaches perform well in different scenarios. Specifically, if good quality training data are available from prior events, then classification approaches perform better; however, if such training data are not available, then unsupervised retrieval methods outperform supervised classification approaches.

An Improved Word Representation for Deep Learning Based NER in Indian Languages

Named Entity Recognition (NER) is the process of identifying the elementary units in a text document and classifying them into predefined categories such as person, location, organization and so forth. NER plays an important role in many Natural Language Processing applications like information retrieval, question answering, machine translation and so forth. Resolving the ambiguities of lexical items involved in a text document is a challenging task. NER in Indian languages is always a complex task due to their morphological richness and agglutinative nature. Even though different solutions were proposed for NER, it is still an unsolved problem. Traditional approaches to Named Entity Recognition were based on the application of hand-crafted features to classical machine learning techniques such as Hidden Markov Model (HMM), Support Vector Machine (SVM), Conditional Random Field (CRF) and so forth. But the introduction of deep learning techniques to the NER problem changed the scenario, where the state of art results have been achieved using deep learning architectures. In this paper, we address the problem of effective word representation for NER in Indian languages by capturing the syntactic, semantic and morphological information. We propose a deep learning based entity extraction system for Indian languages using a novel combined word representation, including character-level, word-level and affix-level embeddings. We have used ‘ARNEKT-IECSIL 2018’ shared data for training and testing. Our results highlight the improvement that we obtained over the existing pre-trained word representations.

An adverse drug effect mentions extraction method based on weighted online recurrent extreme learning machine

Background and objectiveAutomatic extraction of adverse drug effect (ADE) mentions from biomedical texts is a challenging research problem that has attracted significant attention from the pharmacovigilance and biomedical text mining communities. Indeed, deep learning based methods have recently been employed to solve this issue with great success. However, they fail to effectively identify the boundary of mentions. In this paper, we propose a weighted online recurrent extreme learning machine (WOR-ELM) based method to overcome this drawback. MethodsThe proposed method for ADE mentions extraction from biomedical texts is divided into two stages: span detection and ADE mentions classification. At the first stage, we identify the boundary of the mentions irrespective of their types with a WOR-ELM in a given sentence. At the second stage, another WOR-ELM is used to classify the identified mentions to the appropriate type. Both stages use the concatenation of character-level and word-level embeddings as features. The character-level embedding is obtained using a modified online recurrent extreme learning machine, whereas the word-level embedding is obtained from a pre-trained model. ResultsSeveral experiments were carried out on a well-known ADE corpus to evaluate the effectiveness and demonstrate the usefulness of the proposed method. The obtained results show that our method achieves an F-score of 87.5%, which outperforms the current state-of-the-art methods. ConclusionsOur research results indicate that the proposed method for adverse drug effect mentions extraction from text can significantly improve performance over existing methods. Our experiments show the effectiveness of incorporating word-level and character level embeddings as features for WOR-ELM. They also illustrate the benefits of using IOU segment to represent ADE mentions.