Dependency Parsing Research Articles

Neural Character-Level Syntactic Parsing for Chinese

In this work, we explore character-level neural syntactic parsing for Chinese with two typical syntactic formalisms: the constituent formalism and a dependency formalism based on a newly released character-level dependency treebank. Prior works in Chinese parsing have struggled with whether to de ne words when modeling character interactions. We choose to integrate full character-level syntactic dependency relationships using neural representations from character embeddings and richer linguistic syntactic information from human-annotated character-level Parts-Of-Speech and dependency labels. This has the potential to better understand the deeper structure of Chinese sentences and provides a better structural formalism for avoiding unnecessary structural ambiguities. Specifically, we first compare two different character-level syntax annotation styles: constituency and dependency. Then, we discuss two key problems for character-level parsing: (1) how to combine constituent and dependency syntactic structure in full character-level trees and (2) how to convert from character-level to word-level for both constituent and dependency trees. In addition, we also explore several other key parsing aspects, including di erent character-level dependency annotations and joint learning of Parts-Of-Speech and syntactic parsing. Finally, we evaluate our models on the Chinese Penn Treebank (CTB) and our published Shanghai Jiao Tong University Chinese Character Dependency Treebank (SCDT). The results show the e effectiveness of our model on both constituent and dependency parsing. We further provide empirical analysis and suggest several directions for future study.

Locality and expectation effects in Hindi preverbal constituent ordering

We investigate the relative impact of two influential theories of language comprehension, viz., Dependency Locality Theory (Gibson, 2000; DLT) and Surprisal Theory (Hale, 2001; Levy, 2008), on preverbal constituent ordering in Hindi, a predominantly SOV language with flexible word order. Prior work in Hindi has shown that word order scrambling is influenced by information structure constraints in discourse. However, the impact of cognitively grounded factors on Hindi constituent ordering is relatively underexplored. We test the hypothesis that dependency length minimization is a significant predictor of syntactic choice, once information status and surprisal measures (estimated from n-gram i.e., trigram and incremental dependency parsing models) have been added to a machine learning model. Towards this end, we setup a framework to generate meaning-equivalent grammatical variants of Hindi sentences by linearizing preverbal constituents of projective dependency trees in the Hindi-Urdu Treebank (HUTB) corpus of written text. Our results indicate that dependency length displays a weak effect in predicting reference sentences (amidst variants) over and above the aforementioned predictors. Overall, trigram surprisal outperforms dependency length and parser surprisal by a huge margin and our analyses indicate that maximizing lexical predictability is the primary driving force behind preverbal constituent ordering choices in Hindi. The success of trigram surprisal notwithstanding, dependency length minimization predicts non-canonical reference sentences having fronted direct objects over variants containing the canonical word order, cases where surprisal estimates fail due to their bias towards frequent structures and word sequences. Locality effects persist over the Given-New preference of subject-object ordering in Hindi. Accessibility and local statistical biases discussed in the sentence processing literature are plausible explanations for the success of trigram surprisal. Further, we conjecture that the presence of case markers is a strong factor potentially overriding the pressure for dependency length minimization in Hindi. Finally, we discuss the implications of our findings for the information locality hypothesis and theories of language production.

Joint learning of morphology and syntax with cross-level contextual information flow

AbstractWe propose an integrated deep learning model for morphological segmentation, morpheme tagging, part-of-speech (POS) tagging, and syntactic parsing onto dependencies, using cross-level contextual information flow for every word, from segments to dependencies, with an attention mechanism at horizontal flow. Our model extends the work of Nguyen and Verspoor ((2018). Proceedings of the CoNLL Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. The Association for Computational Linguistics, pp. 81–91.) on joint POS tagging and dependency parsing to also include morphological segmentation and morphological tagging. We report our results on several languages. Primary focus is agglutination in morphology, in particular Turkish morphology, for which we demonstrate improved performance compared to models trained for individual tasks. Being one of the earlier efforts in joint modeling of syntax and morphology along with dependencies, we discuss prospective guidelines for future comparison.

Multi Layered Rule-Based Technique for Explicit Aspect Extraction from Online Reviews

In the field of sentiment analysis, extracting aspects or opinion targets from user reviews about a product is a key task. Extracting the polarity of an opinion is much more useful if we also know the targeted Aspect or Feature. Rule based approaches, like dependency-based rules, are quite popular and effective for this purpose. However, they are heavily dependent on the authenticity of the employed parts-of-speech (POS) tagger and dependency parser. Another popular rule based approach is to use sequential rules, wherein the rules formulated by learning from the user’s behavior. However, in general, the sequential rule-based approaches have poor generalization capability. Moreover, existing approaches mostly consider an aspect as a noun or noun phrase, so these approaches are unable to extract verb aspects. In this article, we have proposed a multi-layered rule-based (ML-RB) technique using the syntactic dependency parser based rules along with some selective sequential rules in separate layers to extract noun aspects. Additionally, after rigorous analysis, we have also constructed rules for the extraction of verb aspects. These verb rules primarily based on the association between verb and opinion words. The proposed multi-layer technique compensates for the weaknesses of individual layers and yields improved results on two publicly available customer review datasets. The F1 score for both the datasets are 0.90 and 0.88, respectively, which are better than existing approaches. These improved results can be attributed to the application of sequential/syntactic rules in a layered manner as well as the capability to extract both noun and verb aspects.

A Model for Cross-Domain Opinion Target Extraction in Sentiment Analysis

Opinion target extraction is one of the core tasks in sentiment analysis on text data. In recent years, dependency parser–based approaches have been commonly studied for opinion target extraction. However, dependency parsers are limited by language and grammatical constraints. Therefore, in this work, a sequential pattern-based rule mining model, which does not have such constraints, is proposed for cross-domain opinion target extraction from product reviews in unknown domains. Thus, knowing the domain of reviews while extracting opinion targets becomes no longer a requirement. The proposed model also reveals the difference between the concepts of opinion target and aspect, which are commonly confused in the literature. The model consists of two stages. In the first stage, the aspects of reviews are extracted from the target domain using the rules automatically generated from source domains. The aspects are also transferred from the source domains to a target domain. Moreover, aspect pruning is applied to further improve the performance of aspect extraction. In the second stage, the opinion target is extracted among the aspects extracted at the former stage using the rules automatically generated for opinion target extraction. The proposed model was evaluated on several benchmark datasets in different domains and compared against the literature. The experimental results revealed that the opinion targets of the reviews in unknown domains can be extracted with higher accuracy than those of the previous works.

HPSG-Inspired Joint Neural Constituent and Dependency Parsing in O($n^3$) Time Complexity

HPSG-Inspired Joint Neural Constituent and Dependency Parsing in O($n^3$) Time Complexity

Predicting Chinese Phrase-Level Sentiment Intensity in Valence-Arousal Dimensions With Linguistic Dependency Features

Phrase-level sentiment intensity prediction is difficult due to the inclusion of linguistic modifiers (e.g., negators, degree adverbs, and modals) potentially resulting in an intensity shift or polarity reversal for the modified words. This study develops a graph-based Chinese parser based on the deep biaffine attention model to obtain dependency structures and relations. These obtained dependency features are then used in our proposed Weighted-sum Tree GRU network to predict phrase-level sentiment intensity in the valence-arousal dimensions. Dependency parsing results using the Sinica Treebank indicate that our graph-based model outperforms transition-based methods such as MLP and stack-LSTM with identical findings for English dependency parsing. Experimental results on the Chinese EmoBank indicate that our Weighted-sum Tree GRU network model outperforms other transformer-based neural networks such as BERT, ALBERT, XLNET and ELECTRA, reflecting the effectiveness of linguistic dependencies in phrase-level sentiment intensity predication tasks. In addition, our proposed model requires fewer parameters and less inference time for quantitative analysis, making the proposed model is relatively lightweight and efficient.

A Hybrid Deep Dependency Parsing Approach Enhanced With Rules and Morphology: A Case Study for Turkish

Fully data-driven, deep learning-based models are usually designed as language-independent and have been shown to be successful for many natural language processing tasks. However, when the studied language is not high-resource and the amount of training data is insufficient, these models can benefit from the integration of natural language grammar-based information.We propose two approaches to dependency parsing especially for languages with restricted amount of training data. Our first approach combines a state-of-the-art deep learning-based parser with a rule-based approach and the second one incorporates morphological information into the parser. In the rule-based approach, the parsing decisions made by the rules are encoded and concatenated with the vector representations of the input words as additional information to the deep network. The morphology-based approach proposes different methods to include the morphological structure of words into the parser network. Experiments are conducted on three different Turkish treebanks and the results suggest that integration of explicit knowledge about the target language to a neural parser through a rule-based parsing system and morphological analysis leads to more accurate annotations and hence, increases the parsing performance in terms of attachment scores. The proposed methods are developed for Turkish, but can be adapted to other languages as well.

Performance analysis of low dimensional word embeddings to support green computing

It has become increasingly important to pay attention how much energy we use to operate various Artificial Intelligence (AI) and Machine Learning (ML) systems. In order to implement environmentally responsible solutions we need to reconsider our used storage resources and computational power. Training a natural language model is a time and energy demanding process. In recent years the language models are becoming extremely large and the trend is growing. The building process of these models are consuming an extremely large amount of computational power hence these demands huge amounts of energy. In our research we trained and evaluated low dimensional word2vec embedding models and analyzed their performance on building transition based dependency parsers to show that low dimensional models are still competitive and in many use cases may be sufficient.

Aspect-Level Sentiment Analysis via a Syntax-Based Neural Network

Aspect-level sentiment classification (ALSC); a fine-grained task of sentiment analysis holds the promise of machines communicating knowledge of different aspects of a product/service with humans. Specifically, ALSC aims at inferring the sentiment expressed toward a specific aspect term in a user-generated textual content. It is a long-standing problem in sentiment analysis, and its usefulness cannot be underestimated. Recent studies employ the dependency tree and restructure it to model the syntactic relationship among words in text. However, the restructuring process destructs the structural information of the original dependency tree. As a solution, we first construct a syntax graph that preserves the structural information of the dependency tree. We then propose a reliable syntax-based neural network model that performs a thorough search on the syntax graph to effectively find the relevant contextual information with respect to the aspect term for the sentence encoding. Noting that dependency trees parsed from existing dependency parsers may contain incorrect syntactic dependencies due to grammatical errors in a sentence, we adopt a convolutional layer that takes into account the relations among features of words in a local neighborhood to mitigate the issues brought by incorrect syntactic dependencies. Our results on benchmark datasets demonstrate that our model outperforms the previous methods and achieves state-of-the-art results for the ALSC task.

Syntactic Factors Associated With Performance of Dependency Parsers Using Stack-Pointer Network and Graph Attention Networks Between English and Korean

A Stack-Pointer Network (StackPtr) parser is a pointer network with an internal stack on the decoder. Several studies use the StackPtr as the backbone of a dependency parser because it can traverse a parse tree depth-first without backtracking and can handle high-order parsing information easily thanks to the internal stack. The parser can use information from previously derived subtrees stored in the internal stack upon selecting a child node. In this work, we introduce a new StackPtr parser with Graph Attention Networks (GATs) that can encode a previously derived subtree. We evaluated our proposed parser on the Sejong and Everyone’s corpora for Korean and on the Penn Treebank and Universal Dependency corpora for English. In addition, we analyzed and compared our proposed parser with other variants of the StackPtr parser, examining the syntactic information that each parser can reference at every decoding step. We found that Korean parse trees tend to have more consecutive immediate single-child nodes than English parse trees. The proposed StackPtr parser with GATs performed best on almost all metrics for Korean because it can utilize more context to analyze these parse trees by grasping Korean syntactic factors than any other variants. However, for English, no particular variant of the StackPtr parser outperforms the others.

Neural Collaborative Filtering for Chinese Movies Based on Aspect-Aware Implicit Interactions

Aspect information mining from user comments has become an important means to improve the performance of recommendation systems (RSs). This is because aspect information in comments is fine-grained and tends to reflect the interactions and preferences of users over items in multiple dimensions. These interactions are different from ratings, which are often explicit and linear. Most current RSs based on aspect information learn the contribution of explicit interactions of aspects in a linear manner, while ignoring the implicit features and non-linear interactions of aspects. Since Chinese grammar is greatly different with English grammar, there are few recommendation models based on Chinese movie comment aspects. In this work, we propose an architecture, named aspect-based neural collaborative filtering (ANCF), to extract comment aspect terms based on rules formulated in Chinese dependency parsing. The proposed ANCF integrates a generalized tensor factorization and a tensorized multi-layer perceptrons into the neural network to capture user-item-aspect interactions in a mixed linear and nonlinear way. The aspect potential interaction vector and the actual interaction vector are layered and fused into tensor processing, which can reduce the tensor sparsity and solve the cold start problem of collaborative filtering to a certain extent. Performance results show that the proposed model outperforms some of the traditional ones in terms of recommendation accuracy and effectiveness.

Skeleton parsing for complex question answering over knowledge bases

Answering complex questions involving multiple relations over knowledge bases is a challenging task. Many previous works rely on dependency parsing. However, errors in dependency parsing would influence their performance, in particular for long complex questions. In this paper, we propose a novel skeleton grammar to represent the high-level structure of a complex question. This lightweight formalism and its BERT-based parsing algorithm help to improve the downstream dependency parsing. To show the effectiveness of skeleton, we develop two question answering approaches: skeleton-based semantic parsing (called SSP) and skeleton-based information retrieval (called SIR). In SSP, skeleton helps to improve structured query generation. In SIR, skeleton helps to improve path ranking. Experimental results show that, thanks to skeletons, our approaches achieve state-of-the-art results on three datasets: LC-QuAD 1.0, GraphQuestions, and ComplexWebQuestions 1.1.

Neural factoid geospatial question answering

Existing question answering systems struggle to answer factoid questions when geospatial information is involved. This is because most systems cannot accurately detect the geospatial semantic elements from the natural language questions, or capture the semantic relationships between those elements. In this paper, we propose a geospatial semantic encoding schema and a semantic graph representation which captures the semantic relations and dependencies in geospatial questions. We demonstrate that our proposed graph representation approach aids in the translation from natural language to a formal, executable expression in a query language. To decrease the need for people to provide explanatory information as part of their question and make the translation fully automatic, we treat the semantic encoding of the question as a sequential tagging task, and the graph generation of the query as a semantic dependency parsing task. We apply neural network approaches to automatically encode the geospatial questions into spatial semantic graph representations. Compared with current template-based approaches, our method generalises to a broader range of questions, including those with complex syntax and semantics. Our proposed approach achieves better results on GeoData201 than existing methods.

Instance-Based Neural Dependency Parsing

AbstractInterpretable rationales for model predictions are crucial in practical applications. We develop neural models that possess an interpretable inference process for dependency parsing. Our models adopt instance-based inference, where dependency edges are extracted and labeled by comparing them to edges in a training set. The training edges are explicitly used for the predictions; thus, it is easy to grasp the contribution of each edge to the predictions. Our experiments show that our instance-based models achieve competitive accuracy with standard neural models and have the reasonable plausibility of instance-based explanations.

Tri-training for Dependency Parsing Domain Adaptation

In recent years, the research on dependency parsing focuses on improving the accuracy of the domain-specific (in-domain) test datasets and has made remarkable progress. However, there are innumerable scenarios in the real world that are not covered by the dataset, namely, the out-of-domain dataset. As a result, parsers that perform well on the in-domain data usually suffer from significant performance degradation on the out-of-domain data. Therefore, to adapt the existing in-domain parsers with high performance to a new domain scenario, cross-domain transfer learning methods are essential to solve the domain problem in parsing. This paper examines two scenarios for cross-domain transfer learning: semi-supervised and unsupervised cross-domain transfer learning. Specifically, we adopt a pre-trained language model BERT for training on the source domain (in-domain) data at the subword level and introduce self-training methods varied from tri-training for these two scenarios. The evaluation results on the NLPCC-2019 shared task and universal dependency parsing task indicate the effectiveness of the adopted approaches on cross-domain transfer learning and show the potential of self-learning to cross-lingual transfer learning.

Aspect-based Sentiment Analysis using Dependency Parsing

In this paper, an aspect-based Sentiment Analysis (SA) system for Hindi is presented. The proposed system assigns a separate sentiment towards the different aspects of a sentence as well as it evaluates the overall sentiment expressed in a sentence. In this work, Hindi Dependency Parser (HDP) is used to determine the association between an aspect word and a sentiment word (using Hindi SentiWordNet) and works on the idea that closely connected words come together to express a sentiment about a certain aspect. By generating a dependency graph, the system assigns the sentiment to an aspect having a minimum distance between them and computes the overall polarity of the sentence. The system achieves an accuracy of 83.2% on a corpus of movie reviews and its results are compared with baselines as well as existing works on SA. From the results, it has been observed that the proposed system has the potential to be used in emerging applications like SA of product reviews, social media analysis, etc.

Morphological Tagging and Lemmatization in the Albanian Language

Abstract An important element of Natural Language Processing is parts of speech tagging. With fine-grained word-class annotations, the word forms in a text can be enhanced and can also be used in downstream processes, such as dependency parsing. The improved search options that tagged data offers also greatly benefit linguists and lexicographers. Natural language processing research is becoming increasingly popular and important as unsupervised learning methods are developed. There are some aspects of the Albanian language that make the creation of a part-of-speech tag set challenging. This research provides a discussion of those issues linguistic phenomena and presents a proposal for a part-of-speech tag set that can adequately represent them. The corpus contains more than 250,000 tokens, each annotated with a medium-sized tag set. The Albanian language’s syntagmatic aspects are adequately represented. Additionally, in this paper are morphologically and part-of-speech tagged corpora for the Albanian language, as well as lemmatize and neural morphological tagger trained on these corpora. Based on the held-out evaluation set, the model achieves 93.65% accuracy on part-of-speech tagging, The morphological tagging rate was 85.31 % and the lemmatization rate was 88.95%. Furthermore, the TF-IDF technique weighs terms and with the scores are highlighted words that have additional information for the Albanian corpus.

Towards structured NLP interpretation via graph explainers

AbstractNatural language processing (NLP) models have been increasingly deployed in real‐world applications, and interpretation for textual data has also attracted dramatic attention recently. Most existing methods generate feature importance interpretation, which indicate the contribution of each word towards a specific model prediction. Text data typically possess highly structured characteristics and feature importance explanation cannot fully reveal the rich information contained in text. To bridge this gap, we propose to generate structured interpretations for textual data. Specifically, we pre‐process the original text using dependency parsing, which could transform the text from sequences into graphs. Then graph neural networks (GNNs) are utilized to classify the transformed graphs. In particular, we explore two kinds of structured interpretation for pre‐trained GNNs: edge‐level interpretation and subgraph‐level interpretation. Experimental results over three text datasets demonstrate that the structured interpretation can better reveal the structured knowledge encoded in the text. The experimental analysis further indicates that the proposed interpretations can faithfully reflect the decision‐making process of the GNN model.

Embedding from Language Models (ELMos)- based Dependency Parser for Indonesian Language

The goal of dependency parsing is to seek a functional relationship among words. For instance, it tells the subject-object relation in a sentence. Parsing the Indonesian language requires information about the morphology of a word. Indonesian grammar relies heavily on affixation to combine root words with affixes to form another word. Thus, morphology information should be incorporated. Fortunately, it can be encoded implicitly by word representation. Embeddings from Language Models (ELMo) is a word representation which be able to capture morphology information. Unlike most widely used word representations such as word2vec or Global Vectors (GloVe), ELMo utilizes a Convolutional Neural Network (CNN) over characters. With it, the affixation process could ideally encoded in a word representation. We did an analysis using nearest neighbor words and T-distributed Stochastic Neighbor Embedding (t-SNE) word visualization to compare word2vec and ELMo. Our result showed that ELMo representation is richer in encoding the morphology information than it's counterpart. We trained our parser using word2vec and ELMo. To no surprise, the parser which uses ELMo gets a higher accuracy than word2vec. We obtain Unlabeled Attachment Score (UAS) at 83.08 for ELMo and 81.35 for word2vec. Hence, we confirmed that morphology information is necessary, especially in a morphologically rich language like Indonesian. Keywords: ELMo, Dependency Parser, Natural Language Processing, word2vec