Dependency Tree Research Articles

Promoting the Knowledge of Source Syntax in Transformer NMT Is Not Needed

The utility of linguistic annotation in neural machine translation seemed to had been established in past papers. The experiments were however limited to recurrent sequence-to-sequence architectures and relatively small data settings. We focus on the state-of-the-art Transformer model and use comparably larger corpora. Specifically, we try to promote the knowledge of source-side syntax using multi-task learning either through simple data manipulation techniques or through a dedicated model component. In particular, we train one of Transformer attention heads to produce source-side dependency tree. Overall, our results cast some doubt on the utility of multi-task setups with linguistic information. The data manipulation techniques, recommended in previous works, prove ineffective in large data settings. The treatment of self-attention as dependencies seems much more promising: it helps in translation and reveals that Transformer model can very easily grasp the syntactic structure. An important but curious result is, however, that identical gains are obtained by using trivial "linear trees" instead of true dependencies. The reason for the gain thus may not be coming from the added linguistic knowledge but from some simpler regularizing effect we induced on self-attention matrices.

A tree-based neural network model for biomedical event trigger detection

Biomedical event trigger detection is a heated research topic since its important role in biomedical event extraction. Previous studies show that syntactic features are very crucial for the task. However, existing methods largely focus on traditional statistical models, and usually capture syntactic features by extracting a set of manually-crafted features based on dependency tree. This limits the performance of the task. In this paper, we propose a tree-based neural network model, which can automatically learn syntactic features from dependency tree for trigger detection. Specifically, we use a recursive neural network to represent whole dependency tree globally, to better incorporate dependency-based syntax information. Results on MLEE and BioNLP-09 datasets show that the proposed model achieves 80.28% and 73.24% F1 score, respectively, outperforming traditional statistical models and neural baseline systems.

Negation and speculation scope detection using recursive neural conditional random fields

Negation and speculation scope detection is an important task in natural language processing. Previous studies all show that syntactic information is crucial to the task. However, these work mainly focuses on human-designed discrete features and local features extracted from dependency tree, limiting the performance of the task. In this paper, we propose a recursive neural network sequence labeling model, representing whole dependency tree globally and learning automatically syntactic features, for the task. Specifically, recursive neural network first learns a high-level representation for the words in the context of each sentence, and captures the cue word with its target scope through the global dependency structure. Then, CRFs layer takes the representation from recursive neural network as input to jointly decode labels for the whole sentence. Experimental results on English dataset BioScope and Chinese dataset CNeSp show that our model outperforms the state-of-the-art systems.

Semantic relation extraction using sequential and tree-structured LSTM with attention

Semantic relation extraction is crucial to automatically constructing a knowledge graph (KG), and it supports a variety of downstream natural language processing (NLP) tasks such as query answering (QA), semantic search and textual entailment. In addition, the semantic relation extraction task is mainly responsible for identifying entity pairs from raw texts and extracting the semantic relations between the extracted entity pairs. Existing methods consider only lexical-level features and often ignore syntactic features, resulting in poor relation extraction performance. By analyzing the necessity of the syntactic dependency and the contributions of words in a sentence to relation extraction, this paper proposes an end-to-end method that uses bidirectional tree-structured long short-term memory (LSTM) to extract structural features based on the dependency tree of a sentence. To enhance the performance of the relation extraction, the bidirectional sequential LSTM with attention is used to identify word-based features including the positional information of entity pairs and the contribution of words. Then, structural features and word-based features are concatenated to optimize the relation extraction performance. Finally, the proposed method is used on the SemEval 2010 task 8 and the CoNLL04 datasets to validate its performance. The experimental results show that the proposed method achieves state-of-the-art results on the SemEval 2010 task 8 and the CoNLL04 datasets.

A computational model for measuring discourse complexity

In past studies, the few quantitative approaches to discourse structure were mostly confined to the presentation of the frequency of discourse relations. However, quantitative approaches should take into account both hierarchical and relational layers in the discourse structure. This study considers these factors and addresses the issue of how discourse relations and discourse units are related. It draws upon the available corpora of discourse structure (rhetorical structure theory-discourse treebank (RST-DT)) from a new perspective. Since an RST tree can be converted into a syntactic dependency tree, the data extracted from the RST-DT can be useful for calculating the discourse distance in much the same way as syntactic dependency distance is calculated. Discourse distance is also applicable to measuring the depth of the human processing of discourse. Furthermore, the data derived from the RST-DT are also easily converted into network data. This study finds that discourse structure has its discourse distance minimum and each type of RST relations has its range of discourse distance. The frequency distribution of discourse data basically follows the power law on several levels, while a network approach reveals how discourse units are arranged spatially in regular patterns. The two methods are mutually complementary in revealing the interaction between discourse relations and discourse units in a comprehensive manner, as well as in revealing how people process and comprehend discourse dynamically. Accordingly, we propose merging the two methods so as to yield a computational model for assessing discourse complexity and comprehension.

Analysis of Frontal Sinuses for Personal Identification in a Chinese Sample Using a New Code Number

The study of frontal sinuses for personal identification is a considered approach in the forensic field. In Yoshino's system, the frontal sinus patterns of a given person were formulated as a code number (codY) obtained by arranging the class numbers in each classification item. The aim of this work was to use a new code number (codC) with eight digits that includes other two continuous variables obtained as ratios SOR1 (left frontal sinus area/left orbit area) and SOR2 (right frontal sinus area/right orbit area), comparing the results with that of Yoshino. Digital radiographic images of the skulls of 150 Chinese people were analyzed. Spearman's partial correlation coefficients were evaluated between variables characterizing frontal sinus patterns. We used a maximum-weight dependence tree for statistical analysis. Results demonstrated that codC significantly reduced the probability of having the same personal code number compared with codY. Our scientific approach results valid for personal identification purposes.

Syntax-aware entity representations for neural relation extraction

Distantly supervised relation extraction has been widely used to find novel relational facts between entities from text, and can be easily scaled to very large corpora. Previous studies on neural relation extraction treat this task as a multi-instance learning problem, and encode the sentences in low-dimensional spaces via neural networks. Although great progress has been made, they seldom consider the information represented by entities, which are of great significance to relation extraction. In this article, we propose several methods based on different tree-based models to learn syntax-aware entity representations for neural relation extraction. First, we encode the context of entities on dependency trees as sentence-level entity embedding based on tree-structured neural network models. Then, we utilize inter-sentence attention mechanism to obtain sentence bag level entity embedding over all sentences containing the specified entity pair. Finally, we combine both sentence embedding and entity embedding for relation classification. Experimental results on a widely used real-world dataset indicate that our system performs better than the state-of-the-art systems of relation extraction.

A Deep Sequential Model for Discourse Parsing on Multi-Party Dialogues

Discourse structures are beneficial for various NLP tasks such as dialogue understanding, question answering, sentiment analysis, and so on. This paper presents a deep sequential model for parsing discourse dependency structures of multi-party dialogues. The proposed model aims to construct a discourse dependency tree by predicting dependency relations and constructing the discourse structure jointly and alternately. It makes a sequential scan of the Elementary DiscourseUnits(EDUs)1 in a dialogue. For each EDU, the model decides to which previous EDU the current one should link and what the corresponding relation type is. The predicted link and relation type are then used to build the discourse structure incrementally with a structured encoder. During link prediction and relation classification, the model utilizes not only local information that represents the concerned EDUs, but also global information that encodes the EDU sequence and the discourse structure that is already built at the current step. Experiments show that the proposed model outperforms all the state-of-the-art baselines.

More accurate results on the provable security of AES against impossible differential cryptanalysis

Whether there exist longer impossible differentials than existing ones for a block cipher, is an important problem in the provable security evaluation of a block cipher against impossible differential cryptanalysis. In this paper, we give more accurate results for this problem for the AES. After investigating the differential properties of both the S-box and the linear layer of AES, we theoretically prove that there do not exist impossible concrete differentials longer than 4 rounds for AES by proving that any concrete differential is possible for the 5-round AES, under the only assumption that the round keys are independent and uniformly random. We use a tool, called “(w, d)-Dependent Tree (DT)”, to show how any concrete differential $$\varDelta X \rightarrow \varDelta Z$$ can be connected in the middle of the 5-round AES by some DTs. Our method might shed some light on bounding the length of impossible differentials with the differential properties of the S-boxes considered for some SPN block ciphers.

Improving Aspect Term Extraction With Bidirectional Dependency Tree Representation

Aspect term extraction is one of the important subtasks in aspect-based sentiment analysis. Previous studies have shown that using dependency tree structure representation is promising for this task. However, most dependency tree structures involve only one directional propagation on the dependency tree. In this paper, we first propose a novel bidirectional dependency tree network to extract dependency structure features from the given sentences. The key idea is to explicitly incorporate both representations gained separately from the bottom-up and top-down propagation on the given dependency syntactic tree. An end-to-end framework is then developed to integrate the embedded representations and BiLSTM plus CRF to learn both tree-structured and sequential features to solve the aspect term extraction problem. Experimental results demonstrate that the proposed model outperforms state-of-the-art baseline models on four benchmark SemEval datasets.

Intelligent Analysis Method of Assembly Process Based on Semantic Elements

The assembly process file is a direct expression of the product assembly design intent. In order to improve the quality and efficiency of the assembly, the 3D model of the product and resources needs to be introduced into the virtual assembly environment. However, the assembly process is verified and optimized by assembly simulation. In order to realize the intelligent analysis of assembly process files and the automation of assembly simulation, an intelligent analysis method of assembly process based on semantic elements is proposed. This article takes the assembly process documentation of complex products as the research object. An improved TF-IDF algorithm is proposed to automatically extract key semantic elements in the assembly process file. We quickly resolve previous assembly process data by building assembly key semantic corpora. An assembly semantic dependency tree algorithm based on maximum entropy is proposed to automatically reason the logical relationship of each assembly phrase in the assembly process statement. According to the above algorithm, the assembly semantic intelligent analysis system is developed. The assembly process of a certain type of ground weapon system is taken as an example to verify the system. And the intelligent analysis of the product process file is realized which lays a foundation for realizing the semantic driving of product assembly simulation.

Machine Translation Evaluation Metric Based on Dependency Parsing Model

Most of the syntax-based metrics obtain the similarity by comparing the sub-structures extracted from the trees of hypothesis and reference. These sub-structures cannot represent all the information in the trees because their lengths are limited. To sufficiently use the reference syntax information, a new automatic evaluation metric is proposed based on the dependency parsing model. First, a dependency parsing model is trained using the reference dependency tree for each sentence. Then, the hypothesis is parsed by this dependency parsing model and the corresponding hypothesis dependency tree is generated. The quality of hypothesis can be judged by the quality of the hypothesis dependency tree. Unigram F-score is included in the new metric so that lexicon similarity is obtained. According to experimental results, the proposed metric can perform better than METEOR and BLEU on system level and get comparable results with METEOR on sentence level. To further improve the performance, we also propose a combined metric which gets the best performance on the sentence level and on the system level.

Research on domain terminology recognition based on dependency tree-conditional random field

In view of the inconsistency of Chinese patent information in manual marking and classification, which leads to problems such as missed detection, partial detection and noise of patent search, this paper proposes a method based on the dependency tree-conditional random field(CRF) identification field terminology. The method is based on the modern grammar theory of dependency, using the existing technology to mark the dependency relationship. Finally, the corresponding technical feature words are identified in the results of the dependency labelling, and the training data is used as the training data to train the conditional random field model to identify the domain terminology. The experimental results show that the acquisition of training data through the dependency tree can improve the accuracy, recall and F value of the recognition results.

Leader-aware community detection in complex networks

Community structures are very common in complex networks. Detecting these communities is important for understanding the hidden features of networks. Besides, each community usually has one leader, which presents its significant influence over the whole community. However, most existing methods just focus on the problem of graph clustering, ignoring the role of community leaders. To solve this problem, in this paper, we propose a novel leader-aware community detection algorithm, which can find community structures as well as leaders of each community. This algorithm measures the leadership of each node and lets each one adhere to its local leader, forming dependence trees. Once all dependence trees are definitely settled, the community structures emerge because one tree actually is a cluster. Additionally, each root node of the tree is exactly the leader of corresponding community. This method can quickly determine the belonging of each node. Experimental results on real-world and benchmark networks demonstrate the effectiveness and the efficiency of our algorithm compared with other state-of-the-art approaches.

Employing Dependency Tree in Machine Learning Based Indonesian Factoid Question Answering

We proposed the usage of dependency tree information to increase the accuracy of Indonesian factoid question answering. We employed MSTParser and Universal Dependency corpus to build the Indonesian dependency parser. The dependency tree information as the result of the Indonesian dependency parse is used in the answer finder component of Indonesian factoid question answering system. Here, we used dependency tree information in two ways: 1) as one of the features in machine learning based answer finder (classifying each term in the retrieved passage as part of a correct answer or not); 2) as an additional heuristic rule after conducting the machine learning technique. For the machine learning technique, we combined word based calculation, phrase based calculation and similarity dependency relation based calculation as the complete features. Using 203 data, we were able to enhance the accuracy for the Indonesian factoid QA system compared to related work by only using the phrase information. The best accuracy was 84.34% for the correct answer classification and the best MRR was 0.954.

Predictive ability of covariate-dependent Markov models and classification tree for analyzing rainfall data in Bangladesh

This study attempts to make comparison between different parametric regressive models for the bivariate binary data with a machine learning technique. The data on sequential occurrence of rainfall in consecutive days is considered. The outcomes are classified as rainfall in both days, rainfall in one of the consecutive days, and no rainfall in both days. The occurrence of rainfall in consecutive days is analyzed by using statistical models with covariate dependence and classification tree for the period from 1980 to 2014. We have used relative humidity, minimum temperature, maximum temperature, sea level pressure, sunshine hour, and cloud cover in the model as covariates. The binary outcome variable is defined as the occurrence or non-occurrence of rainfall. Five regions of Bangladesh are considered in this study and one station from each region is selected on the basis of two criteria: (i) contains fewer missing values and (ii) representative of the regional characteristics geographically. Several measures are used to compare the models based on Markov chain and classification tree. It is found that for yearly data, both the Markov model and classification tree performed satisfactorily. However, the seasonal data show variation of rainfall. In some seasons, both models perform equally good such as monsoon, pre-monsoon, and post-monsoon, but in the winter season, the Markov model works poorly whereas classification tree fails to work. Additionally, we also observe that the Markov model performed consistently for each season and performs better compared with the classification tree. It has been demonstrated that the covariate-dependent Markov models can be used as classifiers alternative to the classification tree. It is revealed that the predictive ability of the covariate-dependent Markov model based on Markovian assumption performs either better or equally good compared with the classification tree. The joint models also consistently showed better predictive performance compared with regressive model for whole year data as well as for several seasonal data.

Natural Language Generation Using Dependency Tree Decoding for Spoken Dialog Systems

In this paper, we propose a new natural language generation (NLG) method for spoken dialog systems and demonstrate its capacity. Studies on NLG often employ sequence decoding, which generates the words comprising a sentence in sequential order and uses the input generated by each word in the previous step. In contrast, we propose a decoding method that employs a sequence generated by traversing a dependency tree with feed input to a pair consisting of a parent and sibling in the dependency tree. As a result, the most important words are generated first, thereby enabling words with greater relevance to be fed into the process. At prediction time, our model generates dependency trees and converts the trees into sentences. The proposed decoding method was evaluated by re-implementing a semantically controlled long short-term memory structure for NLG, and the input and predicted sequence were converted to allow dependency tree decoding. The experimental results indicated that our suggested approach, i.e., dependency tree decoding, dramatically elevates the BLEU-score and naturalness. Furthermore, when creating sentences with ${n}$ -best using dependency tree decoding, the word diversity of the output sentences was increased by approximately 6%, offering a more diverse sentence pattern.

Improving Distantly Supervised Relation Classification With Attention and Semantic Weight

Distantly supervised relation classification aims at identifying the relationship between two given entities and plays an essential part in natural language processing (NLP). Although distant supervision is able to generate labeled data automatically, it is facing with the problem of noisy data due to the wrong labeling problems. The attention mechanism is one of the most popular methods to reduce the influence of mislabeled data. However, regardless of the correlation among relations, the most existing methods treat all relationships as independent classes. In general, the definitions of relations contain rich semantic information, which improves the performance of the model, especially when classifying long-tail relations which lacks training data. Based on this idea, we propose a novel neural network architecture with an attention mechanism in this paper. First, we use bidirectional GRU to encode relation definitions as the context representations of relations. Then, we use the merge attention mechanism to make full use of the hidden states obtained by the GRU. To help the model make full use of the context of the entities, we also introduce semantic weights, calculated by the length of the shortest path between entities and words in the dependency tree. We conduct experiments on the widely used New York Times relation extraction corpus, and the results demonstrate that our model outperforms most of the state-of-the-art models.

Щодо укладання електронного словника моделей багатокомпонентних складних речень української мови

The purpose of this study is to construct an automatic syntactic analysis (ASA) and, as a result, to compile a dictionary of models of multicomponent complex sentences for studying the fectures of the linear structure of Ukrainian text. The process includes two-stages: the first stage is an automatic syntactic analysis of the hierarchical type which results in building of a dependency tree (DT), in the second stage, the sentence structure information is automatically extracted from the obtained graph. ASA is a package of operations performed with a string of morphological information (the result of AMA work) representing the incoming text for determination of syntactic relations between text units. The outgoing text for the ACA is a string of information reduced after the AMA to wordforms. We have studied features of the linear structure of 2000 Ukrainian language sentences in journalistic genre (selection of 52000 words use). Based on the obtained results, we have constructed the real models of the syntactic structure of sentences, in which the relations between simple clauses were presented. All grammatical situations of the linear context were possible manifestations of models in the text. Based on that data, the algorithm for the automatic generation of a complex sentence model was created. These models are linear syntax grammar. All types of syntactic connection between the main and subordinate clauses are recorded algorithmically. Thus, it is possible to build the interpretations of the linear structure of the Ukrainian language sentence almost not using lexical-semantic information. The theoretical value of the paper is in extension of our knowledge about the structure of the syntactic level of the language and the variety of mechanisms functioning at that level. The applied value, is first of all, in creation of the dictionary of compatibility of compound (coordinated) and complex (subordinated) sentences, and in the possibility of constructing requests to the Ukrainian language Corpus in order to mine from the text definite models sentences, creating own dictionaries of authors and styles.

Tree-Structured Neural Networks With Topic Attention for Social Emotion Classification

Social emotion classification studies the emotion distribution evoked by an article among numerous readers. Although recently neural network-based methods can improve the classification performance compared with the previous word-emotion and topic-emotion approaches, they have not fully utilized some important sentence language features and document topic features. In this paper, we propose a new neural network architecture exploiting both the syntactic information of a sentence and topic distribution of a document. The proposed architecture first constructs a tree-structured long short-term memory (Tree-LSTM) network based on the sentence syntactic dependency tree to obtain a sentence vector representation. For a multi-sentence document, we then use a Chain-LSTM network to obtain the document representation from its sentences’ hidden states. Furthermore, we design a topic-based attention mechanism with two attention levels. The word-level attention is used for weighting words of a single-sentence document and the sentence-level attention for weighting sentences of a multi-sentence document. The experiments on three public datasets show that the proposed scheme outperforms the state-of-the-art ones in terms of higher average Pearson correlation coefficient and MicroF1 performance.