Syntax Tree Research Articles

Comparison of Program Representations on Vulnerability Detection with Graph Neural Networks

As software vulnerabilities have surged, efforts to discover them have increased. The syntactic and semantic information of a program is required to detect vulnerabilities. Each information can be represented as a graph, such as Abstract Syntax Tree and Program Dependency Graph. In this paper, the program representations were extracted using various static analysis tools, including Clang Static Analyzer, Joern, and SVF, and compared using Graph Neural Networks to select the appropriate representations for vulnerability detection in C/C++. From the comparison, PDG shows the best performance among the multiple representations. This result indicates a suitable program representation and a tool for vulnerability detection that can be utilized in research utilizing graph neural networks.

Relation-Aware Graph Transformer for SQL-to-Text Generation

Generating natural language descriptions for structured representation (e.g., a graph) is an important yet challenging task. In this work, we focus on SQL-to-text, a task that maps a SQL query into the corresponding natural language question. Previous work represents SQL as a sparse graph and utilizes a graph-to-sequence model to generate questions, where each node can only communicate with k-hop nodes. Such a model will degenerate when adapted to more complex SQL queries due to the inability to capture long-term and the lack of SQL-specific relations. To tackle this problem, we propose a relation-aware graph transformer (RGT) to consider both the SQL structure and various relations simultaneously. Specifically, an abstract SQL syntax tree is constructed for each SQL to provide the underlying relations. We also customized self-attention and cross-attention strategies to encode the relations in the SQL tree. Experiments on benchmarks WikiSQL and Spider demonstrate that our approach yields improvements over strong baselines.

Vector Symbolic Architectures for Context-Free Grammars

Vector symbolic architectures (VSA) are a viable approach for the hyperdimensional representation of symbolic data, such as documents, syntactic structures, or semantic frames. We present a rigorous mathematical framework for the representation of phrase structure trees and parse trees of context-free grammars (CFG) in Fock space, i.e. infinite-dimensional Hilbert space as being used in quantum field theory. We define a novel normal form for CFG by means of term algebras. Using a recently developed software toolbox, called FockBox, we construct Fock space representations for the trees built up by a CFG left-corner (LC) parser. We prove a universal representation theorem for CFG term algebras in Fock space and illustrate our findings through a low-dimensional principal component projection of the LC parser state. Our approach could leverage the development of VSA for explainable artificial intelligence (XAI) by means of hyperdimensional deep neural computation.

What You See is What it Means! Semantic Representation Learning of Code based on Visualization and Transfer Learning

Recent successes in training word embeddings for Natural Language Processing ( NLP ) tasks have encouraged a wave of research on representation learning for source code, which builds on similar NLP methods. The overall objective is then to produce code embeddings that capture the maximum of program semantics. State-of-the-art approaches invariably rely on a syntactic representation (i.e., raw lexical tokens, abstract syntax trees, or intermediate representation tokens) to generate embeddings, which are criticized in the literature as non-robust or non-generalizable. In this work, we investigate a novel embedding approach based on the intuition that source code has visual patterns of semantics. We further use these patterns to address the outstanding challenge of identifying semantic code clones. We propose the WySiWiM ( ‘ ‘What You See Is What It Means ” ) approach where visual representations of source code are fed into powerful pre-trained image classification neural networks from the field of computer vision to benefit from the practical advantages of transfer learning. We evaluate the proposed embedding approach on the task of vulnerable code prediction in source code and on two variations of the task of semantic code clone identification: code clone detection (a binary classification problem), and code classification (a multi-classification problem). We show with experiments on the BigCloneBench (Java), Open Judge (C) that although simple, our WySiWiM approach performs as effectively as state-of-the-art approaches such as ASTNN or TBCNN. We also showed with data from NVD and SARD that WySiWiM representation can be used to learn a vulnerable code detector with reasonable performance (accuracy ∼90%). We further explore the influence of different steps in our approach, such as the choice of visual representations or the classification algorithm, to eventually discuss the promises and limitations of this research direction.

Detecting Web-Based Attacks with SHAP and Tree Ensemble Machine Learning Methods

Attacks using Uniform Resource Locators (URLs) and their JavaScript (JS) code content to perpetrate malicious activities on the Internet are rampant and continuously evolving. Methods such as blocklisting, client honeypots, domain reputation inspection, and heuristic and signature-based systems are used to detect these malicious activities. Recently, machine learning approaches have been proposed; however, challenges still exist. First, blocklist systems are easily evaded by new URLs and JS code content, obfuscation, fast-flux, cloaking, and URL shortening. Second, heuristic and signature-based systems do not generalize well to zero-day attacks. Third, the Domain Name System allows cybercriminals to easily migrate their malicious servers to hide their Internet protocol addresses behind domain names. Finally, crafting fully representative features is challenging, even for domain experts. This study proposes a feature selection and classification approach for malicious JS code content using Shapley additive explanations and tree ensemble methods. The JS code features are obtained from the Abstract Syntax Tree form of the JS code, sample JS attack codes, and association rule mining. The malicious and benign JS code datasets obtained from Hynek Petrak and the Majestic Million Service were used for performance evaluation. We compared the performance of the proposed method to those of other feature selection methods in the task of malicious JS code content detection. With a recall of 0.9989, our experimental results show that the proposed approach is a better prediction model.

Accusative Licensing of Nouns in Turkish

The observation that some Turkish nouns license accusative marking of their complements has puzzled linguistics for the last three decades. The first and the most common hypothesis is that such nouns are part of light-verb compounds in their infinitive nominal form where the verbal part is dropped. In this paper, I argue that light-verb hypothesis is incorrect by providing several constructions where it fails to account for. I propose an alternative, two-part hypothesis. The first part is independent of the light-verb constructions and claims that in Turkish, the accusative case is licensed by verb stems rather than finite verbs. This is a natural corollary of a yet larger hypothesis about the structure of syntax trees for languages like Turkish which claims that the syntax trees go deeper than the word level. The leaves of such trees are morphemes or morpheme groups and the whole of infection/derivation is considered within a single morphosyntactic structure. The implication is that as far as the case-licensing is concerned, only the verb stems license accusative case. In constructions where a deverbal noun seems to be licensing accusative case, it is actually the verb stem in the noun’s derivation tree that licenses the accusative case. The second part of the hypothesis is the claim that the case-licensing nouns are actually lexicalizations involving multiple adjacent leaves in the morphosyntactic tree. The most common manifestations of this are the lexicalizations of verb stem-participle pairs through replacement with templatic surface forms borrowed from Arabic. The new proposal naturally explains the observation that accusative licensing is most common for a small set of words borrowed from Arabic. I illustrate that these words are the templatic forms of their multi-morpheme counterparts in Turkish which correspond to deverbal nouns, adjectives or adverbs.

Syntax‐based metamorphic relation prediction via the bagging framework

AbstractSoftware testing is an indispensable part of the software engineering industry, which guarantees product reliability and safety. Traditional testing approaches face the testing Oracle problem, they are difficult to construct the expected outputs with the increasing of program complexity. As a result, metamorphic testing, which tests the program by examining the relationship between the execution results, is proposed. However, existing manual metamorphic relation construction requires huge effects of domain experts, and automatic methods are unstable and inefficient due to the insufficient software feature mining. Hence, we proposed a multi‐dimensional program structure‐based metamorphic relation prediction approach, which is composed of feature extraction and prediction model building. In the feature extraction stage, the testing program is converted to multiple intermediate structures (such as control flow graphs and abstract syntax trees) to explore its features. In the prediction model building stage, the extracted feature set is used as the training set, and a novel semi‐supervised support vector machine‐bagging‐K‐nearest neighbors algorithm is designed to train the prediction model. Besides, a two‐phase hybrid granularity search algorithm is proposed to improve the prediction performance by selecting the optimal number of weak classifiers. Compared with existing approaches, our proposed model can improve the accuracy by around 14%.

Understanding security vulnerabilities in student code: A case study in a non-security course

Secure coding education is quite important for students to acquire the skills to quickly adapt to the evolving threats towards the software they are expected to create once they graduate. Educators are also more aware of this situation and incorporate teaching security in their respective fields. An effective application of this is only possible by cultivating the teaching and learning perspectives. Understanding the security awareness and practice of students is useful as an initial step to create a baseline for teaching methods and content. In this paper, we first survey to investigate how students approach security and what could motivate them to learn and apply security practices. Then, we analyze the source code for 6 semesters of coding assignments for 2 tasks using a source code vulnerability analysis tool. In our analysis, we report the types of vulnerabilities and various aspects between them while incorporating the effect of student grades. We then explore the lexical and structural features of security in student code using data analysis and machine learning. For the lexical analysis, we build a classifier to extract informative features and for the structural analysis, we utilize Syntax Trees to represent code and perform clustering in terms of structural features where clusters themselves yield different vulnerability levels.

How to Build Your Own ASP-based System?!

AbstractAnswer Set Programming, or ASP for short, has become a popular and sophisticated approach to declarative problem solving. Its popularity is due to its attractive modeling-grounding-solving workflow that provides an easy approach to problem solving, even for laypersons outside computer science. However, in contrast to ASP’s ease of use, the high degree of sophistication of the underlying technology makes it even hard for ASP experts to put ideas into practice whenever this involves modifying ASP’s machinery. For addressing this issue, this tutorial aims at enabling users to build their own ASP-based systems. More precisely, we show how the ASP system clingo can be used for extending ASP and for implementing customized special-purpose systems. To this end, we propose two alternatives. We begin with a traditional AI technique and show how metaprogramming can be used for extending ASP. This is a rather light approach that relies on clingo’s reification feature to use ASP itself for expressing new functionalities. The second part of this tutorial uses traditional programming (in Python) for manipulating clingo via its application programming interface. This approach allows for changing and controlling the entire model-ground-solve workflow of ASP. Central to this is clingo’s new Application class that allows us to draw on clingo’s infrastructure by customizing processes similar to the one in clingo. For instance, we may apply manipulations to programs’ abstract syntax trees, control various forms of multi-shot solving, and set up theory propagators for foreign inferences. A cross-sectional structure, spanning meta as well as application programming, is clingo’s intermediate format, aspif, that specifies the interface among the underlying grounder and solver. We illustrate the aforementioned concepts and techniques throughout this tutorial by means of examples and several nontrivial case studies. In particular, we show how clingo can be extended by difference constraints and how guess-and-check programming can be implemented with both meta and application programming.

A BERT-Based Generation Model to Transform Medical Texts to SQL Queries for Electronic Medical Records: Model Development and Validation.

BackgroundElectronic medical records (EMRs) are usually stored in relational databases that require SQL queries to retrieve information of interest. Effectively completing such queries can be a challenging task for medical experts due to the barriers in expertise. Existing text-to-SQL generation studies have not been fully embraced in the medical domain.ObjectiveThe objective of this study was to propose a neural generation model that can jointly consider the characteristics of medical text and the SQL structure to automatically transform medical texts to SQL queries for EMRs.MethodsWe proposed a medical text–to-SQL model (MedTS), which employed a pretrained Bidirectional Encoder Representations From Transformers model as the encoder and leveraged a grammar-based long short-term memory network as the decoder to predict the intermediate representation that can easily be transformed into the final SQL query. We adopted the syntax tree as the intermediate representation rather than directly regarding the SQL query as an ordinary word sequence, which is more in line with the tree-structure nature of SQL and can also effectively reduce the search space during generation. Experiments were conducted on the MIMICSQL dataset, and 5 competitor methods were compared.ResultsExperimental results demonstrated that MedTS achieved the accuracy of 0.784 and 0.899 on the test set in terms of logic form and execution, respectively, which significantly outperformed the existing state-of-the-art methods. Further analyses proved that the performance on each component of the generated SQL was relatively balanced and offered substantial improvements.ConclusionsThe proposed MedTS was effective and robust for improving the performance of medical text–to-SQL generation, indicating strong potential to be applied in the real medical scenario.

Software defect prediction using hybrid model (CBIL) of convolutional neural network (CNN) and bidirectional long short-term memory (Bi-LSTM).

In recent years, the software industry has invested substantial effort to improve software quality in organizations. Applying proactive software defect prediction will help developers and white box testers to find the defects earlier, and this will reduce the time and effort. Traditional software defect prediction models concentrate on traditional features of source code including code complexity, lines of code, etc. However, these features fail to extract the semantics of source code. In this research, we propose a hybrid model that is called CBIL. CBIL can predict the defective areas of source code. It extracts Syntax Tree (AST) tokens as vectors from source code. Mapping and word embedding turn integer vectors into dense vectors. Then, Convolutional Neural Network (CNN) extracts the semantics of AST tokens. After that, Bidirectional Long Short-Term Memory (Bi-LSTM) keeps key features and ignores other features in order to enhance the accuracy of software defect prediction. The proposed model CBIL is evaluated on a sample of seven open-source Java projects of the PROMISE dataset. CBIL is evaluated by applying the following evaluation metrics: F-measure and area under the curve (AUC). The results display that CBIL model improves the average of F-measure by 25% compared to CNN, as CNN accomplishes the top performance among the selected baseline models. In average of AUC, CBIL model improves AUC by 18% compared to Recurrent Neural Network (RNN), as RNN accomplishes the top performance among the selected baseline models used in the experiments.

Multi-triage: A multi-task learning framework for bug triage

Assigning developers and allocating issue types are two important tasks in the bug triage process. Existing approaches tackle these two tasks separately, which is time-consuming due to repetition of effort and negating the values of correlated information between tasks. In this paper, a multi-triage model is proposed that resolves both tasks simultaneously via multi-task learning (MTL). First, both tasks can be regarded as a classification problem, based on historical issue reports. Second, performances on both tasks can be improved by jointly interpreting the representations of the issue report information. To do so, a text encoder and abstract syntax tree (AST) encoder are used to extract the feature representation of bug descriptions and code snippets accordingly. Finally, due to the disproportionate ratio of class labels in training datasets, the contextual data augmentation approach is introduced to generate syntactic issue reports to balance the class labels. Experiments were conducted on eleven open-source projects to demonstrate the effectiveness of this model compared with state-of-the-art methods.

Ten Trees a Day: How learning science and a story of multilingual buffalo help students learn syntax

This paper describes an activity designed to help students improve skills in drawing syntax tree structures without significantly increasing instructor grading time. In this formative exercise, students draw ten trees prior to each class period, correct their own work, and reflect on their mistakes. This assignment incorporates many practices that research on learning suggests are essential for understanding and retention of material. In addition, this exercise incorporates some best practices on effective feedback. The activity works best when students understand the science behind it, so discussion of the pedagogical reasons for the exercise is essential. Further, overt discussion of how to learn helps students develop effective skills for learning linguistics. Self-correct homework assignments like this can be applied to many courses that involve learning skills or terminology.

MS-Transformer: Introduce multiple structural priors into a unified transformer for encoding sentences

Transformers have been widely utilized in recent NLP studies. Unlike CNNs or RNNs, the vanilla Transformer is position-insensitive, and thus is incapable of capturing the structural priors between sequences of words. Existing studies commonly apply one single mask strategy on Transformers for incorporating structural priors while failing at modeling more abundant structural information of texts. In this paper, we aim at introducing multiple types of structural priors into Transformers, proposing the Multiple Structural Priors Guided Transformer (MS-Transformer) that transforms different structural priors into different attention heads by using a novel multi-mask based multi-head attention mechanism. In particular, we integrate two categories of structural priors, including the sequential order and the relative position of words. For the purpose of capturing the latent hierarchical structure of the texts, we extract these information not only from the word contexts but also from the dependency syntax trees. Experimental results on three tasks show that MS-Transformer achieves significant improvements against other strong baselines.

Improving Ponzi Scheme Contract Detection Using Multi-Channel TextCNN and Transformer.

With the development of blockchain technologies, many Ponzi schemes disguise themselves under the veil of smart contracts. The Ponzi scheme contracts cause serious financial losses, which has a bad effect on the blockchain. Existing Ponzi scheme contract detection studies have mainly focused on extracting hand-crafted features and training a machine learning classifier to detect Ponzi scheme contracts. However, the hand-crafted features cannot capture the structural and semantic feature of the source code. Therefore, in this study, we propose a Ponzi scheme contract detection method called MTCformer (Multi-channel Text Convolutional Neural Networks and Transofrmer). In order to reserve the structural information of the source code, the MTCformer first converts the Syntax Tree (AST) of the smart contract code to the specially formatted code token sequence via the Structure-Based Traversal (SBT) method. Then, the MTCformer uses multi-channel TextCNN (Text Convolutional Neural Networks) to learn local structural and semantic features from the code token sequence. Next, the MTCformer employs the Transformer to capture the long-range dependencies of code tokens. Finally, a fully connected neural network with a cost-sensitive loss function in the MTCformer is used for classification. The experimental results show that the MTCformer is superior to the state-of-the-art methods and its variants in Ponzi scheme contract detection.

Intelligent mining vulnerabilities in python code snippets

In the software development process, many developers learn from code snippets in the open-source community to implement specific functions. However, few people think about whether these code have vulnerabilities, which provides channels for developing unsafe programs. To this end, this paper constructs a source code snippets vulnerability mining system named PyVul based on deep learning to automatically detect the security of code snippets in the open source community. PyVul builds abstract syntax tree (AST) for the source code to extract its code feature, and then introduces the bidirectional long-term short-term memory (BiLSTM) neural network algorithm to detect vulnerability codes. If it is vulnerable code, the further constructed a multi-classification model could analyze the context discussion contents in associated threads, to classify the code vulnerability type based the content description. Compared with traditional detection methods, this method can identify vulnerable code and classify vulnerability type. The accuracy of the proposed model can reach 85%. PyVul also found 138 vulnerable code snippets in the real public open-source community. In the future, it can be used in the open-source community for vulnerable code auditing to assist users in safe development.

A Neural Machine Translation Model Based on Sequence to Dependency

A supervised self-attention network is proposed to introduce the dependency structure into the Transformer model. This scheme is mainly proposed based on the characteristics of the self-attention mechanism in Transformer, which converts the dependency syntax tree into two equivalent adjacency matrices, and then uses the adjacency matrix to supervise the self-attention network in the Transformer’s encoder. So that Transformer learns how to model the dependency structure of the source language. Although this scheme is simple, the effect is remarkable. In particular, there is no need to modify the network structure, as long as two additional loss functions are added in the process of training Transformer. In the decoding process, there is no need to use an external syntactic analyzer to perform syntactic analysis on the source language, but it can automatically construct the dependent syntactic structure of the source language. Experiments demonstrate that the quality of the constructed dependency syntactic structure is also good.

Reasoning about effect interaction by fusion

Effect handlers can be composed by applying them sequentially, each handling some operations and leaving other operations uninterpreted in the syntax tree. However, the semantics of composed handlers can be subtle---it is well known that different orders of composing handlers can lead to drastically different semantics. Determining the correct order of composition is a non-trivial task. To alleviate this problem, this paper presents a systematic way of deriving sufficient conditions on handlers for their composite to correctly handle combinations, such as the sum and the tensor, of the effect theories separately handled. These conditions are solely characterised by the clauses for relevant operations of the handlers, and are derived by fusing two handlers into one using a form of fold/build fusion and continuation-passing style transformation. As case studies, the technique is applied to commutative and distributive interaction of handlers to obtain a series of results about the interaction of common handlers: (a) equations respected by each handler are preserved after handler composition; (b) handling mutable state before any handler gives rise to a semantics in which state operations are commutative with any operations from the latter handler; (c) handling the writer effect and mutable state in either order gives rise to a correct handler of the commutative combination of these two theories.

Deep Contrast Learning Approach for Address Semantic Matching

Address is a structured description used to identify a specific place or point of interest, and it provides an effective way to locate people or objects. The standardization of Chinese place name and address occupies an important position in the construction of a smart city. Traditional address specification technology often adopts methods based on text similarity or rule bases, which cannot handle complex, missing, and redundant address information well. This paper transforms the task of address standardization into calculating the similarity of address pairs, and proposes a contrast learning address matching model based on the attention-Bi-LSTM-CNN network (ABLC). First of all, ABLC use the Trie syntax tree algorithm to extract Chinese address elements. Next, based on the basic idea of contrast learning, a hybrid neural network is applied to learn the semantic information in the address. Finally, Manhattan distance is calculated as the similarity of the two addresses. Experiments on the self-constructed dataset with data augmentation demonstrate that the proposed model has better stability and performance compared with other baselines.

DFlow: A Data Flow Analysis Tool for C/C++

AbstractAbstract syntax trees (ASTs), control flow graphs (CFGs), and data flow analysis (DFA) are prerequisites for static and dynamic analysis and vulnerability detection for programs; thus, obtaining them is significant. Recently, many tools related to generating ASTs, CFGs, and DFA have been proposed. However, most tools can only construct ASTs, very few can construct ASTs and CFGs, and almost none can construct all three. The vast majority of AST, CFG, and DFA tools are for other languages (e.g., Java and Python), and while a few are for C/C++, they are implemented in other languages, creating complex working environments, and overreliance on other language‐related libraries. To address these shortcomings, we present a DFA tool, DFlow, for C/C++. First, a lexical/grammatical analyzer generated by Flex and Bison is used to analyze the program. Second, an AST is constructed from the results; then, a CFG is obtained from the analysis results and the information from the AST. Finally, based on the AST and CFG, DFA is performed, and the vulnerabilities of simple programs are determined. We test some common vulnerable code and common weakness enumeration slicing code, which show the effectiveness of DFlow in program data flow analysis and vulnerability checking. The results show that our tool can implement ASTs, CFGs, and DFA, and we add some rules to the tool for vulnerability detection. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.