Abstract Syntax Tree Research Articles

A Novel Refactoring and Semantic Aware Abstract Syntax Tree Differencing Tool and a Benchmark for Evaluating the Accuracy of Diff Tools

A Novel Refactoring and Semantic Aware Abstract Syntax Tree Differencing Tool and a Benchmark for Evaluating the Accuracy of Diff Tools

Malware classification through Abstract Syntax Trees and L-moments

The ongoing evolution of malware presents a formidable challenge to cybersecurity: identifying unknown threats. Traditional detection methods, such as signatures and various forms of static analysis, inherently lag behind these evolving threats. This research introduces a novel approach to malware detection by leveraging the robust statistical capabilities of L-moments and the structural insights provided by Abstract Syntax Trees (ASTs) and applying them to PowerShell. L-moments, recognized for their resilience to outliers and adaptability to diverse distributional shapes, are extracted from network analysis measures like degree centrality, betweenness centrality, and closeness centrality of ASTs. These measures provide a detailed structural representation of code, enabling a deeper understanding of its inherent behaviors and patterns. This approach aims to detect not only known malware but also uncover new, previously unidentified threats. A comprehensive comparison with traditional static analysis methods shows that this approach excels in key performance metrics such as accuracy, precision, recall, and F1 score. These results demonstrate the significant potential of combining L-moments derived from network analysis with ASTs in enhancing malware detection. While static analysis remains an essential tool in cybersecurity, the integration of L-moments and advanced network analysis offers a more effective and efficient response to the dynamic landscape of cyber threats. This study paves the way for future research, particularly in extending the use of L-moments and network analysis into additional areas.

Pattern Mining-Based Warning Prioritization by Refining Abstract Syntax Tree

Static code analysis tools (SATs) are widely used to detect potential defects in software projects. However, the usability of SATs is seriously hindered by a large number of unactionable warnings. Currently, many warning prioritization approaches are proposed to improve the usability of SATs. These approaches mainly extract different warning features to capture the statistical or historical information of warnings, thereby ranking actionable warnings in front of unactionable warnings. Such features are extracted by extremely relying on domain knowledge. However, the precise domain knowledge is difficult to be acquired. Also, the domain knowledge obtained in a project cannot be directly applied to other projects due to different application scenarios among different projects. To address the above problem, we propose a pattern mining-based warning prioritization approach based on the warning-related Abstract Syntax Tree (AST). To automatically mine actionable warning patterns, our approach leverages an advanced technique to collect actionable warnings, designs an algorithm to extract the warning-related AST, and mines patterns from ASTs of all actionable warnings. To prioritize the newly reported warnings, our approach combines exact and fuzzing matching techniques to calculate the similarity score between patterns of the newly reported warnings and the mined actionable warning patterns. We compare our approach with four typical baselines on five open-source and large-scale Java projects. The results show that our approach outperforms four baselines and achieves the maximum MAP (0.76) and MRR (2.19). Besides, a case study on Defect4J dataset demonstrates that our approach can discover 83% of true defects in the top 10 warnings.

Deriving Dependently-Typed OOP from First Principles

The expression problem describes how most types can easily be extended with new ways to produce the type or new ways to consume the type, but not both. When abstract syntax trees are defined as an algebraic data type, for example, they can easily be extended with new consumers, such as print or eval , but adding a new constructor requires the modification of all existing pattern matches. The expression problem is one way to elucidate the difference between functional or data-oriented programs (easily extendable by new consumers) and object-oriented programs (easily extendable by new producers). This difference between programs which are extensible by new producers or new consumers also exists for dependently typed programming, but with one core difference: Dependently-typed programming almost exclusively follows the functional programming model and not the object-oriented model, which leaves an interesting space in the programming language landscape unexplored. In this paper, we explore the field of dependently-typed object-oriented programming by deriving it from first principles using the principle of duality. That is, we do not extend an existing object-oriented formalism with dependent types in an ad-hoc fashion, but instead start from a familiar data-oriented language and derive its dual fragment by the systematic use of defunctionalization and refunctionalization. Our central contribution is a dependently typed calculus which contains two dual language fragments. We provide type- and semantics-preserving transformations between these two language fragments: defunctionalization and refunctionalization. We have implemented this language and these transformations and use this implementation to explain the various ways in which constructions in dependently typed programming can be explained as special instances of the general phenomenon of duality.

Development of Code Evaluation System based on Abstract Syntax Tree

Analyzing and evaluating student-generated code poses a big headache for programming education. Code evaluation is a delicate task requiring accuracy, efficiency, and error-checking. This paper considers one of the many useful tools in code evaluation. It's Abstract Syntax Trees (AST). AST is a robust tool for reading, filtering, and weighing student code. It is built from tree regular expressions for common programming patterns. This paper considers, firstly, integrating AST into code evaluation projects. Secondly and most importantly to the present implementation work is that Python programming language provides excellent compatibility for implementing such a program. Taking advantage of Python's feature that allows the inputting of code as a string, AST records and tracks everything about the user's own source. With AST being integrated into platforms for education, how to evaluate student code has been completely altered; as sites such as LeetCode and HackerRank testify. This paper presents an accurate, efficient and error-aware approach to code evaluation by improving on educational websites incorporating AST. Adapting well to the dynamics of programming education, this elaborate assessment system will provide students with truthful assessments and profound feedback on their coding capabilities. This article shows how AST-based evaluation has revolutionized the way programming is taught, and thoroughly explores its role in evaluating code.

Comparing semantic graph representations of source code: The case of automatic feedback on programming assignments

Static source code analysis techniques are gaining relevance in automated assessment of programming assignments as they can provide less rigorous evaluation and more comprehensive and formative feedback. These techniques focus on source code aspects rather than requiring effective code execution. To this end, syntactic and semantic information encoded in textual data is typically represented internally as graphs, after parsing and other preprocessing stages. Static automated assessment techniques, therefore, draw inferences from intermediate representations to determine the correctness of a solution and derive feedback. Consequently, achieving the most effective semantic graph representation of source code for the specific task is critical, impacting both techniques? accuracy, outcome, and execution time. This paper aims to provide a thorough comparison of the most widespread semantic graph representations for the automated assessment of programming assignments, including usage examples, facets, and costs for each of these representations. A benchmark has been conducted to assess their cost using the Abstract Syntax Tree (AST) as a baseline. The results demonstrate that the Code Property Graph (CPG) is the most feature-rich representation, but also the largest and most space-consuming (about 33% more than AST).

Deobfuscating JavaScript Code Using Character-Based Tokenization

The JavaScript code deployed goes through the process of minification, in which variables are renamed using single-character names and spaces are removed in order for the files to have a smaller size, thus loading faster. Because of this, the code becomes unintelligible, making it harder to be analyzed manually. Since JavaScript experts can under- stand it, machine learning approaches to deobfuscate the minified file are possible. Thus, we propose a technique that finds a fitting name for each obfuscated variable, which is both intuitive and meaningful based on the usage of that variable, based on a Sequence-to-Sequence model, which generates the name character by character to cover all the possible variable names. The proposed approach achieves an average exact name generation accuracy of 70.53%, outperforming the state-of-the-art by 12%. Keywords and phrases: JavaScript deobfuscation, variable name prediction, Deep Learning, Recurrent Neural Network, Abstract Syntax Tree.

The Source Code Comment Generation based on Abstract Syntax Tree

Code summarization provides the main aim described in natural language of the given function; it can benefit many tasks in software engineering. Due to the special grammar and syntax structure of programming languages and various shortcomings of different deep neural networks, the accuracy of existing code summarization approaches is not good enough. We proposes to use abstract syntax trees for source code summarization .Our solution is inspired by recent advances in neural machine translation, as well as an approach called SBT by Hu et al. We evaluate our approach using the automated metric BLEU and compare it to other relevant models.

Analysis of Program Representations Based on Abstract Syntax Trees and Higher-Order Markov Chains for Source Code Classification Task

In this paper we consider the research and development of classifiers that are trained to predict the task solved by source code. Possible applications of such task detection algorithms include method name prediction, hardware–software partitioning, programming standard violation detection, and semantic code duplication search. We provide the comparative analysis of modern approaches to source code transformation into vector-based representations that extend the variety of classification and clustering algorithms that can be used for intelligent source code analysis. These approaches include word2vec, code2vec, first-order and second-order Markov chains constructed from abstract syntax trees (AST), histograms of assembly language instruction opcodes, and histograms of AST node types. The vectors obtained with the forementioned approaches are then used to train such classification algorithms as k-nearest neighbor (KNN), support vector machine (SVM), random forest (RF), and multilayer perceptron (MLP). The obtained results show that the use of program vectors based on first-order AST-based Markov chains with an RF-based classifier leads to the highest accuracy, precision, recall, and F1 score. Increasing the order of Markov chains considerably increases the dimensionality of a vector, without any improvements in classifier quality, so we assume that first-order Markov chains are best suitable for real world applications. Additionally, the experimental study shows that first-order AST-based Markov chains are least sensitive to the used classification algorithm.

Deep learning with class-level abstract syntax tree and code histories for detecting code modification requirements

Improving code quality is one of the most significant issues in the software industry. Deep learning is an emerging area of research for detecting code smells and addressing refactoring requirements. The aim of this study is to develop a deep learning-based system for code modification analysis to predict the locations and types of code modifications, while significantly reducing the need for manual labeling. We created an experimental dataset by collecting historical code data from open-source project repositories on the Internet. We introduce a novel class-level abstract syntax tree-based code embedding method for code analysis. A recurrent neural network was employed to effectively identify code modification requirements. Our system achieves an average accuracy of approximately 83% across different repositories and 86% for the entire dataset. These findings indicate that our system provides higher performance than the method-based and text-based code embedding approaches. In addition, we performed a comparative analysis with a static code analysis tool to justify the readiness of the proposed model for deployment. The correlation coefficient between the outputs demonstrates a significant correlation of 67%. Consequently, this research highlights that the deep learning-based analysis of code histories empowers software teams in identifying potential code modification requirements.

Embedding by Unembedding

Embedding is a language development technique that implements the object language as a library in a host language. There are many advantages of the approach, including being lightweight and the ability to inherit features of the host language. A notable example is the technique of HOAS, which makes crucial use of higher-order functions to represent abstract syntax trees with binders. Despite its popularity, HOAS has its limitations. We observe that HOAS struggles with semantic domains that cannot be naturally expressed as functions, particularly when open expressions are involved. Prominent examples of this include incremental computation and reversible/bidirectional languages. In this paper, we pin-point the challenge faced by HOAS as a mismatch between the semantic domain of host and object language functions, and propose a solution. The solution is based on the technique of unembedding , which converts from the finally-tagless representation to de Bruijn-indexed terms with strong correctness guarantees. We show that this approach is able to extend the applicability of HOAS while preserving its elegance. We provide a generic strategy for Embedding by Unembedding, and then demonstrate its effectiveness with two substantial case studies in the domains of incremental computation and bidirectional transformations. The resulting embedded implementations are comparable in features to the state-of-the-art language implementations in the respective areas.

Development of an algorithm for code clone detection in source code based on abstract syntax tree

The object of research of this work is the algorithm for searching for duplicates in the program code based on the Abstract Syntaxes Tree (AST). The main tasks solved within the framework of this study are the detection of duplicate code and the search for vulnerabilities in the program code. The obtained results showed that the proposed algorithm is resistant to type 1 and 2 clones, which means its effectiveness in detecting similar code fragments with identical or variant text. However, for type 3 and 4 clones, the algorithm may show less efficiency due to the change in the AST structure for these types of clones. Experimental studies of the proposed algorithm showed that the algorithm can detect matches between unrelated files due to the presence of typical AST chains present in many programs. This can lead to a certain level of false positives in the detection of duplicates. Testing of the algorithm in the task of finding vulnerabilities showed that: The best recognition is observed for the «SQL injection» vulnerability, but it also has the highest number of false positives. Memory leak and null pointer dereferencing vulnerabilities are detected with equal effectiveness and false positives. «Buffer overflow» has the lowest recognition rate but fewer false positives compared to «SQL injection». The study showed that the use of AST allows for the effective detection of duplicate code and vulnerabilities in the software code. The developed tool can help software developers reduce maintenance efforts, improve code quality, and ensure software product security.

Snippet Comment Generation Based on Code Context Expansion

Code commenting plays an important role in program comprehension. Automatic comment generation helps improve software maintenance efficiency. The code comments to annotate a method mainly include header comments and snippet comments. The header comment aims to describe the functionality of the entire method, thereby providing a general comment at the beginning of the method. The snippet comment appears at multiple code segments in the body of a method, where a code segment is called a code snippet. Both of them help developers quickly understand code semantics, thereby improving code readability and code maintainability. However, existing automatic comment generation models mainly focus more on header comments because there are public datasets to validate the performance. By contrast, it is challenging to collect datasets for snippet comments because it is difficult to determine their scope. Even worse, code snippets are often too short to capture complete syntax and semantic information. To address this challenge, we propose a novel S nippet C omment Gen eration approach called SCGen . First, we utilize the context of the code snippet to expand the syntax and semantic information. Specifically, 600,243 snippet code-comment pairs are collected from 959 Java projects. Then, we capture variables from code snippets and extract variable-related statements from the context. After that, we devise an algorithm to parse and traverse abstract syntax tree (AST) information of code snippets and corresponding context. Finally, SCGen generates snippet comments after inputting the source code snippet and corresponding AST information into a sequence-to-sequence-based model. We conducted extensive experiments on the dataset we collected to evaluate our SCGen . Our approach obtains 18.23 in BLEU-4 metrics, 18.83 in METEOR, and 23.65 in ROUGE-L, which outperforms state-of-the-art comment generation models.

TaintGuard: Preventing implicit privilege leakage in smart contract based on taint tracking at abstract syntax tree level

Solidity smart contracts play an important role in enabling interactions and collaborations among users in a decentralized blockchain-based community, and their security has received much attention in the industry. One of the most serious security issues faced by Solidity smart contracts is that the use of delegatecall to make cross-contract calls will tamper with contract privilege, which is difficult to detect. To improve the robustness of smart contract privilege control mechanisms at the level of source code, this paper designs and implements a new cross-contract static analysis tool called TaintGuard based on the abstract syntax tree of Solidity code and by combining taint analysis and instrumentation monitoring. TaintGuard uses solc to obtain the abstract syntax tree of the contract source code and then traverses the nodes of the tree to filter call relations that use delegatecall for cross-contract calls. Then, TaintGuard obtains the control flow graph within the target function, determines whether there are problematic paths in the control flow graph that may cause implicit leakage of contract privileges through static taint analysis, and instruments monitoring code in the corresponding position of the program. In this way, the contract status in the runtime stage is monitored to prevent the contract privileges from being tampered with by malicious callers. Finally, TaintGuard is evaluated against existing static analysis tools for Solidity smart contracts. The evaluation results indicate that TaintGuard outperforms other tools in terms of execution efficiency. Also, it can effectively fix the contract code with privilege leaks with a acceptable overhead.

An Abstract Syntax Tree based static fuzzing mutation for vulnerability evolution analysis

An Abstract Syntax Tree based static fuzzing mutation for vulnerability evolution analysis

"A Multi-Pass Compiler with Code Optimized Abstract Syntax Tree"

"A Multi-Pass Compiler with Code Optimized Abstract Syntax Tree"

Storage State Analysis and Extraction of Ethereum Blockchain Smart Contracts

In migrating and upgrading an Ethereum smart contract, it is necessary to transfer both the code as well as the stored data. Various methods attempt to migrate or upgrade a smart contract, but they are mostly manual, error-prone, and applicable only before deployment. Further, they have challenges in extracting the storage state of complex mapping data structures along with their keys. In this work, we present Smartmuv as an automatic source-code-based static analysis tool to analyze and extract the state from the storage-trie of smart contracts. Based on the abstract syntax tree and the control flow graphs of the Solidity source code, the tool analyzes each state variable including mapping types along the inheritance hierarchy. It also provides the upgrade algorithm that initializes the extracted state in the constructor of new smart contract. Smartmuv safely approximates the origin of the keys used in the mapping to extract values and has been able to extract the mapping state of 23,673 smart contracts with 95.7% overall precision. Moreover, we also validate the Smartmuv’s extracted state with the third-party tool Etherscan.

Towards Learning Generalizable Code Embeddings Using Task-agnostic Graph Convolutional Networks

Code embeddings have seen increasing applications in software engineering (SE) research and practice recently. Despite the advances in embedding techniques applied in SE research, one of the main challenges is their generalizability. A recent study finds that code embeddings may not be readily leveraged for the downstream tasks that the embeddings are not particularly trained for. Therefore, in this article, we propose GraphCodeVec , which represents the source code as graphs and leverages the Graph Convolutional Networks to learn more generalizable code embeddings in a task-agnostic manner. The edges in the graph representation are automatically constructed from the paths in the abstract syntax trees, and the nodes from the tokens in the source code. To evaluate the effectiveness of GraphCodeVec , we consider three downstream benchmark tasks (i.e., code comment generation, code authorship identification, and code clones detection) that are used in a prior benchmarking of code embeddings and add three new downstream tasks (i.e., source code classification, logging statements prediction, and software defect prediction), resulting in a total of six downstream tasks that are considered in our evaluation. For each downstream task, we apply the embeddings learned by GraphCodeVec and the embeddings learned from four baseline approaches and compare their respective performance. We find that GraphCodeVec outperforms all the baselines in five out of the six downstream tasks, and its performance is relatively stable across different tasks and datasets. In addition, we perform ablation experiments to understand the impacts of the training context (i.e., the graph context extracted from the abstract syntax trees) and the training model (i.e., the Graph Convolutional Networks) on the effectiveness of the generated embeddings. The results show that both the graph context and the Graph Convolutional Networks can benefit GraphCodeVec in producing high-quality embeddings for the downstream tasks, while the improvement by Graph Convolutional Networks is more robust across different downstream tasks and datasets. Our findings suggest that future research and practice may consider using graph-based deep learning methods to capture the structural information of the source code for SE tasks.

A Heterogeneous Graph to Abstract Syntax Tree Framework for Text-to-SQL.

Text-to-SQL is the task of converting a natural language utterance plus the corresponding database schema into a SQL program. The inputs naturally form a heterogeneous graph while the output SQL can be transduced into an abstract syntax tree (AST). Traditional encoder-decoder models ignore higher-order semantics in heterogeneous graph encoding and introduce permutation biases during AST construction, thus incapable of exploiting the refined structure knowledge precisely. In this work, we propose a generic heterogeneous graph to abstract syntax tree (HG2AST) framework to integrate dedicated structure knowledge into statistics-based models. On the encoder side, we leverage a line graph enhanced encoder (LGESQL) to iteratively update both node and edge features through dual graph message passing and aggregation. On the decoder side, a grammar-based decoder first constructs the equivalent SQL AST and then transforms it into the desired SQL via post-processing. To avoid over-fitting permutation biases, we propose a golden tree-oriented learning (GTL) algorithm to adaptively control the expanding order of AST nodes. The graph encoder and tree decoder are combined into a unified framework through two auxiliary modules. Extensive experiments on various text-to-SQL datasets, including single/multi-table, single/cross-domain, and multilingual settings, demonstrate the superiority and broad applicability.

A Category Theoretic View of Contextual Types: From Simple Types to Dependent Types

We describe the categorical semantics for a simply typed variant and a simplified dependently typed variant of Cocon , a contextual modal type theory where the box modality mediates between the weak function space that is used to represent higher-order abstract syntax (HOAS) trees and the strong function space that describes (recursive) computations about them. What makes Cocon different from standard type theories is the presence of first-class contexts and contextual objects to describe syntax trees that are closed with respect to a given context of assumptions. Following M. Hofmann’s work, we use a presheaf model to characterise HOAS trees. Surprisingly, this model already provides the necessary structure to also model Cocon . In particular, we can capture the contextual objects of Cocon using a comonad ♭ that restricts presheaves to their closed elements. This gives a simple semantic characterisation of the invariants of contextual types (e.g. substitution invariance) and identifies Cocon as a type-theoretic syntax of presheaf models. We further extend this characterisation to dependent types using categories with families and show that we can model a fragment of Cocon without recursor in the Fitch-style dependent modal type theory presented by Birkedal et al.