Large Code Repositories Research Articles

On Representation Learning-based Methods for Effective, Efficient, and Scalable Code Retrieval

Code retrieval consists of finding relevant code snippets regarding a programmer’s query — an increasingly important task due to software ubiquity. Although significant progress has been made, there is still a need for refinement and improvement in current code retrieval solutions. Besides effective, we argue that solutions for the code retrieval task should be efficient and scalable to ever-increasing large code repositories. This paper introduces xCoFormer, the first representation learning-based model for the code retrieval task in the literature, which meets all these requirements. Our contributions include: (i) an interactive tag-training method that efficiently places a query closer to its relevant codes in the embedding space and (ii) use of a specialized loss function (N-pair) better suited to the code retrieval task. To evaluate our proposal, we conducted experiments with different versions of xCoFormer against several state-of-the-art deep learning models in datasets with different properties. In our experiments, xCoFormer produced the best cost-effectiveness tradeoff — it is as effective as our closest competitor while being thousands of times faster at search time. xCoFormer’s attention mechanism also (partially) meets the desirable requirement of explainability by exposing the “reasoning” of the model’s predictions. Finally, when used as a “proxy” method, fine-tuned with domain-specific code language models, such as Unixcoder and CodeBERT, our proposal achieves gains of more than 33% in effectiveness when compared to the use of general purpose language models such as BERT and RoBERTa.

I2R: Intra and inter-modal representation learning for code search

Code search, which locates code snippets in large code repositories based on natural language queries entered by developers, has become increasingly popular in the software development process. It has the potential to improve the efficiency of software developers. Recent studies have demonstrated the effectiveness of using deep learning techniques to represent queries and codes accurately for code search. In specific, pre-trained models of programming languages have recently achieved significant progress in code searching. However, we argue that aligning programming and natural languages are crucial as there are two different modalities. Existing pre-train models based approaches for code search do not effectively consider implicit alignments of representations across modalities (inter-modal representation). Moreover, the existing methods do not take into account the consistency constraint of intra-modal representations, making the model ineffective. As a result, we propose a novel code search method that optimizes both intra-modal and inter-modal representation learning. The alignment of the representation between the two modalities is achieved by introducing contrastive learning. Furthermore, the consistency of intra-modal feature representation is constrained by KL-divergence. Our experimental results confirm the model’s effectiveness on seven different test datasets. This paper proposes a code search method that significantly improves existing methods. Our source code is publicly available on GitHub.1

CCStokener: Fast yet accurate code clone detection with semantic token

Code clone detection refers to the discovery of identical or similar code fragments in the code repository. AST-based, PDG-based, and DL-based tools can achieve good results on detecting near-miss clones (i.e., clones with small differences or gaps) by using syntax and semantic information, but they are difficult to apply to large code repositories due to high time complexity. Traditional token-based tools can rapidly detect clones by the low-cost index (i.e., low frequency or k-lines tokens) on sequential source code, but most of them have the poor capability on detecting near-miss clones because of the lack of semantic information.In this study, we propose a fast yet accurate code clone detection tool with the semantic token, called CCStokener. The idea behind the semantic token is to enhance the detection capability of token-based tool via complementing the traditional token with semantic information such as the structural information around the token and its dependency with other tokens in form of n-gram. Specifically, we extract the type of relevant nodes in the AST path of every token and transform these types into a fixed-dimensional vector, then model its semantic information by applying n-gram on its related tokens. Meanwhile, our tool adopts and improves the location–filtration–verification process also used in CCAligner and LVMapper, during which process we build the low-cost k-tokens index to quickly locate the candidate code blocks and speed up detection efficiency. Our experiments show that CCStokener achieves excellent accuracy on detecting more near-miss clone pairs, which exhibits the best recall on Moderately Type-3 clones and detects more true positive clones on four java open-source projects. Moreover, CCStokener attains the best generalization and transferability compared with two DL-based tools (i.e., ASTNN, TBCCD).

ASTENS-BWA: Searching partial syntactic similar regions between source code fragments via AST-based encoded sequence alignment

Code search is a common activity in software development, and code-to-code search can benefit in a wide range of use-case scenarios. Code-to-code search uses a code fragment as the query for searching similar code fragments from large corpora. The results of a search can be applied to some software engineering tasks, such as search-based code recommendation, data-driven program repairing, and software plagiarism detection. To be put into daily use, the code-to-code search needs to find similar code fragments accurately and efficiently in a large dataset. Some search engines can locate exactly similar code, but are not able to search syntactical clones. Therefore, we propose ASTENS-BWA, a novel approach for searching syntactic similar code regions between code fragments via a tree-based sequence alignment. Source code has been transformed into a tree-based sequence that contains the structure information, and a sequence alignment algorithm has been applied to find similar regions. We evaluate ASTENS-BWA on three different tasks, the results demonstrate that our approach can find syntactical similar regions for programming code and retrieve similar code fragments fast and with high accuracy. As a code clone detection tool, ASTENS-BWA can report clone pairs in a high recall, but it needs manually check to reduce the false alarms. ASTENS-BWA is scalable and can report cloned code fragments in seconds for a code corpus of million lines of code. • Novel code clone search approach that scales to large source code repositories. • Quick searching syntactical clones between code fragments via tree-based representation. • Code clone search approach that can outperform state-of-the-art techniques. • Application of the proposed approach towards code recommendation and clone detection.

Scalable Source Code Similarity Detection in Large Code Repositories

Source code similarity are increasingly used in application development to identify clones, isolate bugs, and find copy-rights violations. Similar code fragments can be very problematic due to the fact that errors in the original code must be fixed in every copy. Other maintenance changes, such as extensions or patches, must be applied multiple times. Furthermore, the diversity of coding styles and flexibility of modern languages makes it difficult and cost ineffective to manually inspect large code repositories. Therefore, detection is only feasible by automatic techniques. We present an efficient and scalable approach for similar code fragment identification based on source code control flow graphs fingerprinting. The source code is processed to generate control flow graphs that are then hashed to create a unique fingerprint of the code capturing semantics as well as syntax similarity. The fingerprints can then be efficiently stored and retrieved to perform similarity search between code fragments. Experimental results from our prototype implementation supports the validity of our approach and show its effectiveness and efficiency in comparison with other solutions.

Augmenting and structuring user queries to support efficient free-form code search

Source code terms such as method names and variable types are often different from conceptual words mentioned in a search query. This vocabulary mismatch problem can make code search inefficient. In this paper, we present COde voCABUlary (CoCaBu), an approach to resolving the vocabulary mismatch problem when dealing with free-form code search queries. Our approach leverages common developer questions and the associated expert answers to augment user queries with the relevant, but missing, structural code entities in order to improve the performance of matching relevant code examples within large code repositories. To instantiate this approach, we build GitSearch, a code search engine, on top of GitHub and Stack Overflow Q&A data. We evaluate GitSearch in several dimensions to demonstrate that (1) its code search results are correct with respect to user-accepted answers; (2) the results are qualitatively better than those of existing Internet-scale code search engines; (3) our engine is competitive against web search engines, such as Google, in helping users solve programming tasks; and (4) GitSearch provides code examples that are acceptable or interesting to the community as answers for Stack Overflow questions.

Scalable code clone detection and search based on adaptive prefix filtering

Code clone detection is a well-known software engineering problem that aims to detect all the groups of code blocks or code fragments that are functionally equivalent in a code base. It has numerous and wide ranging important uses in areas such as software metrics, plagiarism detection, aspect mining, copyright infringement investigation, code compaction, virus detection, and detecting bugs. A scalable code clone detection technique, able to process large source code repositories, is crucial in the context of multi-project or Internet-scale code clone detection scenarios. In this paper, we focus on improving the scalability of code clone detection, relative to current state of the art techniques. Our adaptive prefix filtering technique improves the performance of code clone detection for many common execution parameters, when tested on common benchmarks. The experimental results exhibit improvements for commonly used similarity thresholds of between 40% and 80%, in the best case decreasing the execution time up to 11% and increasing the number of filtered candidates up to 63%.

Efficient plagiarism detection for large code repositories

AbstractUnauthorized re‐use of code by students is a widespread problem in academic institutions, and raises liability issues for industry. Manual plagiarism detection is time‐consuming, and current effective plagiarism detection approaches cannot be easily scaled to very large code repositories. While there are practical text‐based plagiarism detection systems capable of working with large collections, this is not the case for code‐based plagiarism detection. In this paper, we propose techniques for detecting plagiarism in program code using text similarity measures and local alignment. Through detailed empirical evaluation on small and large collections of programs, we show that our approach is highly scalable while maintaining similar levels of effectiveness to that of the popular JPlag and MOSS systems. Copyright © 2006 John Wiley & Sons, Ltd.