Code Completion Tools Research Articles

Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions

Large-scale generative models have enabled the development of AI-powered code completion tools to assist programmers in writing code. Like all AI-powered tools, these code completion tools are not always accurate and can introduce bugs or even security vulnerabilities into code if not properly detected and corrected by a human programmer. One technique that has been proposed and implemented to help programmers locate potential errors is to highlight uncertain tokens. However, little is known about the effectiveness of this technique. Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits, and is subjectively preferred by study participants. In contrast, highlighting tokens according to their probability of being generated does not provide any benefit over the baseline with no highlighting. We further explore the design space of how to convey uncertainty in AI-powered code completion tools and find that programmers prefer highlights that are granular, informative, interpretable, and not overwhelming. This work contributes to building an understanding of what uncertainty means for generative models and how to convey it effectively.

Special Issue on Informatics Education: Exploring the Impact of GitHub Copilot on Health Informatics Education.

The use of artificial intelligence-driven code completion tools, particularly the integration of GitHub Copilot with Visual Studio, has potential implications for Health Informatics education, particularly for students learning SQL and Python. This study aims to evaluate the effectiveness of these tools in solving or assisting with the solution of problems found in Health Informatics coursework, ranging from simple to complex. The study assesses the performance of GitHub Copilot in generating code for Health Informatics coding assignments from graduate classes, with a focus on the impact of detailed explanations on the tool's effectiveness. Findings reveal that GitHub Copilot can generate correct code for straightforward problems. The correctness and effectiveness of solutions decrease with problem complexity, and the tool struggles with the most challenging problems, although performance on complex problems improves with more detailed explanations. The study underscores the relevance of these tools to programming in Health Informatics education but also highlights the need for critical evaluation by students. It concludes with a call for educators to adapt swiftly to this rapidly evolving technology.

Non-Autoregressive Line-Level Code Completion

Software developers frequently use code completion tools to accelerate software development by suggesting the following code elements. Researchers usually employ AutoRegressive (AR) decoders to complete code sequences in a left-to-right, token-by-token fashion. To improve the accuracy and efficiency of code completion, we argue that tokens within a code statement have the potential to be predicted concurrently. In this article, we first conduct an empirical study to analyze the dependency among the target tokens in line-level code completion. The results suggest that it is potentially practical to generate all statement tokens in parallel. To this end, we introduce SANAR, a simple and effective syntax-aware non-autoregressive model for line-level code completion. To further improve the quality of the generated code, we propose an adaptive and syntax-aware sampling strategy to boost the model’s performance. The experimental results obtained from two widely used datasets indicate that our model outperforms state-of-the-art code completion approaches of similar model size by a considerable margin, and is faster than these models with up to 9× speed-up. Moreover, the extensive results additionally demonstrate that the enhancements achieved by SANAR become even more pronounced with larger model sizes, highlighting their significance.

Promoting open science in test-driven software experiments

A core principle of open science is the clear, concise and accessible publication of empirical data, including “raw” observational data as well as processed results. However, in empirical software engineering there are no established standards (de jure or de facto) for representing and “opening” observations collected in test-driven software experiments — that is, experiments involving the execution of software subjects in controlled scenarios. Execution data is therefore usually represented in ad hoc ways, often making it abstruse and difficult to access without significant manual effort. In this paper we present new data structures designed to address this problem by clearly defining, correlating and representing the stimuli and responses used to execute software subjects in test-driven experiments. To demonstrate their utility, we show how they can be used to promote the repetition, replication and reproduction of experimental evaluations of AI-based code completion tools. We also show how the proposed data structures facilitate the incremental expansion of execution data sets, and thus promote their repurposing for new experiments addressing new research questions.

DeepVS: an efficient and generic approach for source code modelling usage

The source code suggestions provided by current IDEs are mostly dependent on static type learning. These suggestions often end up proposing irrelevant suggestions for a peculiar context. Recently, deep learning-based approaches have shown great potential in the modeling of source code for various software engineering tasks. However, these techniques lack adequate generalization and resistance to acclimate the use of such models in a real-world software development environment. This letter presents \textit{DeepVS}, an end-to-end deep neural code completion tool that learns from existing codebases by exploiting the bidirectional Gated Recurrent Unit (BiGRU) neural net. The proposed tool is capable of providing source code suggestions instantly in an IDE by using pre-trained BiGRU neural net. The evaluation of this work is two-fold, quantitative and qualitative. Through extensive evaluation on ten real-world open-source software systems, the proposed method shows significant performance enhancement and its practicality. Moreover, the results also suggest that \textit{DeepVS} tool is capable of suggesting zero-day (unseen) code tokens by learning coding patterns from real-world software systems.

Improving code completion with program history

Code completion is a widely used productivity tool. It takes away the burden of remembering and typing the exact names of methods or classes: As a developer starts typing a name, it provides a progressively refined list of candidates matching the name. However, the candidate list usually comes in alphabetic order, i.e., the environment is only second-guessing the name based on pattern matching, relying on human intervention to pick the correct one. Finding the correct candidate can thus be cumbersome or slower than typing the full name. We present an approach to improve code completion based on recorded program histories. We define a benchmarking procedure measuring the accuracy of a code completion engine and apply it to several completion algorithms on a dataset consisting of the history of several systems. Further, we use the change history data to improve the results offered by code completion tools. Finally, we propose an alternative interface for completion tools that we released to developers and evaluated.