Code Generation Approach Research Articles

On the Effectiveness of Large Language Models in Domain-Specific Code Generation

Large language models (LLMs) such as ChatGPT have shown remarkable capabilities in code generation. Despite significant achievements, they rely on enormous training data to acquire a broad spectrum of open-domain knowledge. Besides, their evaluation revolves around open-domain benchmarks like HumanEval, which primarily consist of programming contests. Therefore, it is hard to fully characterize the intricacies and challenges associated with particular domains (e.g., web, game, and math). In this paper, we conduct an in-depth study of the LLMs in domain-specific code generation. Our results demonstrate that LLMs exhibit sub-optimal performance in generating domain-specific code, due to their limited proficiency in utilizing domain-specific libraries. We further observe that incorporating API knowledge as prompts can empower LLMs to generate more professional code. Based on these findings, we further investigate how to effectively incorporate API knowledge into the code generation process. We experiment with three strategies for incorporating domain knowledge, namely, external knowledge inquirer, chain-of-thought prompting, and chain-of-thought fine-tuning. We refer to these strategies as a new code generation approach called DomCoder . Experimental results show that all strategies of DomCoder improve the effectiveness of domain-specific code generation under certain settings.

Self-Planning Code Generation with Large Language Models

Although large language models (LLMs) have demonstrated impressive ability in code generation, they are still struggling to address the complicated intent provided by humans. It is widely acknowledged that humans typically employ planning to decompose complex problems and schedule solution steps prior to implementation. To this end, we introduce planning into code generation to help the model understand complex intent and reduce the difficulty of problem-solving. This paper proposes a self-planning code generation approach with large language models, which consists of two phases, namely planning phase and implementation phase. Specifically, in the planning phase, LLM plans out concise solution steps from the intent combined with few-shot prompting. Subsequently, in the implementation phase, the model generates code step by step, guided by the preceding solution steps. We conduct extensive experiments on various code-generation benchmarks across multiple programming languages. Experimental results show that self-planning code generation achieves a relative improvement of up to 25.4% in Pass@1 compared to direct code generation, and up to 11.9% compared to Chain-of-Thought of code generation. Moreover, our self-planning approach also enhances the quality of the generated code with respect to correctness, readability, and robustness, as assessed by humans.

MOBICAT: a model-driven engineering approach for automatic GUI code generation for Android applications

MOBICAT: a model-driven engineering approach for automatic GUI code generation for Android applications

An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs

In the digital era of increasing software complexity, improving the development efficiency of safety-critical software is a challenging task faced by academia and industry in domains such as nuclear energy, aviation, the automotive industry, and rail transportation. Recently, people have been excited about using pre-trained large language models (LLMs) such as ChatGPT and GPT-4 to generate code. Professionals in the safety-critical software field are intrigued by the code generation capabilities of LLMs. However, there is currently a lack of systematic case studies in this area. Aiming at the need for automated code generation in safety-critical domains such as nuclear energy and the automotive industry, this paper conducts a case study on generating safety-critical software code using GPT-4 as the tool. Practical engineering cases from the industrial domain are employed. We explore different approaches, including code generation based on overall requirements, specific requirements, and augmented prompts. We propose a novel prompt engineering method called Prompt-FDC that integrates basic functional requirements, domain feature generalization, and domain constraints. This method improves code completeness from achieving 30% functions to 100% functions, increases the code comment rate to 26.3%, and yields better results in terms of code compliance, readability, and maintainability. The code generation approach based on LLMs also introduces a new software development process and V-model lifecycle for safety-critical software. Through systematic case studies, we demonstrate that, with appropriate prompt methods, LLMs can auto-generate safety-critical software code that meets practical engineering application requirements. It is foreseeable that LLMs can be applied to various engineering domains to improve software safety and development efficiency.

A DSL-based Approach for Code Generation and Navigation Process Management in a Single Page Application

Single page applications (SPAs) are web applications that offer a fast and responsive user experience, but also pose some challenges in terms of navigation process management and platform compatibility. In this paper, we propose a DSL-based approach for code generation and management of SPA navigation process, based on the visual vocabulary proposed by Garrett. We use a Domain-Specific Language (DSL) to describe the elements of the SPA, such as pages, connectors, and transitions, in a concise and abstract way. We also propose a metamodel for Garrett vocabulary, which defines the concepts and relationships of the DSL. Then, we apply a Model-Driven Engineering (MDE) technique to transform the DSL specification into a platform-specific project architecture and basic code for an SPA. We present a case study of using our DSL to generate an Angular SPA and evaluate the benefits of our approach in terms of productivity, maintainability, and reusability.

Automatic generation of security protocols attacks specifications and implementations

Confidence in a communication protocol’s security is a key requirement for its deployment and long-term maintenance. Checking if a vulnerability exists and is exploitable requires extensive expertise. The research community has advocated for a systematic approach with formal methods to model and automatically test a protocol against a set of desired security properties. As verification tools reach conclusions, the applicability of their results still requires expert scrutiny. We propose a code generation approach to automatically build both an abstract specification and a concrete implementation of a Dolev-Yao intruder from an abstract attack trace, bridging the gap between theoretical attacks discovered by formal means and practical ones. Through our case studies, we focus on attack traces from the OFMC model checker, Alice&Bob specifications and Java implementations. We introduce a proof-of-concept workflow for concrete attack validation that allows to conveniently integrate, in a user-friendly way, formal methods results into a Model-Driven Development process and at the same time automatically generate a program that allows to demonstrate the attack in practice. In fact, in this contribution, we produce high-level and concrete attack narrations that are both human and machine readable.

BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach

Compiler optimization plays an increasingly important role to boost the performance of machine learning models for data processing and management. With increasingly complex data, the dynamic tensor shape phenomenon emerges for ML models. However, existing ML compilers either can only handle static shape models or expose a series of performance problems for both operator fusion optimization and code generation in dynamic shape scenes. This paper tackles the main challenges of dynamic shape optimization: the fusion optimization without shape value, and code generation supporting arbitrary shapes. To tackle the fundamental challenge of the absence of shape values, it systematically abstracts and excavates the shape information and designs a cross-level symbolic shape representation. With the insight that what fusion optimization relies upon is tensor shape relationships between adjacent operators rather than exact shape values, it proposes the dynamic shape fusion approach based on shape information propagation. To generate code that adapts to arbitrary shapes efficiently, it proposes a compile-time and runtime combined code generation approach. Finally, it presents a complete optimization pipeline for dynamic shape models and implements an industrial-grade ML compiler, named BladeDISC. The extensive evaluation demonstrates that BladeDISC outperforms PyTorch, TorchScript, TVM, ONNX Runtime, XLA, Torch Inductor (dynamic shape), and TensorRT by up to 6.95×, 6.25×, 4.08×, 2.04×, 2.06×, 7.92×, and 4.16× (3.54×, 3.12×, 1.95×, 1.47×, 1.24×, 2.93×, and 1.46× on average) in terms of end-to-end inference speedup on the A10 and T4 GPU, respectively. BladeDISC's source code is publicly available at https://github.com/alibaba/BladeDISC.

Improving domain-specific neural code generation with few-shot meta-learning

Context:Neural code generation aims to automatically generate code snippets guided by Natural Language Descriptions (NLDs). In recent years, various neural code generation models for mainstream Programming Languages (PLs), such as Java and Python, have been proposed and demonostrated significant success in prior studies. Nonetheless, due to the scarcity of available training examples for some domain-specific PLs, such as Solidity, Bash, and Clojure, simply adopting previous neural models may lead to overfitting and inadequate learning. Objective:To overcome this challenge, we propose MetaCoder, a novel meta-learning code generation approach that efficiently extracts general-purpose knowledge from a large-scale source language and rapidly adapts to domain-specific scenarios, even with relatively few samples. Method:MetaCoder employs MAML, a powerful few-shot meta-learning method, to construct a transfer learning framework. This framework learns general-purpose knowledge from large-scale source languages and applies it in domain-specific target languages. To acquire more general-purpose knowledge, heterogeneous sub-tasks are constructed from the source language during the pre-training phase of MAML. As such, combining with CodeBERT and K-means, we design an unsupervised category assignment method for code generation samples, thereby exploiting the n-way k-shot rule to construct the heterogeneous sub-tasks. Consequently, MetaCoder can be applied to the code generation field. Results:We evaluate MetaCoder with both tree-based (e.g., TreeGen) and sequence-based (e.g., CodeGPT) backbones on two domain-specific PLs, including Solidity and Bash. Extensive experiments demonstrate the superior performance of our approach compared to baselines and verified its capability of code generation visually in practice. Conclusion:MetaCoder effectively extracts general-purpose knowledge from large-scale source languages, thereby enhancing model performance. Therefore, we highly recommend MetaCoder as a code generation approach for domain-specific PLs.

A syntax-guided multi-task learning approach for Turducken-style code generation

A syntax-guided multi-task learning approach for Turducken-style code generation

Runtime Construction of Large-Scale Spiking Neuronal Network Models on GPU Devices

Simulation speed matters for neuroscientific research: this includes not only how quickly the simulated model time of a large-scale spiking neuronal network progresses but also how long it takes to instantiate the network model in computer memory. On the hardware side, acceleration via highly parallel GPUs is being increasingly utilized. On the software side, code generation approaches ensure highly optimized code at the expense of repeated code regeneration and recompilation after modifications to the network model. Aiming for a greater flexibility with respect to iterative model changes, here we propose a new method for creating network connections interactively, dynamically, and directly in GPU memory through a set of commonly used high-level connection rules. We validate the simulation performance with both consumer and data center GPUs on two neuroscientifically relevant models: a cortical microcircuit of about 77,000 leaky-integrate-and-fire neuron models and 300 million static synapses, and a two-population network recurrently connected using a variety of connection rules. With our proposed ad hoc network instantiation, both network construction and simulation times are comparable or shorter than those obtained with other state-of-the-art simulation technologies while still meeting the flexibility demands of explorative network modeling.

ExploitGen: Template-augmented exploit code generation based on CodeBERT

Exploit code is widely used for detecting vulnerabilities and implementing defensive measures. However, automatic generation of exploit code for security assessment is a challenging task. In this paper, we propose a novel template-augmented exploit code generation approach ExploitGen based on CodeBERT. Specifically, we first propose a rule-based Template Parser to generate template-augmented natural language descriptions (NL). Both the raw and template-augmented NL sequences are encoded to context vectors by the respective encoders. For better learning semantic information, ExploitGen incorporates a semantic attention layer, which uses the attention mechanism to extract and calculate each layer’s representational information. In addition, ExploitGen computes the interaction information between the template information and the semantics of the raw NL and designs a residual connection to append the template information into the semantics of the raw NL. Comprehensive experiments on two datasets show the effectiveness of ExploitGen after comparison with six state-of-the-art baselines. Apart from the automatic evaluation, we conduct a human study to evaluate the quality of generated code in terms of syntactic and semantic correctness. The results also confirm the effectiveness of ExploitGen.

MarianCG: a code generation transformer model inspired by machine translation

The idea that computers can build their own programs is extremely significant, and many researchers are working on this challenge. Code generation is described as the process of generating executable code that can be run directly on the computer and fulfills the natural language requirements. It is an intriguing topic that might assist developers to learn a new software technology or programming language, or it could be a simple technique to help in coding through the description of the natural language code developer. In this paper, we present MarianCG, a code generation Transformer model used to tackle the code generation challenge of generating python code from natural language descriptions. Marian neural machine translation (NMT), which is the core model of the Microsoft Translator, is the basis for our NL-to-Code translation engine and is the heart of the teaching model. MarianMT is the teacher language model in our study, and it is one of the most successful machine translation transformers. In our approach, we use a sinusoidal positional embedding technique to represent the position of each token in the text, as well as no layer normalization embedding. Our code generation approach, MarianCG, is based on fine-tuning a machine translation pre-trained language model. This allows us to demonstrate that the pre-trained translation model can also operate and work as a code generation model. The proposed model outperforms recent state-of-the-art models in the problem of code generation when trained on the CoNaLa and DJANGO datasets. MarianCG model scores a BLEU score of 34.43 and an exact match accuracy of 10.2% on the CoNaLa dataset. Also, this model records a BLEU score of 90.41 and an exact match accuracy of 81.83% on the DJANGO dataset. The implementation of MarianCG model and relevant resources are available at https://www.github.com/AhmedSSoliman/MarianCG-NL-to-Code.

AJFCode: An Approach for Full Aspect-Oriented Code Generation from Reusable Aspect Models

AJFCode: An Approach for Full Aspect-Oriented Code Generation from Reusable Aspect Models

A Complex Java Code Generator for ACL2 Based on a Shallow Embedding of ACL2 in Java

This paper describes a code generator that translates ACL2 constructs to corresponding Java constructs, according to a shallow embedding of ACL2 in Java. Starting from purely functional ACL2 code, the generated Java code exhibits imperative and object-oriented features like destructive updates, loops, and overloading. The overall translation from ACL2 to Java is fairly elaborate, consisting of several ACL2-to-ACL2 pre-translation steps, an ACL2-to-Java proper translation step, and several Java-to-Java post-translation steps. Experiments suggest that the generated Java code is not much slower than the ACL2 code. The code generator can also recognize, and translate to Java, ACL2 representations of certain Java constructs, forerunning a code generation approach based on a shallow embedding of Java in ACL2 (i.e. going the other way). This code generator builds upon, and significantly extends, a simple Java code generator for ACL2 based on a deep embedding of ACL2 in Java.

Code Generation Approach Supporting Complex System Modeling based on Graph Pattern Matching

Code generation is an effective way to drive the complex system development in model-based systems engineering. Currently, different code generators are developed for different modeling languages to deal with the development of systems with multi-domain. There are a lack of unified code generation approaches for multi-domain heterogeneous models. In addition, existing methods lack the ability to flexibly query and transform complex model structures to the target code, resulting in poor transformation efficiency. To solve the above problems, this paper proposes a unified approach which supports the code generation of heterogeneous models with complex model structure. First, The KARMA language based on GOPPRR-E meta-modeling approach is used for the unified formalism of models built by different modeling languages. Second, the code generation approach based on graph pattern matching is proposed to realize the query and transformation of complex model structures. Then, the syntax for code generation is integrated into KARMA and a compiler for code generation is developed. Finally, a case of unmanned vehicle system is taken to validate the effectiveness of the proposed approach.

Honeysuckle

In-app privacy notices can help smartphone users make informed privacy decisions. However, they are rarely used in real-world apps, since developers often lack the knowledge, time, and resources to design and implement them well. We present Honeysuckle, a programming tool that helps Android developers build in-app privacy notices using an annotation-based code generation approach facilitated by an IDE plugin, a build system plugin, and a library. We conducted a within-subjects study with 12 Android developers to evaluate Honeysuckle. Each participant was asked to implement privacy notices for two popular open-source apps using the Honeysuckle library as a baseline as well as the annotation-based approach. Our results show that the annotation-based approach helps developers accomplish the task faster with significantly lower cognitive load. Developers preferred the annotation-based approach over the library approach because it was much easier to learn and use and allowed developers to achieve various types of privacy notices using a unified code format, which can enhance code readability and benefit team collaboration.

Code Generation for Productive, Portable, and Scalable Finite Element Simulation in Firedrake

Creating scalable, high-performance PDE-based simulations requires an appropriate combination of models, discretizations, and solvers. The required combination changes with the application and with the available hardware, yet software development time is a severely limited resource for most scientists and engineers. Here we demonstrate that generating simulation code from a high-level Python interface provides an effective mechanism for creating high-performance simulations from very few lines of user code. We demonstrate that moving from one supercomputer to another can require significant algorithmic changes to achieve scalable performance, but that the code generation approach enables these algorithmic changes to be achieved with minimal development effort.

Automatic Code Generation for High-performance Discontinuous Galerkin Methods on Modern Architectures

SIMD vectorization has lately become a key challenge in high-performance computing. However, hand-written explicitly vectorized code often poses a threat to the software’s sustainability. In this publication, we solve this sustainability and performance portability issue by enriching the simulation framework dune-pdelab with a code generation approach. The approach is based on the well-known domain-specific language UFL but combines it with loopy, a more powerful intermediate representation for the computational kernel. Given this flexible tool, we present and implement a new class of vectorization strategies for the assembly of Discontinuous Galerkin methods on hexahedral meshes exploiting the finite element’s tensor product structure. The performance-optimal variant from this class is chosen by the code generator through an auto-tuning approach. The implementation is done within the open source PDE software framework Dune and the discretization module dune-pdelab. The strength of the proposed approach is illustrated with performance measurements for DG schemes for a scalar diffusion reaction equation and the Stokes equation. In our measurements, we utilize both the AVX2 and the AVX512 instruction set, achieving 30% to 40% of the machine’s theoretical peak performance for one matrix-free application of the operator.

Code generation approaches for parallel geometric multigrid solvers

Abstract Software development for applications in computational science and engineering has become complex in recent years. This is mainly due to the increasing parallelism and heterogeneity in modern computer architectures and to the more realistic physical and mathematical models that have to be processed. One idea to address this issue is to use code generation techniques. In contrast to manual implementations in a general-purpose computing language, they allow to integrate automatic code transforms to produce efficient code for different models and platforms. As an example the numerical solution of an elliptic partial differential equation via generated geometric multigrid solvers is considered. We present three code generation approaches for it and discuss their advantages and disadvantages with respect to performance, portability, and productivity.

Semantic driven code generation for networking testbed experimentation

ABSTRACTThis paper investigates how semantic technologies can be used for automatic code generation with a goal to provide easier to use and more efficient working environment to users of large complex infrastructures such as networking testbeds. A new Semantic Driven Code Generation (SecGENE) approach is proposed in the light of a testbed-driven experimentation. The target use-case scenario focuses on heterogeneous telecommunication systems and optimal spectrum usage. The SecGENE approach is examined on the example of semantic coordination process in an environment containing coordinated as well as uncoordinated WiFi networks.Abbreviations: CLOnt – Command Line ontology; ExFOnt – Experiment Flow Ontology; MSAO – Meta System Architecture ontology; NVF – Network Virtualization Function; OEDL – OMF Experiment Description Language; OMN- Open-Multinet ontology; RDFO – A standard W3C Resource Description Framework ontology; SOEDL– Semantic OEDL