Automated Software Engineering Research Articles

StructCoder: Structure-Aware Transformer for Code Generation

There has been a recent surge of interest in automating software engineering tasks using deep learning. This article addresses the problem of code generation, in which the goal is to generate target code given source code in a different language or a natural language description. Most state-of-the-art deep learning models for code generation use training strategies primarily designed for natural language. However, understanding and generating code requires a more rigorous comprehension of the code syntax and semantics. With this motivation, we develop an encoder-decoder Transformer model in which both the encoder and decoder are explicitly trained to recognize the syntax and dataflow in the source and target codes, respectively. We not only make the encoder structure aware by leveraging the source code’s syntax tree and dataflow graph, but we also support the decoder in preserving the syntax and dataflow of the target code by introducing two novel auxiliary tasks: Abstract Syntax Tree (AST) path prediction and dataflow prediction. To the best of our knowledge, this is the first work to introduce a structure-aware Transformer decoder that models both syntax and dataflow to enhance the quality of generated code. The proposed StructCoder model achieves state-of-the-art performance on code translation and text-to-code generation tasks in the CodeXGLUE benchmark and improves over baselines of similar size on the APPS code generation benchmark. Our code is publicly available at https://github.com/reddy-lab-code-research/StructCoder/ .

Summary of the 5th International Workshop on Automated andverifiable Software sYstem DEvelopment (ASYDE)co-located with the 38th IEEE/ACM ASE 2023

Over the past three decades, automation in software development has gone mainstream. Software development teams strive to automate as much of the software development activities as possible, spanning requirements specification, system modeling, code generation, testing, deployment, verification, as well as release phases, project status reporting and system maintenance. Automation helps to reduce development time and cost, as well as to concentrate knowledge by bringing quality into every step of the development process. The Workshop on Automated and verifiable Software sYstem DEvelopment (ASYDE) provided a forum to share and discuss innovative contributions to research and practice related to novel software engineering approaches to automated and verifiable development of software systems. The 5th edition took place on September 11th, 2023, in Kirchberg, Luxembourg. Notably, this marked the inaugural co-location of ASYDE with the IEEE/ACM International Conference on Automated Software Engineering (ASE).

USE CASE TOOLS FOR IDENTIFICATION SOFTWARE REQUIREMENTS: STATE OF ART

The requirement analysis step of the Software Engineering Life Cycle (SELC) is the most crucial in order to ensure excellent quality. Besides, requirements specifications must be compared to the consistency, completeness, and correctness (3C's). Thus, dealing with Natural Language (NL) requirements makes this more challenging than usual., The problems that appear at this stage, if they are not detected or proceed, will lead to errors in the system and sometimes to the failure of the entire system; in addition to the effort required and the cost of maintenance will be large. Accordingly,to overcome these problems, differenttools have been developed and implemented to analyse the requirements through Natural Language Processing (NLP). These tools facilitate the process of eliciting requirements through Unified Modelling Language (UML) diagrams. Additionally, creating UML diagrams from NL specifications is a very difficult process. In particular, few efforts have been made in the work of extracting NL specifications, so these diagrams facilitate understanding and analysis of requirements and reduce the time of completion very significantly compared to the manual method. Using automated software engineering tools presents high-quality software that can be produced to help software developers and engineers. The main aim of this research is to concentrate onanalysing requirements based on the Use Case Diagram, which is a more popular one in UML diagram. Thus, this research will support other researchers in understanding and specifying the useful tools and mechanisms that can help them to analyse the requirements through Natural Language Processing (NLP) based UML diagrams

CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models

Pre-trained programming language (PL) models (such as CodeT5, CodeBERT, GraphCodeBERT, etc.,) have the potential to automate software engineering tasks involving code understanding and code generation. However, these models operate in the natural channel of code, i.e., primarily concerned with the human understanding of code. They are not robust to changes in the input and thus, are potentially susceptible to adversarial attacks in the natural channel. We propose, Code Attack, a simple yet effective black-box attack model that uses code structure to generate effective, efficient, and imperceptible adversarial code samples and demonstrates the vulnerabilities of the state-of-the-art PL models to code-specific adversarial attacks. We evaluate the transferability of CodeAttack on several code-code (translation and repair) and code-NL (summarization) tasks across different programming languages. Code Attack outperforms state-of-the-art adversarial NLP attack models to achieve the best overall drop in performance while being more efficient, imperceptible, consistent, and fluent. The code can be found at https://github.com/reddy-lab-code-research/CodeAttack.

An Interview with John Grundy - 2023 SIGSOFT Awardee

John Grundy received the 2023 SIGSOFT Distinguished Service Award for continued outstanding service to the software engineering community, including the repeated successful organization of flagship conferences, and leadership in the Australasian research and education community. He received a Ph.D. in computer science from the University of Auckland (New Zealand), and he is currently Australian Laureate Fellow and Professor of Software Engineering at Monash University. Beyond this, John Grundy is also Fellow of Automated Software Engineering, Fellow of Engineers Australia, Certified Professional Engineer, Engineering Executive, Senior Member of the ACM and Senior Member of the IEEE.

An exploratory study of software engineering in heavy-duty mobile machine automation

As the amount and complexity of software for automating heavy-duty mobile machinery is increasing, software engineering in this domain is becoming more important. To characterize the industry’s current state of software engineering and its issues to guide future research, we performed an empirical exploratory study. We interviewed 16 software engineering professionals from 13 different companies conducting business in heavy-duty mobile machines and their automation. The interviews were analyzed qualitatively, and quantification of the analysis results is presented. We first create an overview of software engineering in the heavy-duty mobile machinery industry. We then identify problem areas affecting software development and discuss some of the possible solutions found in literature. Our findings indicate that the major problem areas faced in the industry that require more research are its digital transformation, autonomous machine functional safety, low availability of workforce for developing software for robotic mobile machines and the lack of established software standards.

A classification approach for software requirements towards maintainable security

As trends continue to move toward the introduction of intelligent methods to automate software engineering processes, security requirements classification is rapidly turning into a highly potent field for the software engineering community. There are several models for classifying security requirements proposed in the literature. However, their adoption and use is constrained by the absence of substantial datasets to allow for the replication and generalization of studies, using more advanced machine-learning techniques. Furthermore, most researchers in this area, consider Maintainability as purely a non-functional requirement with no relation to security. This has been identified to be a major source of security concerns. The main objective of this study is to propose a software requirements classification approach that considers maintainability as a security requirement. This seeks to ensure that maintenance efforts don't lead to new software vulnerabilities that were previously not present during deployment. A mixed research methodology is adopted as qualitative data is collected from students’ project documentation, labelled, and transformed into quantitative form during analysis. As a culmination of this process, a validated original publicly accessible, labelled software requirements dataset of student software project requirements (DOSSPRE) is obtained and presented to support the approach. It contains 1317 software requirements, including security requirements, functional requirements, and other non-functional requirements. Two versions of the dataset are presented: one for binary classification of security requirements vs. non-security requirements and the other for multi-class classification tasks with various more granular security requirements vs. non-security requirements. In both instances, well-known machine learning algorithms are used to verify the dataset. Support Vector Machine (SVM) and Logistic Regression were the top performers in multi-class classification with an average Accuracy of 86% in both cases. Multinomial Nave Bayes topped the other machine learning techniques in binary classification with 91% Precision, 69% Recall, 78% F1-Score, and Accuracy of 86%. The dataset is accessible on this link https://data.mendeley.com/datasets/23xtbvk6yp/1

A Paradigm Shift in Automating Software Engineering Tasks: Bots

Increasing developer productivity and improving software quality are enduring challenges in software engineering. Addressing these challenges has influenced research toward improved automation and formalisms to realize software development tasks, along the way seeking for disruptive ways to support software developers. The past 55 years of software engineering has seen a number of paradigm shifts, fundamental changes in our underlying assumptions which led to making some progress in developing better quality software and disruptive improvements. For example, agile and lean software development approaches challenged our assumptions about the way software development tasks are orchestrated and communicated. Agile and lean software development philosophies advocate elimination of waste by emphasizing value-added tasks and encouraging improved understanding of what tasks stay in inventory.1 Embracing agile and lean software development approaches with discipline led software engineers to prioritize software engineering activities with a bias towards visible delivery, most often leading to improved software quality and developer well being when done right.

A survey of deep learning based text-to-SQL generation

Text-to-SQL is an important application of automation software engineering. It is also a research hotspot in the field of semantic parsing. The text-to-SQL task aims to automatically generate the SQL statement according to the natural language description. It allows nonprofessionals to access the database without understanding SQL syntax. With the development of large-scale text-to-SQL datasets and artificial intelligence technologies, the text-to-SQL task is also making great progress. Compared with the traditional text-to-SQL generation, the deep learning-based text-to-SQL has the advantages of high accuracy, flexibility, and iterative learning. In recent years, several studies have focused on SQL generation based on deep learning. This research summarizes existing works from the aspects of text-to-SQL scenarios, datasets, model structures, and evaluation methods.

Ground-Truth Deficiencies in Software Engineering: When Codifying the Past Can Be Counterproductive

In software engineering, the objective function of human decision makers might be influenced by many factors. Relying on historical data as the ground truth may give rise to systems that automate software engineering decisions by mimicking past suboptimal behavior. We describe the problem and offer some strategies.

ASE4Games 2021 Workshop Summary

The first edition of the workshop on Automated Software Engineering For Games (ASE4Games 2021) was held virtually on November 14th, 2021, co-located with the 36v IEEE/ACM International Conference on Automated Software Engineering (ASE 2021). Five papers from all over the world were submitted, two full-papers and two short-papers were accepted. The program also featured a keynote by Mathieu Nayrolles, researcher at Ubisoft, entitled Automated Software Engineering for AAA Games.

Report on the 2nd Workshop on Human Centric Software Engineering & Cyber Security (HCSE&CS 2021)

As the creators, designers, coders, testers, users, and occasional abusers of all software systems-including cyber security systems - humans should be at the centre of all design and development efforts. Despite this, most software engineering and cyber security research and practices tend to be function, data, or process oriented. In contrast, human-centric software engineering focuses on the human-centric issues critical to successful software systems' engineering. The aim of the International Workshop on Human Centric Software Engineering & Cyber Security (HCSE&CS) was to provide a venue for sharing research ideas and outcomes on enhanced theory, models, tools, and capability for next-generation human-centric software engineering and cyber security. The Second HCSE&CS Workshop was held on 15 November 2021 in conjunction with ASE 2021, the 36th IEEE/ACM International Conference on Automated Software Engineering. It was originally intended to be held in Melbourne, Australia but was instead held virtually due to the COVID-19 pandemic. This post-workshop report provides an overview of the aims and motivation of the workshop as well as a summary of the presentations and discussions that took place during the workshop.

One- and Two-Phase Software Requirement Classification Using Ensemble Deep Learning.

Recently, deep learning (DL) has been utilized successfully in different fields, achieving remarkable results. Thus, there is a noticeable focus on DL approaches to automate software engineering (SE) tasks such as maintenance, requirement extraction, and classification. An advanced utilization of DL is the ensemble approach, which aims to reduce error rates and learning time and improve performance. In this research, three ensemble approaches were applied: accuracy as a weight ensemble, mean ensemble, and accuracy per class as a weight ensemble with a combination of four different DL models—long short-term memory (LSTM), bidirectional long short-term memory (BiLSTM), a gated recurrent unit (GRU), and a convolutional neural network (CNN)—in order to classify the software requirement (SR) specification, the binary classification of SRs into functional requirement (FRs) or non-functional requirements (NFRs), and the multi-label classification of both FRs and NFRs into further experimental classes. The models were trained and tested on the PROMISE dataset. A one-phase classification system was developed to classify SRs directly into one of the 17 multi-classes of FRs and NFRs. In addition, a two-phase classification system was developed to classify SRs first into FRs or NFRs and to pass the output to the second phase of multi-class classification to 17 classes. The experimental results demonstrated that the proposed classification systems can lead to a competitive classification performance compared to the state-of-the-art methods. The two-phase classification system proved its robustness against the one-phase classification system, as it obtained a 95.7% accuracy in the binary classification phase and a 93.4% accuracy in the second phase of NFR and FR multi-class classification.

Recent Advances in Intelligent Source Code Generation: A Survey on Natural Language Based Studies.

Source Code Generation (SCG) is a prevalent research field in the automation software engineering sector that maps specific descriptions to various sorts of executable code. Along with the numerous intensive studies, diverse SCG types that integrate different scenarios and contexts continue to emerge. As the ultimate purpose of SCG, Natural Language-based Source Code Generation (NLSCG) is growing into an attractive and challenging field, as the expressibility and extremely high abstraction of the input end. The booming large-scale dataset generated by open-source code repositories and Q&A resources, the innovation of machine learning algorithms, and the development of computing capacity make the NLSCG field promising and give more opportunities to the model implementation and perfection. Besides, we observed an increasing interest stream of NLSCG relevant studies recently, presenting quite various technical schools. However, many studies are bound to specific datasets with customization issues, producing occasional successful solutions with tentative technical methods. There is no systematic study to explore and promote the further development of this field. We carried out a systematic literature survey and tool research to find potential improvement directions. First, we position the role of NLSCG among various SCG genres, and specify the generation context empirically via software development domain knowledge and programming experiences; second, we explore the selected studies collected by a thoughtfully designed snowballing process, clarify the NLSCG field and understand the NLSCG problem, which lays a foundation for our subsequent investigation. Third, we model the research problems from technical focus and adaptive challenges, and elaborate insights gained from the NLSCG research backlog. Finally, we summarize the latest technology landscape over the transformation model and depict the critical tactics used in the essential components and their correlations. This research addresses the challenges of bridging the gap between natural language processing and source code analytics, outlines different dimensions of NLSCG research concerns and technical utilities, and shows a bounded technical context of NLSCG to facilitate more future studies in this promising area.

Automatic Program Repair

Following along with the theme of this issue of IEEE Software, this column reports on papers about automatic program repair (APR) from the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE20), the 35th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW20), and the 13th IEEE International Conference on Software Testing, Validation and Verification (ICST20). Feedback or suggestions are welcome. In addition, if you try or adopt any of the practices included in the column, please send us and the authors a note about your experiences.

Reactive Auto-Completion of Modeling Activities

Assisting and automating software engineering tasks is a state-of-the-art way to support stakeholders of development projects. A common assistance function of IDEs is the auto-completion of source code. Assistance functions, such as auto-completion, are almost entirely missing in modeling tools though auto-completion in general gains continuously more importance in software development. We analyze a user’s performed editing operations in order to anticipate modeling activities and to recommend appropriate auto-completions for them. Editing operations are captured as events and modeling activities are defined as complex event patterns, facilitating the matching by complex-event-processing. The approach provides adapted auto-completions reactively upon each editing operation of the user. We implemented the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RapMOD</i> prototype as add-in for the modeling tool Sparx Enterprise Architect™ . A controlled user experiment with 37 participants performing modeling tasks demonstrated the approach’s potential to reduce modeling effort significantly. Users having auto-completions available for a modeling scenario performed the task 27 percent faster, needed to perform 56 percent less actions, and perceived the task 29 percent less difficult.

An Empirical Framework to Investigate the Impact of Bug Fixing on Internal Quality Attributes

Software testing is the process of fixing bugs by changing the design of the software or simple logic of the software. Researchers have proposed many tools and methods using machine learning techniques to assist practitioners in decision making and automating software engineering tasks. These tools help to find the faulty classes at the starting phase of the software development life cycle. After finding faulty classes using these tools, testers used different techniques to find and fix these faults. The early identification of the faults or bugs fixing process helps to improves the quality of software and reduces the cost required to fix these faults or bugs. The primary objective of this work is to understand whether faults present in code elements are indicators of problems in the design of the software or not. This work investigates the impact of bug fixing operation on four popular internal quality attributes such as complexity, cohesion, inheritance, and coupling. The above investigation has been validated using thirteen different projects. Furthermore, we have also investigated the possibility of prediction models for predicting changes in internal quality attributes. These prediction models are trained using five different classifiers on balance data as well as original data and validated using fivefold cross-validation. The experimental results show that the predictive power of models using LSSVM with the polynomial kernel is better as compared to other techniques. The experimental results also show that the bugs are present in the class having at least one critical attribute in more than 80% of cases. Furthermore, the consistent value of AUC reveals that the prediction of changes in the internal quality attribute is possible using source code metrics.

Split Attention Pointer Network for Source Code Language Modeling

There is a growing interest in leveraging Deep Learning (DL) for automating Software Engineering tasks such as program completion. In this paper, we leverage Recurrent Neural Networks (RNNs) for Abstract Syntax Tree (AST)-based code completion. Our approach converts source code into AST nodes and a language model predicts the type and value attributes of next tokens. Our work demonstrates that the attention augmented RNN-based language models are able to understand local context and copy from recent past tokens which have never appeared in the training data set. We observed a drop of performances of both type and value predictions when using a traditional pointer network architecture for out-of-vocabulary (OoV) copying and context understanding, which we call multi-task conflict. To address this challenge, we have devised a new structure of self-attention called Split Attention, where two separate dot-product layers are applied to different parts of the history cache. Based on this structure, we propose a new network called Split Attention Pointer Network (SAPN), which is efficient and flexible in both learning local context and copying OoV tokens from history. The empirical results suggest that our model is superior in syntax-aware generation and OoV token prediction by demonstrating attention behavior similar to human programmers. The results also indicate that our model out performs previous state-of-the-art approaches by more than 6% on widely recognized program completion benchmarks.

Understanding Software Security from Design to Deployment

Analyzing, implementing and maintaining security requirements of software-intensive systems and achieving truly secure software requires planning for security from ground up, and continuously assuring that security is maintained across the software's lifecycle and even after deployment when software evolves. Given the increasing complexity of software systems, new application domains, dynamic and often critical operating conditions, the distributed nature of many software systems, and fast moving markets which put pressure on software vendors, building secure systems from ground up becomes even more challenging. Security-related issues have previously been targeted in software engineering sub-communities and venues. In the second edition of the International Workshop on Security from Design to Deployment (SEAD) at the International Conference on Automated Software Engineering (ASE) 2020, we aimed to bring the research and practitioner communities of requirements engineers, security experts, architects, developers, and testers together to identify foundations, and challenges, and to formulate solutions related to automating the analysis, design, implementation, testing, and maintenance of secure software systems.

The Java Pathfinder Workshop 2019

Java Path nder (JPF) was originally developed as an explicit- state software model checker, and subsequently evolved into an extensible Java bytecode analysis framework that has been suc- cessfully used to implement techniques such as symbolic and con- colic execution, compositional veri cation, parallel execution, in- cremental program analysis, and many more. To share recent research progress with JPF and related tools among the community, we have organized the annual JPF work- shop with the Automated Software Engineering Conference (ASE) 2019, held in San Diego, California, USA. We invited submissions about on-going and existing research, experience, and position papers on topics (1) related to JPF, its extensions and applica- tions in various domains; and (2) Java/Android program analysis in general. This paper gives an overview of all presentations and papers of the workshop, as well the results of the discussions.