Software Engineering Tasks Research Articles

Effective Approach for Fine-Tuning Pre-Trained Models for the Extraction of Texts From Source Codes

This study introduces SR-Text, a robust approach leveraging pre-trained models like BERT and T5 for enhanced text extraction from source codes. Addressing the limitations of traditional manual summarization, our methodology focuses on fine-tuning these models to better understand and generate contextual summaries, thus overcoming challenges such as long-term dependency and dataset quality issues. We conduct a detailed analysis of programming language syntax and semantics to develop syntax-aware text retrieval techniques, significantly boosting the accuracy and relevance of the texts extracted. The paper also explores a hybrid approach that integrates statistical machine learning with rule-based methods, enhancing the robustness and adaptability of our text extraction processes across diverse coding styles and languages. Empirical results from a meticulously curated dataset demonstrate marked improvements in performance metrics: precision increased by 15%, recall by 20%, and an F1 score enhancement of 18%. These improvements underscore the effectiveness of using advanced machine learning models in software engineering tasks. This research not only paves the way for future work in multilingual code summarization but also discusses broader implications for automated software analysis tools, proposing directions for future research to further refine and expand this methodology.

Comparing the Pretrained Models of Source Code by Re-pretraining Under a Unified Setup.

Recent years have seen the successful application of large pretrained models of source code (CodePTMs) to code representation learning, which have taken the field of software engineering (SE) from task-specific solutions to task-agnostic generic models. By the remarkable results, CodePTMs are seen as a promising direction in both academia and industry. While a number of CodePTMs have been proposed, they are often not directly comparable because they differ in experimental setups such as pretraining dataset, model size, evaluation tasks, and datasets. In this article, we first review the experimental setup used in previous work and propose a standardized setup to facilitate fair comparisons among CodePTMs to explore the impacts of their pretraining tasks. Then, under the standardized setup, we re-pretrain CodePTMs using the same model architecture, input modalities, and pretraining tasks, as they declared and fine-tune each model on each evaluation SE task for evaluating. Finally, we present the experimental results and make a comprehensive discussion on the relative strength and weakness of different pretraining tasks with respect to each SE task. We hope our view can inspire and advance the future study of more powerful CodePTMs.

On the impact of multiple source code representations on software engineering tasks — An empirical study

Efficiently representing source code is crucial for various software engineering tasks such as code classification and clone detection. Existing approaches primarily use Abstract Syntax Tree (AST), and only a few focus on semantic graphs such as Control Flow Graph (CFG) and Program Dependency Graph (PDG), which contain information about source code that AST does not. Even though some works tried to utilize multiple representations, they do not provide any insights about the costs and benefits of using multiple representations. The primary goal of this paper is to discuss the implications of utilizing multiple source code representations, specifically AST, CFG, and PDG. We modify an AST path-based approach to accept multiple representations as input to an attention-based model. We do this to measure the impact of additional representations (such as CFG and PDG) over AST. We evaluate our approach on three tasks: Method Naming, Program Classification, and Clone Detection. Our approach increases the performance on these tasks by 11% (F1), 15.7% (Accuracy), and 9.3% (F1), respectively, over the baseline. In addition to the effect on performance, we discuss timing overheads incurred with multiple representations. We envision that this work can provide a base for researchers to explore and experiment with a variety of source code representations for software engineering tasks.

Exploring LLM-based Automated Repairing of Ansible Script in Edge-Cloud Infrastructures

Edge-Cloud system requires massive infrastructures located in closer to the user to minimize latencies in handling Big data. Ansible is one of the most popular Infrastructure as Code (IaC) tools crucial for deploying these infrastructures of the Edge-cloud system. However, Ansible also consists of code, and its code quality is critical in ensuring the delivery of high-quality services within the Edge-Cloud system. On the other hand, the Large Langue Model (LLM) has performed remarkably on various Software Engineering (SE) tasks in recent years. One such task is Automated Program Repairing (APR), where LLMs assist developers in proposing code fixes for identified bugs. Nevertheless, prior studies in LLM-based APR have predominantly concentrated on widely used programming languages (PL), such as Java and C, and there has yet to be an attempt to apply it to Ansible. Hence, we explore the applicability of LLM-based APR on Ansible. We assess LLMs’ performance (ChatGPT and Bard) on 58 Ansible script revision cases from Open Source Software (OSS). Our findings reveal promising prospects, with LLMs generating helpful responses in 70% of the sampled cases. Nonetheless, further research is necessary to harness this approach’s potential fully.

On the effectiveness of graph data augmentation for source code learning

The methodology that employs deep learning to handle software engineering tasks, such as bug detection, is commonly referred to as source code learning. Given the inherent graph nature of source code, graph learning, bolstered by graph neural networks (GNNs), has seen an increasing adoption in the domain of source code learning. Similar to other contexts within deep learning, source code learning also relies on extensive high-quality training data, and the scarcity of such data has become a primary impediment that leads to performance bottlenecks. In practice, data augmentation is often used as a countermeasure to mitigate this issue, by synthesizing additional training data based on existing ones. However, most existing practice of data augmentation in source code learning is limited to simple program transformation methods, such as code refactoring, thus not sufficiently effective. In this work, in light of the graph nature of source code, we propose to apply the data augmentation methods used for graph-structured data in graph learning to the tasks of source code learning, and we conduct a comprehensive empirical study to evaluate whether such new data augmentation approaches bring better effectiveness, in terms of producing more accurate and robust models. Specifically, we evaluate four critical software engineering tasks and seven neural network architectures to assess the effectiveness of five data augmentation methods. Experimental results identify that, compared to the data augmentation-free training approach, the Manifold-Mixup method can significantly improve both the accuracy and robustness of the trained models of source code learning, for up to 1.60% and 4.09%, respectively.

Summary of the 1st Interpretability and Robustness in Neural Software Engineering (InteNSE 2023)

InteNSE is an interdisciplinary workshop for research at the intersection of Machine Learning (ML) and Software Engineering (SE) and would be a pioneer in emphasizing the implicit properties of neural software engineering and analysis. Due to recent computational advancements, ML has become an inseparable part of the SE research community. ML can indeed improve and revolutionize many SE tasks. However, most research in the AI and SE communities consider ML as a closed box, i.e., only considering the final performance of the developed models as an evaluation metric. Ignoring the implicit properties of neural models, such as interpretability and robustness, one cannot validate the model's actual performance, generalizability, and whether it is learning what it is supposed to do. Specifically, in the domain of SE, where the result of ML4SE tools is code synthesis, bug finding, or repair, interpretability and robustness are crucial to ensure the reliability of the products.

A survey on machine learning techniques applied to source code

The advancements in machine learning techniques have encouraged researchers to apply these techniques to a myriad of software engineering tasks that use source code analysis, such as testing and vulnerability detection. Such a large number of studies hinders the community from understanding the current research landscape. This paper aims to summarize the current knowledge in applied machine learning for source code analysis. We review studies belonging to twelve categories of software engineering tasks and corresponding machine learning techniques, tools, and datasets that have been applied to solve them. To do so, we conducted an extensive literature search and identified 494 studies. We summarize our observations and findings with the help of the identified studies. Our findings suggest that the use of machine learning techniques for source code analysis tasks is consistently increasing. We synthesize commonly used steps and the overall workflow for each task and summarize machine learning techniques employed. We identify a comprehensive list of available datasets and tools useable in this context. Finally, the paper discusses perceived challenges in this area, including the availability of standard datasets, reproducibility and replicability, and hardware resources.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Enhancing source code retrieval with joint Bi-LSTM-GNN architecture: A comparative study with ChatGPT-LLM

Retrieving relevant source code from large repositories is a significant and ongoing challenge in the field of software engineering, primarily due to the vast and ever-expanding amount of available code. Existing deep learning methods, although effective to some extent, exhibit limitations in capturing the intricate and complex structural information embedded within source code, which hinders their ability to provide highly accurate retrieval results. This study endeavors to tackle this prominent issue by introducing a novel and innovative approach known as the Joint Bi-directional LSTM and Graph Neural Networks (JBLG) model for source code retrieval. The central aim is to harness the combined strengths and capabilities of Bi-directional Long Short-Term Memory (LSTM) networks and Graph Neural Networks (GNNs) to significantly enhance the model’s capacity to capture and interpret the complex structural characteristics intrinsic to source code. The proposed JBLG model employs a unique fusion of Bi-directional LSTM, which excels in capturing sequential and temporal dependencies within code, and GNN, which is adept at modeling the intricate graph structure of the code. By leveraging this hybrid architecture, the model aims to provide a comprehensive and highly effective solution for source code retrieval tasks. To assess the efficacy of the JBLG model, extensive experiments are conducted, and the model’s performance is evaluated against well-established benchmarks, including LSTM, GNN, and ChatGPT, using two diverse datasets: CodeSearchNet and CosBench datasets. These evaluations span multiple programming languages, ensuring a comprehensive and robust assessment of the model’s capabilities. The experimental results indicate that the JBLG model consistently outperforms its counterparts, including Bi-LSTM, GNN, ChatGPT, and DGMS, across various evaluation metrics. the JBLG model showcases an exceptional ability to handle and extract the intricate structural information inherent in source code, resulting in significantly enhanced retrieval accuracy. The JBLG model emerges as a highly promising solution for real-world source code retrieval applications, with the potential to revolutionize the field. The success of this model underscores the importance of combining deep learning techniques like Bi-directional LSTM and GNNs for tackling complex software engineering challenges. Furthermore, future research directions could involve exploring advanced techniques such as attention mechanisms and extending the model’s applicability to other software engineering tasks like code summarization and code completion. The findings of this study are expected to have a lasting impact on the advancement of source code retrieval methodologies.

API Entity and Relation Joint Extraction from Text via Dynamic Prompt-tuned Language Model

Extraction of Application Programming Interfaces (APIs) and their semantic relations from unstructured text (e.g., Stack Overflow) is a fundamental work for software engineering tasks (e.g., API recommendation). However, existing approaches are rule based and sequence labeling based. They must manually enumerate the rules or label data for a wide range of sentence patterns, which involves a significant amount of labor overhead and is exacerbated by morphological and common-word ambiguity. In contrast to matching or labeling API entities and relations, this article formulates heterogeneous API extraction and API relation extraction task as a sequence-to-sequence generation task and proposes the API Entity-Relation Joint Extraction framework (AERJE), an API entity-relation joint extraction model based on the large pre-trained language model. After training on a small number of ambiguous but correctly labeled data, AERJE builds a multi-task architecture that extracts API entities and relations from unstructured text using dynamic prompts. We systematically evaluate AERJE on a set of long and ambiguous sentences from Stack Overflow. The experimental results show that AERJE achieves high accuracy and discrimination ability in API entity-relation joint extraction, even with zero or few-shot fine-tuning.

The Source Code Comment Generation Based on Deep Reinforcement Learning and Hierarchical Attention

Code summarization provides the main aim described in natural language of the given function, it can benefit many tasks in software engineering. Due to the special grammar and syntax structure of programming languages and various shortcomings of different deep neural networks, the accuracy of existing code summarization approaches is not good enough. This work proposes to adopt the hierarchical attention mechanism to enable the code summarization framework to translate three representations of source code to the hidden spce and then it injects them into a deep reinforcement learning model to enhance the performance of code summarization. We conduct a few of experiments, and the results of which prove that the proposed approaches can obtain better accuracy compared with the baseline approaches.

The Source Code Comment Generation based on Abstract Syntax Tree

Code summarization provides the main aim described in natural language of the given function; it can benefit many tasks in software engineering. Due to the special grammar and syntax structure of programming languages and various shortcomings of different deep neural networks, the accuracy of existing code summarization approaches is not good enough. We proposes to use abstract syntax trees for source code summarization .Our solution is inspired by recent advances in neural machine translation, as well as an approach called SBT by Hu et al. We evaluate our approach using the automated metric BLEU and compare it to other relevant models.

Do Pretrained Language Models Indeed Understand Software Engineering Tasks?

Do Pretrained Language Models Indeed Understand Software Engineering Tasks?

Semantic-Enriched Code Knowledge Graph to Reveal Unknowns in Smart Contract Code Reuse

Programmers who work with smart contract development often encounter challenges in reusing code from repositories. This is due to the presence of two unknowns that can lead to non-functional and functional failures. These unknowns are implicit collaborations between functions and subtle differences among similar functions. Current code mining methods can extract syntax and semantic knowledge (known knowledge), but they cannot uncover these unknowns due to a significant gap between the known and the unknown. To address this issue, we formulate knowledge acquisition as a knowledge deduction task and propose an analytic flow that uses the function clone as a bridge to gradually deduce the known knowledge into the problem-solving knowledge that can reveal the unknowns. This flow comprises five methods: clone detection, co-occurrence probability calculation, function usage frequency accumulation, description propagation, and control flow graph annotation. This provides a systematic and coherent approach to knowledge deduction. We then structure all of the knowledge into a semantic-enriched code Knowledge Graph (KG) and integrate this KG into two software engineering tasks: code recommendation and crowd-scaled coding practice checking. As a proof of concept, we apply our approach to 5,140 smart contract files available on Etherscan.io and confirm high accuracy of our KG construction steps. In our experiments, our code KG effectively improved code recommendation accuracy by 6% to 45%, increased diversity by 61% to 102%, and enhanced NDCG by 1% to 21%. Furthermore, compared to traditional analysis tools and the debugging-with-the-crowd method, our KG improved time efficiency by 30 to 380 seconds, vulnerability determination accuracy by 20% to 33%, and vulnerability fixing accuracy by 24% to 40% for novice developers who identified and fixed vulnerable smart contract functions.

A Comparative Analysis of Regression Models for Software Effort Estimation

Software Effort Estimation is the utmost task in software engineering and project management. This is important to estimate cost properly and the number of people required for a project to be developed. Many techniques have been used to estimate cost, time, schedule and required manpower for software development industries. Nowadays software is developed in a more complex way and its success depends on efficient estimation techniques. In this research, we have compared five regression algorithms on different projects to estimate software effort. The main advantage of these models is they can be used in the early stages of the software life cycle and that can be helpful to project managers to conduct effort estimation efficiently before starting the project. It avoids project overestimation and late delivery. Software size, productivity, complexity and requirement stability are the input vectors for these regression models. The estimated efforts have been calculated using Ridge Regression, Lasso Regression, Elastic Net, Random Forest and Support Vector Regression. We have compared unitedly these models for the first time as software effort estimators. R-squared Score, Mean Squared Error (MSE) and Mean Absolute Error (MAE) are calculated for these regression models. Ridge, Lasso and Elastic Net show comparatively better results among others.

Application of Crowdsourcing in User Experience collection – a case study of Malayalam mobile applications

PurposeThis paper aims to prove the following hypothesis Problem Statement: HYPOTHESIS (1) User Experience collection of mobile applications can be done using the Crowdsourcing mechanism; (2) User Experience collection of mobile applications are influenced by the mindset of Crowdmembers, culture/ethnicity/social background, ease of interface use and rewards, among other factors.Design/methodology/approachThe authors of this paper, did a literature review first to find if Crowdsourcing was applicable and a used method to solve problems in Software Engineering. This helped us to narrow down the application of Crowdsourcing to the Requirements Engineering-Usability (User Experience) collection. User experience collection of two Malayalam language-based mobile applications, AarogyaSetu and BevQ was done as the next step. Incorporating findings from Study I, another study using AarogyaSetu and Manglish was launched as Study II. The results from both cases were consolidated and analyzed. Significant concerns relating to expectations of Crowd members with User Experience collection were unraveled and the purpose of Study was accomplished.Findings(1) Crowdsourcing is and can be used in Software Engineering activities. (2) Crowd members have expectations (motivating factors) of User Interface and other elements that enable them to be an effective contributor. (3) An individual’s environment and mindset (character) are influential in him becoming a contributor in Crowdsourcing. (4) Culture and social practices of a region strongly affects the crowd-participating decision of an individual.Originality/valueThis is purely self-done work. The value of this research work is two-fold. Crowdsourcing is endorsed significant in Software Engineering tasks, especially in User Experience collection of mobile applications. Two, the Crowd service requesters can be careful about designing the questionnaire for Crowdsourcing. They have to be aware and prepared to meet the expectations of the Crowd. This can ensure the active participation of potential contributors. Future researchers can use the results of this work to base their research on similar purposes.

CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models

Pre-trained programming language (PL) models (such as CodeT5, CodeBERT, GraphCodeBERT, etc.,) have the potential to automate software engineering tasks involving code understanding and code generation. However, these models operate in the natural channel of code, i.e., primarily concerned with the human understanding of code. They are not robust to changes in the input and thus, are potentially susceptible to adversarial attacks in the natural channel. We propose, Code Attack, a simple yet effective black-box attack model that uses code structure to generate effective, efficient, and imperceptible adversarial code samples and demonstrates the vulnerabilities of the state-of-the-art PL models to code-specific adversarial attacks. We evaluate the transferability of CodeAttack on several code-code (translation and repair) and code-NL (summarization) tasks across different programming languages. Code Attack outperforms state-of-the-art adversarial NLP attack models to achieve the best overall drop in performance while being more efficient, imperceptible, consistent, and fluent. The code can be found at https://github.com/reddy-lab-code-research/CodeAttack.

MetaTPTrans: A Meta Learning Approach for Multilingual Code Representation Learning

Representation learning of source code is essential for applying machine learning to software engineering tasks. Learning code representation from a multilingual source code dataset has been shown to be more effective than learning from single-language datasets separately, since more training data from multilingual dataset improves the model's ability to extract language-agnostic information from source code. However, existing multilingual training overlooks the language-specific information which is crucial for modeling source code across different programming languages, while only focusing on learning a unified model with shared parameters among different languages for language-agnostic information modeling. To address this problem, we propose MetaTPTrans, a meta learning approach for multilingual code representation learning. MetaTPTrans generates different parameters for the feature extractor according to the specific programming language type of the input code snippet, enabling the model to learn both language-agnostic and language-specific information with dynamic parameters in the feature extractor. We conduct experiments on the code summarization and code completion tasks to verify the effectiveness of our approach. The results demonstrate the superiority of our approach with significant improvements on state-of-the-art baselines.

Evaluating a Large Language Model on Searching for GUI Layouts

The field of generative artificial intelligence has seen significant advancements in recent years with the advent of large language models, which have shown impressive results in software engineering tasks but not yet in engineering user interfaces. Thus, we raise a specific research question: would an LLM-based system be able to search for relevant GUI layouts? To address this question, we conducted a controlled study evaluating how Instigator, an LLM-based system for searching GUI layouts of web pages by generative pre-trained training, would return GUI layouts that are relevant to a given instruction and what would be the user experience of (N =34) practitioners interacting with Instigator. Our results identify a very high similarity and a moderate correlation between the rankings of the GUI layouts generated by Instigator and the rankings of the practitioners with respect to their relevance to a given design instruction. We highlight the results obtained through thirteen UEQ+ scales that characterize the user experience of the practitioner with Instigator, which we use to discuss perspectives for improving such future tools.

Microservice architecture recovery based on intra-service and inter-service features

Microservice architecture supports independent development and deployment; it facilitates software system design and co-development. However, it also brings new challenges to a variety of software engineering tasks, especially in reverse engineering. An improper design or maintenance routine may cause complex invocation, obscure code logic, and complicate service layers, which may lead to difficulties in understanding, even further testing, or maintenance. To reduce the severity of this problem, we present a novel microservice architecture recovery technique that parses the source code to build a fine-grained dependency graph. This process recovers six key information components of the microservice architecture, which helps developers understand the system. Experimental results based on 12 projects show that the recovered accuracy is 94% on average. The results benefit any engineer unfamiliar with the project, increases their answering accuracy by 23.81% on average, and reduces their training time by 65.43% on average.

LEARNING THE EFFICIENT ESTIMATION TECHNIQUES FOR SUCCESSFUL SOFTWARE PROJECT MANAGEMENT

The process of software project management involves planning and supervising the development of software projects to deliver a quality product within the customer’s budget and schedule. This process begins with project planning, which includes estimating the work to be done, required resources, and project schedule. Once these activities are accomplished, a project schedule is established that defines software engineering tasks, identifies responsible parties, and specifies inter-task dependencies. The paper aims to explore project management activities and techniques for estimating project size. Overall, software project management involves managing, allocating, and timing resources to develop software that meets requirements and is delivered within budget and schedule. This paper highlights the significance of employing efficient estimation methods to achieve successful software project management. Estimation plays a critical role in the software development process, as it helps project managers to determine the resources and time required for completing the project. The paper starts by introducing the concept of software project management and highlighting its importance in delivering successful projects. The advantages and disadvantages of each technique are discussed, and guidelines for selecting the appropriate technique for a specific project are provided. The paper also explores the importance of accurate estimation in agile software development and the use of estimation tools to simplify the process. Finally, the paper concludes by summarizing the key takeaways from the discussion and emphasizing the significance of efficient estimation techniques in ensuring successful software project management.