Real-world Software Development Research Articles

Self-collaboration Code Generation via ChatGPT

Although Large Language Models (LLMs) have demonstrated remarkable code-generation ability, they still struggle with complex tasks. In real-world software development, humans usually tackle complex tasks through collaborative teamwork, a strategy that significantly controls development complexity and enhances software quality. Inspired by this, we present a self-collaboration framework for code generation employing LLMs, exemplified by ChatGPT. Specifically, through role instructions, 1) Multiple LLM agents act as distinct ‘experts’, each responsible for a specific subtask within a complex task; 2) Specify the way to collaborate and interact, so that different roles form a virtual team to facilitate each other’s work, ultimately the virtual team addresses code generation tasks collaboratively without the need for human intervention. To effectively organize and manage this virtual team, we incorporate software-development methodology into the framework. Thus, we assemble an elementary team consisting of three LLM roles (i.e., analyst, coder, and tester) responsible for software development’s analysis, coding, and testing stages. We conduct comprehensive experiments on various code-generation benchmarks. Experimental results indicate that self-collaboration code generation relatively improves 29.9%-47.1% Pass@1 compared to the base LLM agent. Moreover, we showcase that self-collaboration could potentially enable LLMs to efficiently handle complex repository-level tasks that are not readily solved by the single LLM agent.

Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation

Recently, large language models (LLMs) have shown an extraordinary ability to understand natural language and generate programming code. It has been a common practice for software engineers to consult LLMs when encountering coding questions. Although efforts have been made to avoid syntax errors and align the code with the intended semantics, the reliability, and robustness of the code generation from LLMs have not yet been thoroughly studied. The executable code is not equivalent to reliable and robust code, especially in the context of real-world software development. For example, the misuse of APIs in the generated code could lead to severe problems, such as resource leaks, program crashes, etc. Existing code evaluation benchmarks and datasets focus on crafting small tasks such as programming questions in coding interviews, which, however, deviates from the problem that developers would ask LLM for real-world coding help. To fill the missing piece, in this work, we propose a dataset RobustAPI for evaluating the reliability and robustness of code generated by LLMs. We collect 1208 coding questions from Stack Overflow on 18 representative Java APIs. We summarize the common misuse patterns of these APIs and evaluate them on current popular LLMs. The evaluation results show that even for GPT-4, 62% of the generated code contains API misuses, which would cause unexpected consequences if the code is introduced into real-world software.

Innovations in Software Configuration: Introducing a Data Comparison Tool Based on the Myers Diff Algorithm

In the realm of software project development, it's not uncommon for changes to emerge at various stages of a project's lifecycle. Such alterations can manifest in virtually every facet of the software development process, from conceptual design decisions down to the minutiae of the source code. To proficiently manage and track these dynamic changes, professionals turn to a specialized toolset known as Software Configuration Management (SCM). One of the standout features that SCM tools bring to the table is the 'diff' capability. This functionality allows developers to identify and review the disparities between two versions of source code. Recognizing the importance and utility of this feature, the primary objective of this research is to create an advanced diff application. This application, by leveraging the Myers Diff algorithm, is meticulously designed to pinpoint and showcase differences in characters between two sets of text-based data. Moreover, it accentuates these differences by visually highlighting the contrasting characters between the two datasets. To ensure the reliability and accuracy of this newly developed tool, we undertook a series of validation tests. We juxtaposed the results from our application against those from a comparable existing tool. Impressively, the discrepancies in results were minimal, with a marginal difference of just 1%. This suggests not only the utility but also the precision of our application in real-world software development scenarios.

Automated regression unit test generation for program merges

Merging other branches into the current working branch is common in collaborative software development. However, developers still heavily rely on the textual merge tools to handle the complicated merge tasks. The latent semantic merge conflicts may fail to be detected and degrade the software quality. Regression testing is able to prevent regression faults and has been widely used in real-world software development. However, the merged software may fail to be well examined by rerunning the existing whole test suite. Intuitively, if the test suite fails to cover the changes of different branches at the same time, the merge conflicts would fail to be detected. Recently, it has been proposed to conduct verification on 3-way merges, but this approach does not support even some common cases such as different changes made to different parts of the program. In this paper, we propose an approach of regression unit test generation specifically for checking program merges according to our proposed test oracles. And our general test oracles support us to examine not only 3-way merges, but also 2-way and octopus merges. Considering the conflicts may arise in other locations besides changed methods of the project, we design an algorithm to select UUTs based on the dependency analysis of the whole project. On this basis, we implement a tool called TOM to generate unit tests for Java program merges. We also design the benchmark MCon4J consisting of 389 conflict 3-way merges and 389 conflict octopus merges to facilitate further studies on this topic. The experimental results show that TOM finds 45 conflict 3- way merges and 87 conflicts octopus merges, while the verification based tool fails to work on MCon4J.

Evaluation of the effectiveness of developing real-world software projects as a motivational device for bridging theory and practice

ABSTRACT While incorporating project-based exercise is a common practice in software engineering education, few studies have been conducted in investigating how real-world project development influences university students’ proactive learning and knowledge transformation. This study aims to evaluate the effectiveness of developing real-world projects with industry engagement in encouraging students to apply knowledge to practice and be more proactive in learning. Using a two-group, post-test quasi-experimental design, the performance between the students taking real-world project development and the students in the control group are compared using descriptive statistics, the independent samples t-test and Welch t-test, accordingly. Both the Spearman’s rank-order and Kendall’s τ-b are used to examine the relationship between students’ practical works and the level of knowledge transformation estimated by the students through online surveys. The results suggest that using real-world projects in the classroom can be an effective motivational device for proactive learning and knowledge transformation. Project-based exercise should be both comprehensive and keeping pace with technology development driven by the software industry evolution to be more effective. The direct interaction with stakeholders, dynamic requirements change, employment of Agile methods, self-organising teams, and using challenging real-world projects, are essential in simulating a real-world software development environment in the classroom.

When should we (not) use the mean magnitude of relative error (MMRE) as an error measure in software development effort estimation?

The mean magnitude of relative error (MMRE) is an error measure frequently used to evaluate and compare the estimation performance of prediction models and software professionals. This paper examines conditions for proper use of MMRE in effort estimation contexts. We apply research on scoring functions to identify the type of estimates that minimizes the expected value of the MMRE. We show that the MMRE is a proper error measure for estimates of the most likely (mode) effort, but not for estimates of the median or mean effort, provided that the effort usage is approximately log-normally distributed, which we argue is a reasonable assumption in many software development contexts. The relevance of the findings is demonstrated on real-world software development data. MMRE is not a proper measure of the accuracy of estimates of the median or mean effort, but may be used for the accuracy evaluation of estimates of most likely effort.

Towards holistic goal centered performance management in software development: lessons from a best practice analysis

There are strong motivating factors for more effective performance measurement practices in software development. Astute practices in this domain are lauded to improve efficiency and effectiveness. However previous studies have shown that measurement in software is intricate, complex and fraught with challenges. Consequently it is poorly managed in practice. Our research seeks to better understand performance management in a real world software development setting in order to identify the challenges and generate a roadmap for improvement. This paper presents findings from an inductive analysis of a radical measurement program in a global software organization. Our study investigates the level at which non-compliance with best practice can explain the company’s disappointing results. We found that a narrow focus on projects, rather than on organizational goals, has seriously hindered its success. We also found that the rate of change in the organization as a whole was impinging on the effective implementation of its measurement program. An analysis of the results demonstrates just how challenging software measurement is. The findings provide an evaluation of best practice relative to the literature that is informed by real industry experience.

Developing CRMSys at SoftTel: Traditional or agile?

ABSTRACT The increasing popularity of agile software development has significantly changed the way organizations plan, design, and implement software applications. However, considering the complexity involved with software development, the decision to choose an agile approach over the traditional waterfall approach is not straightforward. Organizations should carefully evaluate a variety of factors when choosing a development approach. In this teaching case study, the authors provide a discussion case to facilitate the students’ learning of using agile vs. traditional approaches in software development. This case study is developed based on experiences of multiple real-world software development projects and the literature on software development. It is designed to reflect the complexities students are likely to encounter in actual software development projects. The case can be used in undergraduate or graduate level management information systems courses that cover agile software development approaches. Students can gain an understanding of factors to consider when choosing a development approach, which tailors to the organization, the project environment, and the project team.

Formation of communication and teamwork skills of future IT-specialists using project technology

The subject of the study is the formation of communication and teamwork skills of future IT-specialists, using project technology in teaching the administration of computer systems and networks. Content analysis of research has shown that communication and teamwork skills are the most requested and necessary soft skills for future IT professionals. It is offered to use project technology of training for their formation. This technology consists of the approaches and tools application used in real-world software development. To implement this technology, we propose to use mind maps for design. The organization of the working process on the project is implemented in Jira Software, Asana or Trello. Communication among project participants should be implemented with the help of several means: video chats (Zoom, Skype, Google Meet) and written communication in corporate messengers (Slack, Zulip). Experimental verification has shown the efficiency of the proposed design technology implementation.

DeepVS: an efficient and generic approach for source code modelling usage

The source code suggestions provided by current IDEs are mostly dependent on static type learning. These suggestions often end up proposing irrelevant suggestions for a peculiar context. Recently, deep learning-based approaches have shown great potential in the modeling of source code for various software engineering tasks. However, these techniques lack adequate generalization and resistance to acclimate the use of such models in a real-world software development environment. This letter presents \textit{DeepVS}, an end-to-end deep neural code completion tool that learns from existing codebases by exploiting the bidirectional Gated Recurrent Unit (BiGRU) neural net. The proposed tool is capable of providing source code suggestions instantly in an IDE by using pre-trained BiGRU neural net. The evaluation of this work is two-fold, quantitative and qualitative. Through extensive evaluation on ten real-world open-source software systems, the proposed method shows significant performance enhancement and its practicality. Moreover, the results also suggest that \textit{DeepVS} tool is capable of suggesting zero-day (unseen) code tokens by learning coding patterns from real-world software systems.

Modified Pareto archived evolution strategy for the multi-skill project scheduling problem with generalized precedence relations

In this research, we study the multi-skill resource-constrained project scheduling problem, where there are generalized precedence relations between project activities. Workforces are able to perform one or several skills, and their efficiency improves by repeating their skills. For this problem, a mathematical formulation has been proposed that aims to optimize project completion time, reworking risks of activities, and costs of processing the activities, simultaneously. A modified version of the Pareto Archived Evolution Strategy (MV-PAES) is developed to solve the problem. Contrary to the basic PAES, this algorithm operates based on a population of solutions. For the proposed method, we devised crossover and mutation operators, which strengthen this algorithm in exploring solution space. Comprehensive numerical tests have been conducted to evaluate the performance of the MV-PAES in comparison with two other meta-heuristics. The outputs show the excellence of the MV-PAES in comparison with other methods. A real-world software development project has been studied to demonstrate the practicality of the proposed model for real-world environment. The influence of competency evolution has been investigated in this case study. The results imply that the competency evolution has a considerable impact on the objective function values.

A strategic conflict analysis in IT outsourcing using the graph model for conflict resolution

PurposeThe purpose of this paper is to propose a strategic conflict analysis, based on the graph model for conflict resolution (GMCR), that is applied to information technology outsourcing (ITO) in a real-world software development and implementation process in Brazil.Design/methodology/approachBecause the idea of this study is to answer “why” the ITO conflicts occur and “how” they can be avoided, the case study methodology was adopted. The software GMCR II was used to analyze the interactions between an IT vendor and an IT client.FindingsThe results suggest that a lack of relational governance is a critical issue that could be handled to improve the interaction between those involved.Research limitations/implicationsThe main results are restricted to the case study and cannot be generalized. Moreover, a specific limitation of this paper pertains to the use of the GMCR and the consequent difficulty for IT vendors and IT clients to work with a large number of actions and to set preferences for several states of conflict.Practical implicationsThe strategic analysis of outsourcing conflicts provides a holistic view of the current situation that may assist the client and vendor in future decisions and identify guidelines to ensure successful ITO. Therefore, this paper provides an effective guide for clients and vendors to better manage conflicts and establish a contingency vision to avoid such disputes.Originality/valueThe ITO conflict is analyzed using the GMCR, considering both perspectives of the outsourcing process (vendors and clients).

The impact of Use Cases in real-world software development projects: A systematic mapping study

ContextThere is abundant literature on the application of UML Use Cases. However, the impact that these applications have had on real projects (in industry) is not well known. It is necessary to know what the impact of the Use Cases really is, so that both, researchers and professionals can make the most pertinent decisions. ObjectiveTo identify and classify the positive and negative impacts of using Use Cases in real-world settings. MethodWe conducted a systematic mapping study. The search strategies retrieved a set of 4431 papers out of which 47 were selected as primary studies. We defined four facets to classify these studies: 1) the positive impact (advantages), 2) the negative impact (disadvantages), 3) the industry's domain and 4) the type of research reported. ResultsOur study identified eight categories of advantages related to the application of Use Cases. The most mentioned were estimation, analysis and automation. These advantages had a heterogeneous distribution along the years. On the other hand, the granularity of the scenarios described in the Use Cases, the lack of a standardized format for specifying requirements, and the lack of appropriate guidelines for analysing them were the most mentioned disadvantages. We identified a variety of industry domains, grouped into seven categories. As we can expect most of the papers report experiences in the Information Technology domain, followed by financials applications. Almost half the papers applied evaluation research, including an empirical validation. Only one third of the analysed papers reported threats to validity, the most mentioned being generalizability (38%). ConclusionsUse Cases have proven to be a useful tool in software development, particularly during the early stages. The positive effects far outweigh the few negative effects reported, although this may be due to the researchers' tendency of not reporting negative results.

Automatic Real-Time Mining Software Process Activities From SVN Logs Using a Naive Bayes Classifier

The abundance of event data in current software configuration management systems makes it possible to discover software process models automatically by using actual observed behavior. However, traditional process mining algorithms cannot be applied to event logs recorded in software configuration management (SCM) systems, such as SVN, because of missing activity attributes. To address this problem, a software process activity classifier is proposed to build event-activity mapping relationships from software development event streams, revealing activity attributes and associating the activity to the original SVN log. The proposed approach extracts activity from the SVN log based on semantic features and introduces a novel technique based on a naive Bayes approach to associate event activities dynamically. The approach has been applied to two real-world software development process logs, ArgoUML and jEdit , consisting of more than 80,000 events, covering development information from 1998 to 2015. With the application of our approach to such data, activities can be extracted from event logs and a classifier can be constructed for adding activity attributes to new events. The results of the classification are evaluated in terms of precision rate , recall rate, and the F-measure . Overall, two real-world software development process logs are used to validate the method, and the experimental results show that the approach can mine software process activities from SVN log events automatically and in real-time.

Dynamic Software Project Scheduling through a Proactive-Rescheduling Method

Software project scheduling in dynamic and uncertain environments is of significant importance to real-world software development. Yet most studies schedule software projects by considering static and deterministic scenarios only, which may cause performance deterioration or even infeasibility when facing disruptions. In order to capture more dynamic features of software project scheduling than the previous work, this paper formulates the project scheduling problem by considering uncertainties and dynamic events that often occur during software project development, and constructs a mathematical model for the resulting multi-objective dynamic project scheduling problem (MODPSP), where the four objectives of project cost, duration, robustness and stability are considered simultaneously under a variety of practical constraints. In order to solve MODPSP appropriately, a multi-objective evolutionary algorithm based proactive-rescheduling method is proposed, which generates a robust schedule predictively and adapts the previous schedule in response to critical dynamic events during the project execution. Extensive experimental results on 21 problem instances, including three instances derived from real-world software projects, show that our novel method is very effective. By introducing the robustness and stability objectives, and incorporating the dynamic optimization strategies specifically designed for MODPSP, our proactive-rescheduling method achieves a very good overall performance in a dynamic environment.

Input-based adaptive randomized test case prioritization: A local beam search approach

Test case prioritization assigns the execution priorities of the test cases in a given test suite. Many existing test case prioritization techniques assume the full-fledged availability of code coverage data, fault history, or test specification, which are seldom well-maintained in real-world software development projects. This paper proposes a novel family of input-based local-beam-search adaptive-randomized techniques. They make adaptive tree-based randomized explorations with a randomized candidate test set strategy to even out the search space explorations among the branches of the exploration trees constructed by the test inputs in the test suite. We report a validation experiment on a suite of four medium-size benchmarks. The results show that our techniques achieve either higher APFD values than or the same mean APFD values as the existing code-coverage-based greedy or search-based prioritization techniques, including Genetic, Greedy and ART, in both our controlled experiment and case study. Our techniques are also significantly more efficient than the Genetic and Greedy, but are less efficient than ART.

Performance Analysis for Object-Oriented Software: A Systematic Mapping

Performance is a crucial attribute for most software, making performance analysis an important software engineering task. The difficulty is that modern applications are challenging to analyse for performance. Many profiling techniques used in real-world software development struggle to provide useful results when applied to large-scale object-oriented applications. There is a substantial body of research into software performance generally but currently there exists no survey of this research that would help identify approaches useful for object-oriented software. To provide such a review we performed a systematic mapping study of empirical performance analysis approaches that are applicable to object-oriented software. Using keyword searches against leading software engineering research databases and manual searches of relevant venues we identified over 5,000 related articles published since January 2000. From these we systematically selected 253 applicable articles and categorised them according to ten facets that capture the intent, implementation and evaluation of the approaches. Our mapping study results allow us to highlight the main contributions of the existing literature and identify areas where there are interesting opportunities. We also find that, despite the research including approaches specifically aimed at object-oriented software, there are significant challenges in providing actionable feedback on the performance of large-scale object-oriented applications.

Using UML Modeling to Facilitate Three-Tier Architecture Projects in Software Engineering Courses

This article presents the use of a model-centric approach to facilitate software development projects conforming to the three-tier architecture in undergraduate software engineering courses. Many instructors intend that such projects create software applications for use by real-world customers. While it is important that the first version of these applications satisfy the customer by providing the functionality the customer expects and perform reliably and efficiently, it is equally important to be able to accommodate the customer's change requests over the period of the product's lifetime. The challenges in achieving these goals include the lack of real-world software development experience among the student developers and the fact that postdeployment change requests will almost certainly have to be handled by students who are not among the original developers. In this article, we describe how a model-centric approach using UML has been effective in enabling students to develop and maintain eight software applications for small businesses over a 9-year period. We discuss the characteristics of our modeling technique, which include the application of modeling patterns and quality check rules that enable students to create a model that can be clearly and consistently mapped to code. We also describe the nature of these mapping-to-code techniques, emphasizing how they reduce coupling among the implementation's classes. We then discuss our experiences in the classroom with these techniques, focusing on how we have improved our teaching over the years based on the analysis of student performance and feedback. Finally, we compare our approach to related work teaching modeling and the development and maintenance of code in software engineering courses with both extensive and minimal modeling.

A COMPARISON OF STEPWISE AND FUZZY MULTIPLE REGRESSION ANALYSIS TECHNIQUES FOR MANAGING SOFTWARE PROJECT RISKS: ANALYSIS PHASE

Risk is not always avoidable, but it is controllable. The aim of this study is to identify whether those techniques are effective in reducing software failure. This motivates the authors to continue the effort to enrich the managing software project risks with consider mining and quantitative approach with large data set. In this study, two new techniques are introduced namely stepwise multiple regression analysis and fuzzy multiple regression to manage the software risks. Two evaluation procedures such as MMRE and Pred (25) is used to compare the accuracy of techniques. The model’s accuracy slightly improves in stepwise multiple regression rather than fuzzy multiple regression. This study will guide software managers to apply software risk management practices with real world software development organizations and verify the effectiveness of the new techniques and approaches on a software project. The study has been conducted on a group of software project using survey questionnaire. It is hope that this will enable software managers improve their decision to increase the probability of software project success.

Bringing Coq into the World of GCM Distributed Applications

Among all programming paradigms, component-based engineering stands as one of the most followed approaches for real world software development. Its emphasis on clean separation of concerns and reusability makes it appealing for both industrial and research purposes. The Grid Component Model (GCM) endorses this approach in the context of distributed systems by providing all the means to define, compose and dynamically reconfigure component-based applications. While structural reconfiguration is one of the key features of GCM applications, this ability to evolve at runtime poses several challenges w.r.t reliability. In this paper we present Mefresa, a framework for reasoning on the structure of GCM applications. This contribution comes in the form of a formal specification mechanized in the Coq Proof Assistant. Our aim is to demonstrate the benefits of interactive theorem proving for the reasoning on software architectures. We provide a configuration and reconfiguration language for the safe instantiation of distributed systems.