Software Engineering Research Research Articles

Editorial para el número especial de la 12a Conferencia Internacional sobre Investigación e Innovación en Ingeniería de Software

Software Engineering is responsible for analyzing, designing, implementing, testing and maintaining software developments. It is currently a very important area since software is found everywhere in our lives, we are immersed in technological advances. Given the importance of Software Engineering, this special issue presents the research articles in Spanish from the 12th International Conference on Software Engineering Research and Innovation.

A survey on Cryptoagility and Agile Practices in the light of quantum resistance

Context:Crypto-agility, a name that stems from agile methodologies for software development, means the ability to modify quickly and securely cryptographic algorithms in the event of a compromise. The advent of quantum computing poses existential threats to current cryptography, having the power to breach current cryptography systems. Objective:We investigated whether and to what extent agile practices for software development are suited to support crypto-agility, or not. In particular, we discuss their usefulness in the context of substituting current algorithms with quantum-resistant ones. Method:First, we analyzed the literature to define a subset of 15 agile practices potentially relevant to cryptographic software development. Then, we developed a questionnaire to assess the suitability of agile practices for obtaining crypto-agility. We performed a Web search of relevant documents about crypto-agility and quantum resistance and sent their authors the questionnaire. We also sent the questionnaire to cybersecurity officers of four Italian firms. We analyzed and discussed the responses to 32 valid questionnaires. Results:The respondents’ affiliations are evenly distributed between researchers and developers. Most of them are active, or somehow active, in quantum-resistant cryptography and use agile methods. Most of the agile practices are deemed to be quite useful, or very useful to get crypto-agility, the most effective being Continuous Integration and Coding Standards; the least appreciated is Self-organizing Team. Conclusion:According to researchers and developers working in the field, the safe transition of cryptographic algorithms to quantum-resistant ones can benefit from the adoption of many agile practices. Further software engineering research is needed to integrate agile practices in more formal cryptographic software development processes.

Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs

Over the past few years, Large Language Models of Code (Code LLMs) have started to have a significant impact on programming practice. Code LLMs are also emerging as building blocks for research in programming languages and software engineering. However, the quality of code produced by a Code LLM varies significantly by programming language. Code LLMs produce impressive results on high-resource programming languages that are well represented in their training data (e.g., Java, Python, or JavaScript), but struggle with low-resource languages that have limited training data available (e.g., OCaml, Racket, and several others). This paper presents an effective approach for boosting the performance of Code LLMs on low-resource languages using semi-synthetic data. Our approach, called MultiPL-T, generates high-quality datasets for low-resource languages, which can then be used to fine-tune any pretrained Code LLM. MultiPL-T translates training data from high-resource languages into training data for low-resource languages in the following way. 1) We use a Code LLM to synthesize unit tests for commented code from a high-resource source language, filtering out faulty tests and code with low test coverage. 2) We use a Code LLM to translate the code from the high-resource source language to a target low-resource language. This gives us a corpus of candidate training data in the target language, but many of these translations are wrong. 3) We use a lightweight compiler to compile the test cases generated in (1) from the source language to the target language, which allows us to filter our obviously wrong translations. The result is a training corpus in the target low-resource language where all items have been validated with test cases. We apply this approach to generate tens of thousands of new, validated training items for five low-resource languages: Julia, Lua, OCaml, R, and Racket, using Python as the source high-resource language. Furthermore, we use an open Code LLM (StarCoderBase) with open training data (The Stack), which allows us to decontaminate benchmarks, train models without violating licenses, and run experiments that could not otherwise be done. Using datasets generated with MultiPL-T, we present fine-tuned versions of StarCoderBase and Code Llama for Julia, Lua, OCaml, R, and Racket that outperform other fine-tunes of these base models on the natural language to code task. We also present Racket fine-tunes for two very recent models, DeepSeek Coder and StarCoder2, to show that MultiPL-T continues to outperform other fine-tuning approaches for low-resource languages. The MultiPL-T approach is easy to apply to new languages, and is significantly more efficient and effective than alternatives such as training longer.

The Trailer of the ACM 2030 Roadmap for Software Engineering

The landscape of software engineering has dramatically changed. The recent advances in AI, the new opportunities of quantum computing, and the new challenges of sustainability and cyber security upset the software engineering research prospective. The 2030 SE-Roadmap special issue of ACM TOSEM Transactions on Software Engineering and Methodology gives a 360° view of the research challenges of the 30ties with a thorough editorial, four roadmap papers from the ACM TOSEM editorial board, and over 30 peer-reviewed papers from the research community. This paper previews the main content of the 2030 roadmap special issue with a report from the 2030 Software Engineering Roadmap workshop, co-located with ACM SIGSOFT FSE Foundations of Software engineering on July 15th and 16th, 2024 in Porto de Galinhas, Brazil, that spotlighted the software engineering research horizon to feed ideas into the ACM TOSEM special issue. The paper discusses the new challenges to software engineering that emerged in the SE2030 workshop: AI for software engineering, software engineering by and for humans, sustainable software engineering, automatic programming, cyber security, validation and verification, and quantum software engineering.

Mobile Software Engineering is Coming to an End (Like All Good Things Must)

Mobile Software Engineering as a separate research field has been relevant to speeding up Software Engineering research in the area of mobile devices and applications. However, the existence of Mobile Software Engineering as a separate research field is coming to an end. This development has implications for existing and future research. These implications and resulting opportunities are discussed in this paper.

7th International Workshop on Software-intensive Business (IWSiB 2024): Software-intensive Business in the Era of Generative Artificial Intelligence

The 7th International Workshop on Software-intensive Business (IWSiB'24), held on April 16, 2024, in Lisbon, Portugal, as part of the 46th International Conference on Software Engineering (ICSE 2024), brought together diverse research communities to address the pressing challenges faced by software-producing organizations. These challenges, driven by changing demands, evolving technologies, and dynamic environments, require interdisciplinary collaboration across software engineering and business research. The workshop focused on the theme "Software Business in the Era of Generative Artificial Intelligence," highlighting the transformative impact of GenAI and large language models on software businesses. The event featured a keynote by Jan Bosch and showcased 11 accepted papers on a diverse range of topics. Participants engaged in interactive discussions, group activities, and ideation sessions aimed at identifying future challenges and solutions for software-intensive businesses, fostering ongoing community development and cross-disciplinary synergy.

A Meta-Study of Software-Change Intentions

Every software system undergoes changes, for example, to add new features, fix bugs, or refactor code. The importance of understanding software changes has been widely recognized, resulting in various techniques and studies, for instance, on change-impact analysis or classifying developers’ activities. Since changes are triggered by developers’ intentions—something they plan or want to change in the system—many researchers have studied intentions behind changes. While there appears to be a consensus among software-engineering researchers and practitioners that knowing the intentions behind software changes is important, it is not clear how developers can actually benefit from this knowledge. In fact, there is no consolidated, recent overview of the state of the art on software-change intentions (SCIs) and their relevance for software engineering. We present a meta-study of 122 publications, which we used to derive a categorization of SCIs and to discuss motivations, evidence, and techniques relating to SCIs. Unfortunately, we found that individual pieces of research are often disconnected from each other, because a common understanding is missing. Similarly, some publications showcase the potential of knowing SCIs, but more substantial research to understand the practical benefits of knowing SCIs is needed. Our contributions can help researchers and practitioners improve their understanding of SCIs and how SCIs can aid software engineering tasks.

Leveraging Large Language Models for Automating Inductive Qualitative Coding: A Comparative Study of Prompt Engineering Techniques

This study explores the potential of Large Language Models (LLMs), like those from the GPT family, to automate inductive qualitative coding—a process of assigning codes to text excerpts and organizing them into categories, traditionally done manually in social science and software engineering research. Our primary question is: Can LLMs effectively automate inductive qualitative coding? To investigate this, we compared different prompt engineering techniques, including Zero-shot, Few-shot, and Chain-of-Thought (CoT) learning, in coding interview transcripts. While LLMs cannot fully replace human coders, they can aid the process with a human-in-the-loop approach. Few-shot learning showed consistent performance with moderate amounts data, while CoT proved most effective in reducing partial hallucinations. Initially aimed at full automation, our study pivoted to testing prompt strategies after realizing that a human-in-the-loop process would offer better accuracy and flexibility, given the challenges of context and token limits in LLMs. These findings suggest that tailored LLM with adequate prompting techniques can help assist researchers when performing qualitative analysis.

Software stewardship and advancement of a high-performance computing scientific application: QMCPACK

We provide an overview of the software engineering efforts and their impact in QMCPACK, a production-level ab-initio Quantum Monte Carlo open-source code targeting high-performance computing (HPC) systems. Aspects included are: (i) strategic expansion of continuous integration (CI) targeting CPUs, using GitHub Actions own runners, and NVIDIA and AMD GPUs used in pre-exascale systems, (ii) incremental reduction of memory leaks using sanitizers, (iii) incorporation of Docker containers for CI and reproducibility, and (iv) refactoring efforts to improve maintainability, testing coverage, and memory lifetime management. We quantify the value of these improvements by providing metrics to illustrate the shift towards a predictive, rather than reactive, maintenance approach. Our goal, in documenting the impact of these efforts on QMCPACK, is to contribute to the body of knowledge on the importance of research software engineering (RSE) for the stewardship and advancement of community HPC codes to enable scientific discovery at scale.

Guidelines for using financial incentives in software-engineering experimentation

Context:Empirical studies with human participants (e.g., controlled experiments) are established methods in Software Engineering (SE) research to understand developers’ activities or the pros and cons of a technique, tool, or practice. Various guidelines and recommendations on designing and conducting different types of empirical studies in SE exist. However, the use of financial incentives (i.e., paying participants to compensate for their effort and improve the validity of a study) is rarely mentionedObjective:In this article, we analyze and discuss the use of financial incentives for human-oriented SE experimentation to derive corresponding guidelines and recommendations for researchers. Specifically, we propose how to extend the current state-of-the-art and provide a better understanding of when and how to incentivize.Method:We captured the state-of-the-art in SE by performing a Systematic Literature Review (SLR) involving 105 publications from six conferences and five journals published in 2020 and 2021. Then, we conducted an interdisciplinary analysis based on guidelines from experimental economics and behavioral psychology, two disciplines that research and use financial incentives.Results:Our results show that financial incentives are sparsely used in SE experimentation, mostly as completion fees. Especially performance-based and task-related financial incentives (i.e., payoff functions) are not used, even though we identified studies for which the validity may benefit from tailored payoff functions. To tackle this issue, we contribute an overview of how experiments in SE may benefit from financial incentivisation, a guideline for deciding on their use, and 11 recommendations on how to design them.Conclusions:We hope that our contributions get incorporated into standards (e.g., the ACM SIGSOFT Empirical Standards), helping researchers understand whether the use of financial incentives is useful for their experiments and how to define a suitable incentivisation strategy.

Enhancing effort estimation in global software development using a unique combination of Neuro Fuzzy Logic and Deep Learning Neural Networks (NFDLNN)

ABSTRACT Effective project planning and management in the global software development landscape relies on addressing major issues like cost estimation and effort allocation. Timely estimation of software development is a critical focus in software engineering research. With the industry increasingly relying on diverse teams worldwide, accurate estimation becomes vital. Software size serves as a common measure for costs and schedules, but advanced estimation methods consider various variables, such as project purpose, personnel expertise, time and efficiency constraints, and technology requirements. Estimating software costs involve significant financial and strategic commitments, making it crucial to address complexity and versatility related to cost drivers. To achieve enhanced accuracy and convergence, we employ the cuckoo algorithm in our proposed NFDLNN (Neuro Fuzzy Logic and Deep Learning Neural Networks) model. Through extensive validation with industrial project data, using Function Point Analysis as the algorithmic models, our NFA model demonstrates high accuracy in software cost approximation, outperforming existing methods insights of MRE of 3.33, BRE of 0.13, and PI of 74.48. Our research contributes to improved project planning and decision-making processes in global software development endeavours.

SERP4IoT'24 Workshop Report

We face a new software crisis. In 1968, computer scientists learned that developing robust software requires skills, methods, and tools. Today, software and hardware engineers realize that developing a robust Internet of Things (IoT) also pushes the states of their art and practice. Recent news illustrates the many problems IoT faces: from lack of interoperability to broken updates to massive security attacks. In this context, the 6th International Workshop on Software Engineering Research and Practices for the Internet of Things (SERP4IoT) aims to provide a highly interactive forum for researchers and practitioners to address the challenges of, find solutions for, and share experiences with the development, release, and testing of robust software for IoT systems.

Can GPT-4 Replicate Empirical Software Engineering Research?

Can GPT-4 Replicate Empirical Software Engineering Research?

Technical debt in AI-enabled systems: On the prevalence, severity, impact, and management strategies for code and architecture

Context:Artificial Intelligence (AI) is pervasive in several application domains and promises to be even more diffused in the next decades. Developing high-quality AI-enabled systems — software systems embedding one or multiple AI components, algorithms, and models — could introduce critical challenges for mitigating specific risks related to the systems’ quality. Such development alone is insufficient to fully address socio-technical consequences and the need for rapid adaptation to evolutionary changes. Recent work proposed the concept of AI technical debt, a potential liability concerned with developing AI-enabled systems whose impact can affect the overall systems’ quality. While the problem of AI technical debt is rapidly gaining the attention of the software engineering research community, scientific knowledge that contributes to understanding and managing the matter is still limited. Objective:In this paper, we leverage the expertise of practitioners to offer useful insights to the research community, aiming to enhance researchers’ awareness about the detection and mitigation of AI technical debt. Our ultimate goal is to empower practitioners by providing them with tools and methods. Additionally, our study sheds light on novel aspects that practitioners might not be fully acquainted with, contributing to a deeper understanding of the subject. Method:We develop a survey study featuring 53 AI practitioners, in which we collect information on the practical prevalence, severity, and impact of AI technical debt issues affecting the code and the architecture other than the strategies applied by practitioners to identify and mitigate them. Results:The key findings of the study reveal the multiple impacts that AI technical debt issues may have on the quality of AI-enabled systems (e.g., the high negative impact that Undeclared consumers has on security, whereas Jumbled Model Architecture can induce the code to be hard to maintain) and the little support practitioners have to deal with them, limited to apply manual effort for identification and refactoring. Conclusion:We conclude the article by distilling lessons learned and actionable insights for researchers.

Open Archaeology, Open Source? Collaborative practices in an emerging community of archaeological software engineers

In this article, we investigate modes of collaboration in this emerging community of practice using 'open-archaeo ', a curated list of archaeological software, and data on the activity of associated GitHub repositories and users. We conduct an exploratory quantitative analysis to characterise the nature and intensity of these collaborations and map the collaborative networks that emerge from them. We document uneven adoption of open source collaborative practices beyond the basic use of git as a version control system and GitHub to host source code. Most projects do make use of collaborative features and, through shared contributions, we can trace a collaborative network that includes the majority of archaeologists active on GitHub. However, a majority of repositories have 1-3 contributors, with only a few projects distinguished by an active and diverse developer base. Direct collaboration on code or other repository content - as opposed to the more passive, social media-style interaction that GitHub supports – remains very limited. In other words, there is little evidence that archaeologists ' adoption of open-source tools (git and GitHub) has been accompanied by the decentralised, participatory forms of collaboration that characterise other open-source communities. On the contrary, our results indicate that research software engineering in archaeology remains largely embedded in conventional professional norms and organisational structures of academia.

Challenges, adaptations, and fringe benefits of conducting software engineering research with human participants during the COVID-19 pandemic

The COVID-19 pandemic changed the way we live, work and the way we conduct research. With the restrictions of lockdowns and social distancing, various impacts were experienced by many software engineering researchers, especially whose studies depend on human participants. We conducted a mixed methods study to understand the extent of this impact. Through a detailed survey with 89 software engineering researchers working with human participants around the world and a further nine follow-up interviews, we identified the key challenges faced, the adaptations made, and the surprising fringe benefits of conducting research involving human participants during the pandemic. Our findings also revealed that in retrospect, many researchers did not wish to revert to the old ways of conducting human-orienfted research. Based on our analysis and insights, we share recommendations on how to conduct remote studies with human participants effectively in an increasingly hybrid world when face-to-face engagement is not possible or where remote participation is preferred.

Transformers and meta-tokenization in sentiment analysis for software engineering

Sentiment analysis has been used to study aspects of software engineering, such as issue resolution, toxicity, and self-admitted technical debt. To address the peculiarities of software engineering texts, sentiment analysis tools often consider the specific technical lingo practitioners use. To further improve the application of sentiment analysis, there have been two recommendations: Using pre-trained transformer models to classify sentiment and replacing non-natural language elements with meta-tokens. In this work, we benchmark five different sentiment analysis tools (two pre-trained transformer models and three machine learning tools) on 2 gold-standard sentiment analysis datasets. We find that pre-trained transformers outperform the best machine learning tool on only one of the two datasets, and that even on that dataset the performance difference is a few percentage points. Therefore, we recommend that software engineering researchers should not just consider predictive performance when selecting a sentiment analysis tool because the best-performing sentiment analysis tools perform very similarly to each other (within 4 percentage points). Meanwhile, we find that meta-tokenization does not improve the predictive performance of sentiment analysis tools. Both of our findings can be used by software engineering researchers who seek to apply sentiment analysis tools to software engineering data.

Reporting case studies in systematic literature studies—An evidential problem

Context:The term and label, “case study”, is not used consistently by authors of primary studies in software engineering research. It is not clear whether this problem also occurs for systematic literature studies (SLSs). Objective:To investigate the extent to which SLSs in/correctly use the term and label, “case study”, when classifying primary studies. Methods:We systematically collect two sub-samples (2010–2021 & 2022) comprising a total of eleven SLSs and 79 primary studies. We examine the designs of these SLSs, and then analyse whether the SLS authors and the primary-study authors correctly label the respective primary study as a “case study”. Results:76% of the 79 primary studies are misclassified by SLSs (with the two sub-samples having 60% and 81% misclassification, respectively). For 39% of the 79 studies, the SLSs propagate a mislabelling by the original authors, whilst for 37%, the SLSs introduce a new mislabel, thus making the problem worse. SLSs rarely present explicit definitions for “case study” and when they do, the definition is not consistent with established definitions. Conclusions:SLSs are both propagating and exacerbating the problem of the mislabelling of primary studies as “case studies”, rather than – as we should expect of SLSs – correcting the labelling of primary studies, and thus improving the body of credible evidence. Propagating and exacerbating mislabelling undermines the credibility of evidence in terms of its quantity, quality and relevance to both practice and research.

Supporting Emotional Intelligence, Productivity and Team Goals while Handling Software Requirements Changes

Background: Research shows that emotional intelligence (EI) should be used alongside cognitive intelligence during requirements change (RC) handling in Software Engineering (SE), especially in agile settings. Objective: We wanted to study the role of EI in-depth during RC handling. Method: We conducted a mixed-methods study (an interview study followed by a survey study) with 124 software practitioners. Findings: We found the causal condition, intervening condition and causes lead to key direct consequences of regulating own emotions, managing relationships, and extended consequences of sustaining productivity, setting and sustaining team goals. We found several strategies of supporting EI during RC handling. Further, we found strong correlations between six strategies and one being aware of own emotions, regulating own emotions, sustaining team productivity, and setting and sustaining team goals. Conclusion: Empathising with others and tracking commitments and decisions as a team are key strategies that have strong correlations between managing emotions, between sustaining team productivity, and between setting and sustaining team goals. To the best of our knowledge, the framework we present in this paper is the first theoretical framework on EI in SE research. We provide recommendations for software practitioners to consider during RC handling.

Enablers and Barriers of Empathy in Software Developer and User Interactions: A Mixed Methods Case Study

Software engineering (SE) requires developers to collaborate with stakeholders, and understanding their emotions and perspectives is often vital. Empathy is a concept characterising a person’s ability to understand and share the feelings of another. However, empathy continues to be an under-researched human aspect in SE. We studied how empathy is practised between developers and end users using a mixed methods case study. We used an empathy test, observations, and interviews to collect data and socio-technical grounded theory and descriptive statistics to analyse data. We identified the nature of awareness required to trigger empathy and enablers of empathy. We discovered barriers to empathy and a set of potential strategies to overcome these barriers. We report insights on emerging relationships and present a set of recommendations and potential future works on empathy and SE for software practitioners and SE researchers.