Software Engineering Community Research Articles

An Interview with Massimiliano Di Penta - 2024 SIGSOFT Awardee

Massimiliano Di Penta received the 2024 SIGSOFT Early Career Researcher Award for outstanding service to the software engineering community, including serving as program co-chair for all premier software engineering conferences and several other events. Massimiliano received a Ph.D. in Computer Engineering at the University of Sannio (Italy) in 2003. He is now a Full Professor at the University of Sannio (Italy), where he is also the Director of the PhD program in Information Technology for Engineering, and has been Director of the Bachelor's and Master's Degree Programs in Computer Science Engineering.

Open Archaeology, Open Source? Collaborative practices in an emerging community of archaeological software engineers

In this article, we investigate modes of collaboration in this emerging community of practice using 'open-archaeo ', a curated list of archaeological software, and data on the activity of associated GitHub repositories and users. We conduct an exploratory quantitative analysis to characterise the nature and intensity of these collaborations and map the collaborative networks that emerge from them. We document uneven adoption of open source collaborative practices beyond the basic use of git as a version control system and GitHub to host source code. Most projects do make use of collaborative features and, through shared contributions, we can trace a collaborative network that includes the majority of archaeologists active on GitHub. However, a majority of repositories have 1-3 contributors, with only a few projects distinguished by an active and diverse developer base. Direct collaboration on code or other repository content - as opposed to the more passive, social media-style interaction that GitHub supports – remains very limited. In other words, there is little evidence that archaeologists ' adoption of open-source tools (git and GitHub) has been accompanied by the decentralised, participatory forms of collaboration that characterise other open-source communities. On the contrary, our results indicate that research software engineering in archaeology remains largely embedded in conventional professional norms and organisational structures of academia.

What is discussed about Flutter on Stack Overflow (SO) question-and-answer (Q&A) website: An empirical study

Context:Mobile applications are growing in prevalence. Cross-platform application is one of the main challenges within mobile application development. Aiming to facilitate the process of mobile application development, the software engineering community developed multiple solutions to support cross-platform application development. Flutter is a recent cross-platform software development kit (SDK) that has been rising in popularity. Objective:This study aims to gain a better understanding of Flutter stance on Stack Overflow (SO). Method:The study identified and analyzed 176,876 Flutter-related questions to understand the interest towards Flutter. The study utilized Latent Dirichlet Allocation (LDA) to identify Flutter-related topics discussed within the identified questions. Subsequently, a number of heuristics were utilized to gauge the popularity and difficulty of topics. Results:The study revealed that interest towards Flutter was steadily increasing on SO until it dropped in 2023 and SO users discuss 12 Flutter-related topics, with the topic of setup and build automation being the most popular and most difficult. Conclusion:This study highlights the interest towards Flutter on SO, identifies Flutter-related topics discussed on the website, and assess the topics popularity and difficulty. Software researchers, practitioners, educators, and Flutter contributors may utilize the results to steer their future Flutter-related efforts.

Test Optimization in DNN Testing: A Survey

This article presents a comprehensive survey on test optimization in deep neural network (DNN) testing. Here, test optimization refers to testing with low data labeling effort. We analyzed 90 papers, including 43 from the software engineering (SE) community, 32 from the machine learning (ML) community, and 15 from other communities. Our study: (i) unifies the problems as well as terminologies associated with low-labeling cost testing, (ii) compares the distinct focal points of SE and ML communities, and (iii) reveals the pitfalls in existing literature. Furthermore, we highlight the research opportunities in this domain.

Forum Text Processing and Summarization

Frequently Asked Questions (FAQs) are extensively studied in general domains like the medical field, but such frameworks are lacking in domains such as software engineering and open-source communities. This research aims to bridge this gap by establishing the foundations of an automated FAQ Generation and Retrieval framework specifically tailored to the software engineering domain. The framework involves analyzing, ranking, performing sentiment analysis, and summarization techniques on open forums like StackOverflow and GitHub issues. A corpus of Stack Overflow post data is collected to evaluate the proposed framework and the selected models. Integrating state-of-the-art models of string-matching models, sentiment analysis models, summarization models, and the proprietary ranking formula proposed in this paper forms a robust Automatic FAQ Generation and Retrieval framework to facilitate developers' work. String matching, sentiment analysis, and summarization models are evaluated, and F1 scores of 71.31%, 74.90%, and 53.4% were achieved. Given the subjective nature of evaluations in this context, a human review is used to further validate the effectiveness of the overall framework, with assessments made on relevancy, preferred ranking, and preferred summarization. Future work includes improving summarization models by incorporating text classification and summarizing them individually (Kou et al, 2023), as well as proposing feedback loop systems based on human reinforcement learning. Furthermore, efforts will be made to optimize the framework by utilizing knowledge graphs for dimension reduction, enabling it to handle larger corpora effectively

Research artifacts in software engineering publications: Status and trends

The Software Engineering (SE) community has been embracing the open science policy and encouraging researchers to disclose artifacts in their publications. However, the status and trends of artifact practice and quality remain unclear, lacking insights on further improvement. In this paper, we present an empirical study to characterize the research artifacts in SE publications. Specifically, we manually collect 1,487 artifacts from all 2,196 papers published in top-tier SE conferences (ASE, FSE, ICSE, and ISSTA) from 2017 to 2022. We investigate the common practices (e.g., URL location and format, storage websites), maintenance activities (e.g., last update time and URL validity), popularity (e.g., the number of stars on GitHub and characteristics), and quality (e.g., documentation and code smell) of these artifacts. Based on our analysis, we reveal a rise in publications providing artifacts. The usage of Zenodo for sharing artifacts has significantly increased. However, artifacts stored in GitHub tend to receive few stars, indicating a limited influence on real-world SE applications. We summarize the results and provide suggestions to different stakeholders in conjunction with current guidelines.

The Lost World: Characterizing and Detecting Undiscovered Test Smells

Test smell refers to poor programming and design practices in testing and widely spreads throughout software projects. Considering test smells have negative impacts on the comprehension and maintenance of test code and even make code-under-test more defect-prone, it thus has great importance in mining, detecting, and refactoring them. Since Deursen et al. introduced the definition of “test smell”, several studies worked on discovering new test smells from test specifications and software practitioners’ experience. Indeed, many bad testing practices are “observed” by software developers during creating test scripts rather than through academic research and are widely discussed in the software engineering community (e.g., Stack Overflow) [ 70 , 94 ]. However, no prior studies explored new bad testing practices from software practitioners’ discussions, formally defined them as new test smell types, and analyzed their characteristics, which plays a bad role for developers in knowing these bad practices and avoiding using them during test code development. Therefore, we pick up those challenges and act by working on systematic methods to explore new test smell types from one of the most mainstream developers’ Q&A platforms, i.e., Stack Overflow. We further investigate the harmfulness of new test smells and analyze possible solutions for eliminating them. We find that some test smells make it hard for developers to fix failed test cases and trace their failing reasons. To exacerbate matters, we have identified two types of test smells that pose a risk to the accuracy of test cases. Next, we develop a detector to detect test smells from software. The detector is composed of six detection methods for different smell types. These detection methods are both wrapped with a set of syntactic rules based on the code patterns extracted from different test smells and developers’ code styles. We manually construct a test smell dataset from seven popular Java projects and evaluate the effectiveness of our detector on it. The experimental results show that our detector achieves high performance in precision, recall, and F1 score. Then, we utilize our detector to detect smells from 919 real-world Java projects to explore whether the six test smells are prevalent in practice. We observe that these test smells are widely spread in 722 out of 919 Java projects, which demonstrates that they are prevalent in real-world projects. Finally, to validate the usefulness of test smells in practice, we submit 56 issue reports to 53 real-world projects with different smells. Our issue reports achieve 76.4% acceptance by conducting sentiment analysis on developers’ replies. These evaluations confirm the effectiveness of our detector and the prevalence and practicality of new test smell types on real-world projects.

Self-Checking Deep Neural Networks for Anomalies and Adversaries in Deployment

Deep Neural Networks (DNNs) have been widely adopted, yet DNN models are surprisingly unreliable, which raises significant concerns about their use in critical domains. In this work, we propose that runtime DNN mistakes can be quickly detected and properly dealt with <i>in deployment</i>, especially in settings like self-driving vehicles. Just as software engineering (SE) community has developed effective mechanisms and techniques to monitor and check programmed components, our previous work, SelfChecker, is designed to monitor and correct DNN predictions given unintended abnormal test data. SelfChecker triggers an alarm if the decisions given by the internal layer features of the model are inconsistent with the final prediction and provides <i>advice</i> in the form of an alternative prediction. In this paper, we extend SelfChecker to the security domain. Specifically, we describe SelfChecker++, which we designed to target both <i>unintended</i> abnormal test data and <i>intended</i> adversarial samples. Technically, we develop a technique which can transform any runtime inputs triggering alarms into semantically equivalent inputs, then we feed those transformed inputs to the model. Such runtime transformation nullifies any intended crafted samples, making the model immune to adversarial attacks that craft adversarial samples. We evaluated the alarm accuracy of SelfChecker++ on three DNN models and four popular image datasets, and found that SelfChecker++ triggers correct alarms on 63.10% of wrong DNN predictions, and triggers false alarms on 5.77% of correct DNN predictions. We also evaluated the effectiveness of SelfChecker++ in detecting adversarial examples and found it detects on average 70.09% of such samples with advice accuracy that is 20.89% higher than the original DNN model and 18.37% higher than SelfChecker.

Summary of the 1st Interpretability and Robustness in Neural Software Engineering (InteNSE 2023)

InteNSE is an interdisciplinary workshop for research at the intersection of Machine Learning (ML) and Software Engineering (SE) and would be a pioneer in emphasizing the implicit properties of neural software engineering and analysis. Due to recent computational advancements, ML has become an inseparable part of the SE research community. ML can indeed improve and revolutionize many SE tasks. However, most research in the AI and SE communities consider ML as a closed box, i.e., only considering the final performance of the developed models as an evaluation metric. Ignoring the implicit properties of neural models, such as interpretability and robustness, one cannot validate the model's actual performance, generalizability, and whether it is learning what it is supposed to do. Specifically, in the domain of SE, where the result of ML4SE tools is code synthesis, bug finding, or repair, interpretability and robustness are crucial to ensure the reliability of the products.

An Observational Study on React Native (RN) Questions on Stack Overflow (SO)

Mobile applications are continuously increasing in prevalence. One of the main challenges in mobile application development is creating cross-platform applications. To facilitate developing cross-platform applications, the software engineering community created several solutions, one of which is React Native (RN), which is a popular cross-platform framework. The software engineering literature demonstrated the effectiveness of Stack Overflow (SO) in providing real-world perspectives on a variety of technical subjects. Therefore, this study aims to gain a better understanding of the stance of RN on SO. We identified and analyzed 131,620 SO RN-related questions. Moreover, we observed how the interest toward RN on SO evolves over time. Additionally, we utilized Latent Dirichlet Allocation (LDA) to identify RN-related topics that are discussed within the questions. Afterward, we utilized a number of proxy measures to estimate the popularity and difficulty of these topics. The results revealed that interest toward RN on SO was generally increasing. Moreover, RN-related questions revolve around six topics, with the topics of layout and navigation being the most popular and the topic of iOS issues being the most difficult. Software engineering researchers, practitioners, educators, and RN contributors may find the results of this study beneficial in guiding their future RN efforts.

Relating team atmosphere and group dynamics to student software development teams’ performance

ContextWhile the software engineering community (i.e., those involved with engineering software) is constantly in search of insights into team atmosphere and group dynamics and the way these issues impact team performance, little opportunities typically exist to explore this issue. Student projects offer an opportunity for us to understand these issues, and particularly if these students are on the verge of leaving university for post-study work and using similar practices to those used in industry. ObjectiveWe explore a range of student software development projects’ data and students’ open-ended responses to five group dynamics categories: communication, time management, commitment, problem analysis and solving, and initiative and involvement. MethodWe analyse both quantitative and qualitative data to study the variation in group dynamics across teams developing different software and how these variations correlated with team satisfaction. We also explore the group dynamics themes that evolve from students’ open responses in relation to the five categories. Furthermore, we relate the prevalence of the themes to various software development performance metrics, before exploring the opportunity of predicting an optimum team dynamics. ResultsWe observe variations in the way different teams work, but higher performing teams also committed more to their projects. Various group dynamics themes were evident among functional teams, and specific patterns were more pronounced when teams were productive. Further, while there is no specific group dynamics pattern that predicts project success, successful teams were most organised and reflective. ConclusionCompetence may set the tone for positive group dynamics and team performance. Also, an achievement-driven orientation is as important as the soft skills and interpersonal aspects.

Fairness-aware machine learning engineering: how far are we?

Machine learning is part of the daily life of people and companies worldwide. Unfortunately, bias in machine learning algorithms risks unfairly influencing the decision-making process and reiterating possible discrimination. While the interest of the software engineering community in software fairness is rapidly increasing, there is still a lack of understanding of various aspects connected to fair machine learning engineering, i.e., the software engineering process involved in developing fairness-critical machine learning systems. Questions connected to the practitioners’ awareness and maturity about fairness, the skills required to deal with the matter, and the best development phase(s) where fairness should be faced more are just some examples of the knowledge gaps currently open. In this paper, we provide insights into how fairness is perceived and managed in practice, to shed light on the instruments and approaches that practitioners might employ to properly handle fairness. We conducted a survey with 117 professionals who shared their knowledge and experience highlighting the relevance of fairness in practice, and the skills and tools required to handle it. The key results of our study show that fairness is still considered a second-class quality aspect in the development of artificial intelligence systems. The building of specific methods and development environments, other than automated validation tools, might help developers to treat fairness throughout the software lifecycle and revert this trend.

Software Industry Perception of Technical Debt and Its Management

Technical debt (TD) expresses the lack of internal quality directly affecting software evolution. Therefore, it has gained the attention of software researchers and practitioners recently. Software researchers have performed empirical studies to observe the perspective of TD in different software cultures and organizations. However, it is important to replicate such studies in more places and with more practitioners to strengthen the perception of TD. In this paper, we present the results of a set of new research questions from an evolved survey design of a survey replication in the Uruguayan software industry to characterize how the software industry professionals understand, perceive, and adopt TD management (TDM) activities. The results allow us to observe that different participant contexts (startups, government, job roles) show different levels of awareness and perception of TD. Details in the form of the adoption of each TDM activity were presented. We could observe some difficulties in conducting some TDM activities that the practitioners consider very important, especially in TDM and monitoring. Differences in specific organizational contexts like startups and government could indicate the need for research efforts in other software engineering communities that meet their specific TD challenges and needs.

Poison Attack and Poison Detection on Deep Source Code Processing Models

In the software engineering (SE) community, deep learning (DL) has recently been applied to many source code processing tasks, achieving state-of-the-art results. Due to the poor interpretability of DL models, their security vulnerabilities require scrutiny. Recently, researchers have identified an emergent security threat to DL models, namely poison attacks . The attackers aim to inject insidious backdoors into DL models by poisoning the training data with poison samples. The backdoors mean that poisoned models work normally with clean inputs but produce targeted erroneous results with inputs embedded with specific triggers. By using triggers to activate backdoors, attackers can manipulate poisoned models in security-related scenarios ( e.g., defect detection) and lead to severe consequences. To verify the vulnerability of deep source code processing models to poison attacks, we present a poison attack approach for source code named CodePoisoner as a strong imaginary enemy. CodePoisoner can produce compilable and functionality-preserving poison samples and effectively attack deep source code processing models by poisoning the training data with poison samples. To defend against poison attacks, we further propose an effective poison detection approach named CodeDetector . CodeDetector can automatically identify poison samples in the training data. We apply CodePoisoner and CodeDetector to six deep source code processing models, including defect detection, clone detection, and code repair models. The results show that ① CodePoisoner conducts successful poison attacks with a high attack success rate (avg: 98.3%, max: 100%). It validates that existing deep source code processing models have a strong vulnerability to poison attacks. ② CodeDetector effectively defends against multiple poison attack approaches by detecting (max: 100%) poison samples in the training data. We hope this work can help SE researchers and practitioners notice poison attacks and inspire the design of more advanced defense techniques.

Sahand 1.0: A new model for extracting information from source code in object-oriented projects

Providing models that enable developers, architects, and executives to make intelligent decisions about software projects is imperative. Static analyzer tools can extract the information required by developers from the source code to achieve various goals. There are generally two types of tools for analyzing source codes in the software engineering community: commercial tools and open-source tools. Commercial tools provide a limited set of features, pre-built by the tool developers, for extracting information from source code. Open-source tools, while also having limited features, often have fewer features than commercial tools. Extending open-source tools to support new and unembedded features requires a complete knowledge of the programming language used in the tools as well as the architecture used in the modules of tools. Additionally, tool extension requires a complete knowledge of the Abstract Syntax Tree (AST) structure used in those tools. To overcome existing problems, this paper proposes a new model named Sahand, which utilizes a Relational Database Repository (RDP) to store the source code. To this end, utilizing RDP, we propose a new data model to store the source code of the object-oriented projects aiming to extract the required information from it. This new data model provides an infrastructure for developers to extract information from the source code using only the SQL language or its extensions. This model facilitates the extraction of previously inaccessible information, such as the maximum depth of the inheritance tree in the entire project or the maximum method call sequence in the project. The proposed model has been further investigated using two large-scale projects namely JavaParser and Tomcat, consisting of 1802 and 3566 classes, respectively. All source codes and database model used for Sahand are open-source and can be accessed at https://github.com/gClassAcademy/Sahand.

Information needs and presentation in agile software development

Agile software companies applying the DevOps approach require collaboration and information sharing between practitioners in various roles to produce value. Adopting new development practices affects how practitioners collaborate, requiring companies to form a closer connection between business strategy and software development. However, the types of information management, sales, and development needed to plan, evaluate features, and reconcile their expectations with each other need to be clarified. To support practitioners in collaborating and realizing changes to their practices, we investigated what information is needed and how it should be represented to support different stakeholders in their tasks. Compared to earlier research, we adopted a holistic approach—by including practitioners throughout the development process—to better understand the information needs from a broader viewpoint. We conducted six workshops and 12 semi-structured interviews at three Finnish small and medium-sized enterprises from different software domains. Thematic analysis was used to identify information-related issues and information and visualization needs for daily tasks. Three themes were constructed as the result of our analysis. Visual information representation catalyzes stakeholder discussion, and supporting information exchange between stakeholder groups is vital for efficient collaboration in software product development. Additionally, user-centric data collection practices are needed to understand how software products are used and to support practitioners’ daily information needs. We also found that a passive way of representing information, such as a dashboard that would disturb practitioners only when attention is needed, was preferred for daily information needs. The software engineering community should consider reviewing the information needs of practitioners from a more holistic view to better understand how tooling support can benefit information exchange between stakeholder groups when making product development decisions and how those tools should be built to accommodate different stakeholder views.

Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspective

Deep learning (DL) has become a key component of modern software. In the “ big model ” era, the rich features of DL-based software (i.e., DL software) substantially rely on powerful DL models, e.g., BERT, GPT-3, and the recently emerging GPT-4, which are trained on the powerful cloud with large datasets. Hence, training effective DL models has become a vital stage in the whole software lifecycle. When training deep learning models, especially those big models, developers need to parallelize and distribute the computation and memory resources amongst multiple devices (e.g., a cluster of GPUs) in the training process, which is known as distributed deep learning training , or distributed training for short. However, the unique challenges that developers encounter in distributed training process have not been studied in the software engineering community. Given the increasingly heavy dependence of current DL-based software on distributed training, this paper aims to fill in the knowledge gap and presents the first comprehensive study on developers’ issues in distributed training. To this end, we focus on popular DL frameworks that support distributed training (including TensorFlow, PyTorch, Keras, and Horovod) and analyze 1,131 real-world developers’ issues about using these frameworks reported on Stack Overflow and GitHub. We construct a fine-grained taxonomy consisting of 30 categories regarding the fault symptoms and summarize common fix patterns for different symptoms. We find that: (1) many distributed-specific faults and non-distributed-specific faults inherently share the same fault symptoms, making it challenging to debug; (2) most of the fault symptoms have frequent fix patterns; (3) about half of the faults are related to system-level configurations. Based on the results, we suggest actionable implications on research avenues that can potentially facilitate the distributed training to develop DL-based software, such as focusing on the frequent and common fix patterns when designing testing or debugging tools, developing efficient testing and debugging techniques for communication configuration along with the synthesis of network configuration analysis, designing new multi-device checkpoint-and-replay techniques to help reproduction, and designing serverless APIs for cloud platforms.

Qualitative software engineering research: Reflections and guidelines

AbstractResearchers are increasingly recognizing the importance of human aspects in software development. Because qualitative methods are used to explore human behavior in‐depth, we believe that studies using such methods will become more common. Existing qualitative software engineering guidelines do not cover the full breadth of qualitative methods and the knowledge on how to use them like in social sciences. The purpose of this study was to extend the software engineering community's current body of knowledge regarding available qualitative methods and their quality assurance frameworks and to provide recommendations and guidelines for their use. With the support of an epistemological argument and a survey of the literature, we suggest that future research would benefit from (1) utilizing a broader set of research methods, (2) more strongly emphasizing reflexivity, and (3) employing qualitative guidelines and quality criteria. We present an overview of three qualitative methods commonly used in social sciences but rarely seen in software engineering research, namely interpretative phenomenological analysis, narrative analysis, and discourse analysis. Furthermore, we discuss the meaning of reflexivity in relation to the software engineering context and suggest means of fostering it. Our paper will help software engineering researchers better select and then guide the application of a broader set of qualitative research methods.

Making existing software quantum safe: A case study on IBM Db2

The software engineering community is facing challenges from quantum computers (QCs). In the era of quantum computing, Shor's algorithm running on QCs can break asymmetric encryption algorithms that classical computers practically cannot. Though the exact date when QCs will become "dangerous" for practical problems is unknown, the consensus is that this future is near. Thus, the software engineering community needs to start making software ready for quantum attacks and ensure quantum safety proactively. We argue that the problem of evolving existing software to quantum-safe software is very similar to the Y2K bug. Thus, we leverage some best practices from the Y2K bug and propose our roadmap, called 7E, which gives developers a structured way to prepare for quantum attacks. It is intended to help developers start planning for the creation of new software and the evolution of cryptography in existing software. In this paper, we use a case study to validate the viability of 7E. Our software under study is the IBM Db2 database system. We upgrade the current cryptographic schemes to post-quantum cryptographic ones (using Kyber and Dilithium schemes) and report our findings and lessons learned. We show that the 7E roadmap effectively plans the evolution of existing software security features towards quantum safety, but it does require minor revisions. We incorporate our experience with IBM Db2 into the revised 7E roadmap. The U.S. Department of Commerce's National Institute of Standards and Technology is finalizing the post-quantum cryptographic standard. The software engineering community needs to start getting prepared for the quantum advantage era. We hope that our experiential study with IBM Db2 and the 7E roadmap will help the community prepare existing software for quantum attacks in a structured manner.

Can the configuration of static analyses make resolving security vulnerabilities more effective? - A user study

The use of static analysis security testing (SAST) tools has been increasing in recent years. However, previous studies have shown that, when shipped to end users such as development or security teams, the findings of these tools are often unsatisfying. Users report high numbers of false positives or long analysis times, making the tools unusable in the daily workflow. To address this, SAST tool creators provide a wide range of configuration options, such as customization of rules through domain-specific languages or specification of the application-specific analysis scope. In this paper, we study the configuration space of selected existing SAST tools when used within the integrated development environment (IDE). We focus on the configuration options that impact three dimensions, for which a trade-off is unavoidable, i.e., precision, recall, and analysis runtime. We perform a between-subjects user study with 40 users from multiple development and security teams - to our knowledge, the largest population for this kind of user study in the software engineering community. The results show that users who configure SAST tools are more effective in resolving security vulnerabilities detected by the tools than those using the default configuration. Based on post-study interviews, we identify common strategies that users have while configuring the SAST tools to provide further insights for tool creators. Finally, an evaluation of the configuration options of two commercial SAST tools, Fortify and CheckMarx, reveals that a quarter of the users do not understand the configuration options provided. The configuration options that are found most useful relate to the analysis scope.