Software Engineering Community Research Articles

Detecting and Explaining Python Name Errors

Python has become one of the most popular programming languages nowadays but has not received enough attention from the software engineering community. Many errors, either fixed or not yet, have been scattered in the lifetime of Python projects, including popular Python libraries that have already been reused. NameError is among one of those errors that are widespread in the Python community, as confirmed in our empirical study. Yet, our community has not put effort into helping developers mitigate its introductions. To fill this gap, we propose in this work a static analysis-based approach called DENE (short for Detecting and Explaining Name Errors) to automatically detect and explain name errors in Python projects. To this end, DENE builds control-flow graphs for Python projects and leverages a scope-aware reaching definition analysis to locate identifiers that may cause name errors at runtime and report their locations. Experimental results on carefully crafted ground truth demonstrate that DENE is effective in detecting name errors in real-world Python projects. The results also confirm that unknown name errors are still widely presented in popular Python projects and libraries, and the outputs of DENE can indeed help developers understand why the name errors are flagged as such.

Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal

Context:The Open Science (OS) movement promotes the value of making public the research artifacts (datasets, analysis scripts, guidelines, etc.) used during empirical studies. OS is widely known in areas such as Medicine or Biology, where the process of sharing research artifacts is subject to strict protocols. Unfortunately, in Software Engineering (SE), this process is carried out in a non-systematic way, resulting in incomplete or inaccurate material shared by researchers, which hinders the reproducibility and replicability of empirical studies. Nevertheless, in recent years, it seems that the Empirical Software Engineering (ESE) community is embracing some of the proposed OS initiatives, such as the one proposed by the Association for Computing Machinery (ACM), which provides a badge system to evaluate the quality of a research artifact. This badge system has been adopted by several SE conferences as a method of assessing research artifacts. Aims:Focusing on human-oriented experiments (HOEs) in SE, whose research artifacts are more complex than those for computational experiments, this work applies Design Science Research (DSR) with a twofold purpose: (i) review the current status of HOEs research artifacts publication through evaluation of this practice in the most relevant ESE journals , and (ii) propose a structured outline for HOEs research artifacts driven by the aforementioned ACM badging policy. Method:Regarding the first purpose, we carried out a survey to analyze the current status of the publication of research artifacts considering relevant peer review journals and the quality of 106 research artifacts published in these journals with respect to the ACM badging policy. For the second purpose, an iterative process was carried out to review the content of 106 research artifacts research and their concordance with ACM badges, obtaining a structured scheme for HOEs research artifacts that has been validated through a detailed review of 12 research artifacts obtained from some of those of ACM badges in relevant SE conferences. In addition, we validated the proposal in the research artifacts of 2 of our own experiments. Results:Our survey reveals issues such as the 39,70% of journal studies making completely accessible their research artifacts; most of the analyzed research artifacts are incomplete; the most common repositories used in the ESE community to share the research artifacts are GitHub, institutional repositories, and Zenodo. On the other hand, the validated and structured research artifact outline consists of a list of ordered sections containing a set of artifacts, which can be mandatory or not to achieve a particular ACM badge. For its internal validation, several improvement iterations on the first release of the outline have been carried out based on the conference guidelines, the ACM badging policy, and other relevant proposals. Conclusions:Although the ESE community is making great efforts in standardization, review, and digital publishing related to OS, the availability and completeness of research artifacts can be improved. Our proposal for the elaboration of structured research artifact outline meets the requirements of HOEs in SE. Nevertheless, further research is needed not only to improve and externally validate it but also to disseminate its use among the research community.

Benchmarking and Categorizing the Performance of Neural Program Repair Systems for Java

Recent years have seen a rise in neural program repair systems in the software engineering community, which adopt advanced deep learning techniques to automatically fix bugs. Having a comprehensive understanding of existing systems can facilitate new improvements in this area and provide practical instructions for users. However, we observe two potential weaknesses in the current evaluation of NPR systems: ① published systems are trained with varying data, and ② NPR systems are roughly evaluated through the number of totally fixed bugs. Questions such as “what types of bugs are repairable for current systems” cannot be answered yet. Consequently, researchers can not make target improvements in this area and users have no idea of the real affair of existing systems. In this paper, we perform a systematic evaluation of the existing nine state-of-the-art NPR systems. To perform a fair and detailed comparison, we (1) build a new benchmark and framework that supports training and validating the nine systems with unified data, and (2) evaluate retrained systems with detailed performance analysis, especially on the effectiveness and the efficiency. We believe our benchmark tool and evaluation results could offer practitioners the real affairs of current NPR systems and the implications of further facilitating the improvements of NPR.

Uncovering gender gap in academia: A comprehensive analysis within the software engineering community

Gender gap in education has gained considerable attention in recent years, as it carries profound implications for the academic community. However, while the problem has been tackled from a student perspective, research is still lacking from an academic point of view. In this work, our main objective is to address this unexplored area by shedding light on the intricate dynamics of gender gap within the Software Engineering (SE) community. To this aim, we first review how the problem of gender gap in the SE community and in academia has been addressed by the literature so far. Results show that men in SE build more tightly-knit clusters but less global co-authorship relations than women, but the networks do not exhibit homophily. Concerning academic promotions, the Software Engineering community presents a higher bias in promotions to Associate Professors and a smaller bias in promotions to Full Professors than the overall Informatics community.

An Interview with Massimiliano Di Penta - 2024 SIGSOFT Awardee

Massimiliano Di Penta received the 2024 SIGSOFT Early Career Researcher Award for outstanding service to the software engineering community, including serving as program co-chair for all premier software engineering conferences and several other events. Massimiliano received a Ph.D. in Computer Engineering at the University of Sannio (Italy) in 2003. He is now a Full Professor at the University of Sannio (Italy), where he is also the Director of the PhD program in Information Technology for Engineering, and has been Director of the Bachelor's and Master's Degree Programs in Computer Science Engineering.

Open Archaeology, Open Source? Collaborative practices in an emerging community of archaeological software engineers

In this article, we investigate modes of collaboration in this emerging community of practice using 'open-archaeo ', a curated list of archaeological software, and data on the activity of associated GitHub repositories and users. We conduct an exploratory quantitative analysis to characterise the nature and intensity of these collaborations and map the collaborative networks that emerge from them. We document uneven adoption of open source collaborative practices beyond the basic use of git as a version control system and GitHub to host source code. Most projects do make use of collaborative features and, through shared contributions, we can trace a collaborative network that includes the majority of archaeologists active on GitHub. However, a majority of repositories have 1-3 contributors, with only a few projects distinguished by an active and diverse developer base. Direct collaboration on code or other repository content - as opposed to the more passive, social media-style interaction that GitHub supports – remains very limited. In other words, there is little evidence that archaeologists ' adoption of open-source tools (git and GitHub) has been accompanied by the decentralised, participatory forms of collaboration that characterise other open-source communities. On the contrary, our results indicate that research software engineering in archaeology remains largely embedded in conventional professional norms and organisational structures of academia.

What is discussed about Flutter on Stack Overflow (SO) question-and-answer (Q&A) website: An empirical study

Context:Mobile applications are growing in prevalence. Cross-platform application is one of the main challenges within mobile application development. Aiming to facilitate the process of mobile application development, the software engineering community developed multiple solutions to support cross-platform application development. Flutter is a recent cross-platform software development kit (SDK) that has been rising in popularity. Objective:This study aims to gain a better understanding of Flutter stance on Stack Overflow (SO). Method:The study identified and analyzed 176,876 Flutter-related questions to understand the interest towards Flutter. The study utilized Latent Dirichlet Allocation (LDA) to identify Flutter-related topics discussed within the identified questions. Subsequently, a number of heuristics were utilized to gauge the popularity and difficulty of topics. Results:The study revealed that interest towards Flutter was steadily increasing on SO until it dropped in 2023 and SO users discuss 12 Flutter-related topics, with the topic of setup and build automation being the most popular and most difficult. Conclusion:This study highlights the interest towards Flutter on SO, identifies Flutter-related topics discussed on the website, and assess the topics popularity and difficulty. Software researchers, practitioners, educators, and Flutter contributors may utilize the results to steer their future Flutter-related efforts.

Test Optimization in DNN Testing: A Survey

This article presents a comprehensive survey on test optimization in deep neural network (DNN) testing. Here, test optimization refers to testing with low data labeling effort. We analyzed 90 papers, including 43 from the software engineering (SE) community, 32 from the machine learning (ML) community, and 15 from other communities. Our study: (i) unifies the problems as well as terminologies associated with low-labeling cost testing, (ii) compares the distinct focal points of SE and ML communities, and (iii) reveals the pitfalls in existing literature. Furthermore, we highlight the research opportunities in this domain.

Forum Text Processing and Summarization

Frequently Asked Questions (FAQs) are extensively studied in general domains like the medical field, but such frameworks are lacking in domains such as software engineering and open-source communities. This research aims to bridge this gap by establishing the foundations of an automated FAQ Generation and Retrieval framework specifically tailored to the software engineering domain. The framework involves analyzing, ranking, performing sentiment analysis, and summarization techniques on open forums like StackOverflow and GitHub issues. A corpus of Stack Overflow post data is collected to evaluate the proposed framework and the selected models. Integrating state-of-the-art models of string-matching models, sentiment analysis models, summarization models, and the proprietary ranking formula proposed in this paper forms a robust Automatic FAQ Generation and Retrieval framework to facilitate developers' work. String matching, sentiment analysis, and summarization models are evaluated, and F1 scores of 71.31%, 74.90%, and 53.4% were achieved. Given the subjective nature of evaluations in this context, a human review is used to further validate the effectiveness of the overall framework, with assessments made on relevancy, preferred ranking, and preferred summarization. Future work includes improving summarization models by incorporating text classification and summarizing them individually (Kou et al, 2023), as well as proposing feedback loop systems based on human reinforcement learning. Furthermore, efforts will be made to optimize the framework by utilizing knowledge graphs for dimension reduction, enabling it to handle larger corpora effectively

Research artifacts in software engineering publications: Status and trends

The Software Engineering (SE) community has been embracing the open science policy and encouraging researchers to disclose artifacts in their publications. However, the status and trends of artifact practice and quality remain unclear, lacking insights on further improvement. In this paper, we present an empirical study to characterize the research artifacts in SE publications. Specifically, we manually collect 1,487 artifacts from all 2,196 papers published in top-tier SE conferences (ASE, FSE, ICSE, and ISSTA) from 2017 to 2022. We investigate the common practices (e.g., URL location and format, storage websites), maintenance activities (e.g., last update time and URL validity), popularity (e.g., the number of stars on GitHub and characteristics), and quality (e.g., documentation and code smell) of these artifacts. Based on our analysis, we reveal a rise in publications providing artifacts. The usage of Zenodo for sharing artifacts has significantly increased. However, artifacts stored in GitHub tend to receive few stars, indicating a limited influence on real-world SE applications. We summarize the results and provide suggestions to different stakeholders in conjunction with current guidelines.

The Lost World: Characterizing and Detecting Undiscovered Test Smells

Test smell refers to poor programming and design practices in testing and widely spreads throughout software projects. Considering test smells have negative impacts on the comprehension and maintenance of test code and even make code-under-test more defect-prone, it thus has great importance in mining, detecting, and refactoring them. Since Deursen et al. introduced the definition of “test smell”, several studies worked on discovering new test smells from test specifications and software practitioners’ experience. Indeed, many bad testing practices are “observed” by software developers during creating test scripts rather than through academic research and are widely discussed in the software engineering community (e.g., Stack Overflow) [ 70 , 94 ]. However, no prior studies explored new bad testing practices from software practitioners’ discussions, formally defined them as new test smell types, and analyzed their characteristics, which plays a bad role for developers in knowing these bad practices and avoiding using them during test code development. Therefore, we pick up those challenges and act by working on systematic methods to explore new test smell types from one of the most mainstream developers’ Q&A platforms, i.e., Stack Overflow. We further investigate the harmfulness of new test smells and analyze possible solutions for eliminating them. We find that some test smells make it hard for developers to fix failed test cases and trace their failing reasons. To exacerbate matters, we have identified two types of test smells that pose a risk to the accuracy of test cases. Next, we develop a detector to detect test smells from software. The detector is composed of six detection methods for different smell types. These detection methods are both wrapped with a set of syntactic rules based on the code patterns extracted from different test smells and developers’ code styles. We manually construct a test smell dataset from seven popular Java projects and evaluate the effectiveness of our detector on it. The experimental results show that our detector achieves high performance in precision, recall, and F1 score. Then, we utilize our detector to detect smells from 919 real-world Java projects to explore whether the six test smells are prevalent in practice. We observe that these test smells are widely spread in 722 out of 919 Java projects, which demonstrates that they are prevalent in real-world projects. Finally, to validate the usefulness of test smells in practice, we submit 56 issue reports to 53 real-world projects with different smells. Our issue reports achieve 76.4% acceptance by conducting sentiment analysis on developers’ replies. These evaluations confirm the effectiveness of our detector and the prevalence and practicality of new test smell types on real-world projects.

Poison Attack and Poison Detection on Deep Source Code Processing Models

In the software engineering (SE) community, deep learning (DL) has recently been applied to many source code processing tasks, achieving state-of-the-art results. Due to the poor interpretability of DL models, their security vulnerabilities require scrutiny. Recently, researchers have identified an emergent security threat to DL models, namely, poison attacks . The attackers aim to inject insidious backdoors into DL models by poisoning the training data with poison samples. The backdoors mean that poisoned models work normally with clean inputs but produce targeted erroneous results with inputs embedded with specific triggers. By using triggers to activate backdoors, attackers can manipulate poisoned models in security-related scenarios (e.g., defect detection) and lead to severe consequences. To verify the vulnerability of deep source code processing models to poison attacks, we present a poison attack approach for source code named CodePoisoner as a strong imaginary enemy. CodePoisoner can produce compilable and functionality-preserving poison samples and effectively attack deep source code processing models by poisoning the training data with poison samples. To defend against poison attacks, we further propose an effective poison detection approach named CodeDetector . CodeDetector can automatically identify poison samples in the training data. We apply CodePoisoner and CodeDetector to six deep source code processing models, including defect detection, clone detection, and code repair models. The results show that ❶ CodePoisoner conducts successful poison attacks with a high attack success rate (average: 98.3%, maximum: 100%). It validates that existing deep source code processing models have a strong vulnerability to poison attacks. ❷ CodeDetector effectively defends against multiple poison attack approaches by detecting (maximum: 100%) poison samples in the training data. We hope this work can help SE researchers and practitioners notice poison attacks and inspire the design of more advanced defense techniques.

Semantic feature-based test selection for deep neural networks: A frequency domain perspective

While deep neural networks (DNNs) have great potential for applications in security and safety-critical domains, their limited robustness to adversarial samples and out-of-distribution (OOD) samples raise significant concerns. In the software engineering community, significant efforts have been devoted to devising testing techniques that verify the robustness of DNNs. This paper investigates semantic feature-based test selection for DNNs from a frequency domain perspective and propose a novel method called SaFeTS. Specifically, we leverage saliency detection techniques, such as Fourier Phase Transform to extract semantic features from test cases. These features are then clustered to select diverse test cases to evaluate the robustness of DNNs and model retraining. Experiments on CIFAR-10 and SVHN datasets demonstrate that SaFeTS exposes more varied model errors com16 pared to baseline methods. Further, retraining with SaFeTS-selected samples significantly improves adversarial and out-of-distribution robustness over state-of-the-art test selection methods.

Self-Checking Deep Neural Networks for Anomalies and Adversaries in Deployment

Deep Neural Networks (DNNs) have been widely adopted, yet DNN models are surprisingly unreliable, which raises significant concerns about their use in critical domains. In this work, we propose that runtime DNN mistakes can be quickly detected and properly dealt with <i>in deployment</i>, especially in settings like self-driving vehicles. Just as software engineering (SE) community has developed effective mechanisms and techniques to monitor and check programmed components, our previous work, SelfChecker, is designed to monitor and correct DNN predictions given unintended abnormal test data. SelfChecker triggers an alarm if the decisions given by the internal layer features of the model are inconsistent with the final prediction and provides <i>advice</i> in the form of an alternative prediction. In this paper, we extend SelfChecker to the security domain. Specifically, we describe SelfChecker++, which we designed to target both <i>unintended</i> abnormal test data and <i>intended</i> adversarial samples. Technically, we develop a technique which can transform any runtime inputs triggering alarms into semantically equivalent inputs, then we feed those transformed inputs to the model. Such runtime transformation nullifies any intended crafted samples, making the model immune to adversarial attacks that craft adversarial samples. We evaluated the alarm accuracy of SelfChecker++ on three DNN models and four popular image datasets, and found that SelfChecker++ triggers correct alarms on 63.10% of wrong DNN predictions, and triggers false alarms on 5.77% of correct DNN predictions. We also evaluated the effectiveness of SelfChecker++ in detecting adversarial examples and found it detects on average 70.09% of such samples with advice accuracy that is 20.89% higher than the original DNN model and 18.37% higher than SelfChecker.

Summary of the 1st Interpretability and Robustness in Neural Software Engineering (InteNSE 2023)

InteNSE is an interdisciplinary workshop for research at the intersection of Machine Learning (ML) and Software Engineering (SE) and would be a pioneer in emphasizing the implicit properties of neural software engineering and analysis. Due to recent computational advancements, ML has become an inseparable part of the SE research community. ML can indeed improve and revolutionize many SE tasks. However, most research in the AI and SE communities consider ML as a closed box, i.e., only considering the final performance of the developed models as an evaluation metric. Ignoring the implicit properties of neural models, such as interpretability and robustness, one cannot validate the model's actual performance, generalizability, and whether it is learning what it is supposed to do. Specifically, in the domain of SE, where the result of ML4SE tools is code synthesis, bug finding, or repair, interpretability and robustness are crucial to ensure the reliability of the products.

An Observational Study on React Native (RN) Questions on Stack Overflow (SO)

Mobile applications are continuously increasing in prevalence. One of the main challenges in mobile application development is creating cross-platform applications. To facilitate developing cross-platform applications, the software engineering community created several solutions, one of which is React Native (RN), which is a popular cross-platform framework. The software engineering literature demonstrated the effectiveness of Stack Overflow (SO) in providing real-world perspectives on a variety of technical subjects. Therefore, this study aims to gain a better understanding of the stance of RN on SO. We identified and analyzed 131,620 SO RN-related questions. Moreover, we observed how the interest toward RN on SO evolves over time. Additionally, we utilized Latent Dirichlet Allocation (LDA) to identify RN-related topics that are discussed within the questions. Afterward, we utilized a number of proxy measures to estimate the popularity and difficulty of these topics. The results revealed that interest toward RN on SO was generally increasing. Moreover, RN-related questions revolve around six topics, with the topics of layout and navigation being the most popular and the topic of iOS issues being the most difficult. Software engineering researchers, practitioners, educators, and RN contributors may find the results of this study beneficial in guiding their future RN efforts.

Relating team atmosphere and group dynamics to student software development teams’ performance

ContextWhile the software engineering community (i.e., those involved with engineering software) is constantly in search of insights into team atmosphere and group dynamics and the way these issues impact team performance, little opportunities typically exist to explore this issue. Student projects offer an opportunity for us to understand these issues, and particularly if these students are on the verge of leaving university for post-study work and using similar practices to those used in industry. ObjectiveWe explore a range of student software development projects’ data and students’ open-ended responses to five group dynamics categories: communication, time management, commitment, problem analysis and solving, and initiative and involvement. MethodWe analyse both quantitative and qualitative data to study the variation in group dynamics across teams developing different software and how these variations correlated with team satisfaction. We also explore the group dynamics themes that evolve from students’ open responses in relation to the five categories. Furthermore, we relate the prevalence of the themes to various software development performance metrics, before exploring the opportunity of predicting an optimum team dynamics. ResultsWe observe variations in the way different teams work, but higher performing teams also committed more to their projects. Various group dynamics themes were evident among functional teams, and specific patterns were more pronounced when teams were productive. Further, while there is no specific group dynamics pattern that predicts project success, successful teams were most organised and reflective. ConclusionCompetence may set the tone for positive group dynamics and team performance. Also, an achievement-driven orientation is as important as the soft skills and interpersonal aspects.

Fairness-aware machine learning engineering: how far are we?

Machine learning is part of the daily life of people and companies worldwide. Unfortunately, bias in machine learning algorithms risks unfairly influencing the decision-making process and reiterating possible discrimination. While the interest of the software engineering community in software fairness is rapidly increasing, there is still a lack of understanding of various aspects connected to fair machine learning engineering, i.e., the software engineering process involved in developing fairness-critical machine learning systems. Questions connected to the practitioners’ awareness and maturity about fairness, the skills required to deal with the matter, and the best development phase(s) where fairness should be faced more are just some examples of the knowledge gaps currently open. In this paper, we provide insights into how fairness is perceived and managed in practice, to shed light on the instruments and approaches that practitioners might employ to properly handle fairness. We conducted a survey with 117 professionals who shared their knowledge and experience highlighting the relevance of fairness in practice, and the skills and tools required to handle it. The key results of our study show that fairness is still considered a second-class quality aspect in the development of artificial intelligence systems. The building of specific methods and development environments, other than automated validation tools, might help developers to treat fairness throughout the software lifecycle and revert this trend.

Software Industry Perception of Technical Debt and Its Management

Technical debt (TD) expresses the lack of internal quality directly affecting software evolution. Therefore, it has gained the attention of software researchers and practitioners recently. Software researchers have performed empirical studies to observe the perspective of TD in different software cultures and organizations. However, it is important to replicate such studies in more places and with more practitioners to strengthen the perception of TD. In this paper, we present the results of a set of new research questions from an evolved survey design of a survey replication in the Uruguayan software industry to characterize how the software industry professionals understand, perceive, and adopt TD management (TDM) activities. The results allow us to observe that different participant contexts (startups, government, job roles) show different levels of awareness and perception of TD. Details in the form of the adoption of each TDM activity were presented. We could observe some difficulties in conducting some TDM activities that the practitioners consider very important, especially in TDM and monitoring. Differences in specific organizational contexts like startups and government could indicate the need for research efforts in other software engineering communities that meet their specific TD challenges and needs.

Sahand 1.0: A new model for extracting information from source code in object-oriented projects

Providing models that enable developers, architects, and executives to make intelligent decisions about software projects is imperative. Static analyzer tools can extract the information required by developers from the source code to achieve various goals. There are generally two types of tools for analyzing source codes in the software engineering community: commercial tools and open-source tools. Commercial tools provide a limited set of features, pre-built by the tool developers, for extracting information from source code. Open-source tools, while also having limited features, often have fewer features than commercial tools. Extending open-source tools to support new and unembedded features requires a complete knowledge of the programming language used in the tools as well as the architecture used in the modules of tools. Additionally, tool extension requires a complete knowledge of the Abstract Syntax Tree (AST) structure used in those tools. To overcome existing problems, this paper proposes a new model named Sahand, which utilizes a Relational Database Repository (RDP) to store the source code. To this end, utilizing RDP, we propose a new data model to store the source code of the object-oriented projects aiming to extract the required information from it. This new data model provides an infrastructure for developers to extract information from the source code using only the SQL language or its extensions. This model facilitates the extraction of previously inaccessible information, such as the maximum depth of the inheritance tree in the entire project or the maximum method call sequence in the project. The proposed model has been further investigated using two large-scale projects namely JavaParser and Tomcat, consisting of 1802 and 3566 classes, respectively. All source codes and database model used for Sahand are open-source and can be accessed at https://github.com/gClassAcademy/Sahand.