Software Engineering Research Community Research Articles

Test Code Flakiness in Mobile Apps: The Developer’s Perspective

Context:Test flakiness arises when test cases have a non-deterministic, intermittent behavior that leads them to either pass or fail when run against the same code. While researchers have been contributing to the detection, classification, and removal of flaky tests with several empirical studies and automated techniques, little is known about how the problem of test flakiness arises in mobile applications. Objective:We point out a lack of knowledge on: (1) The prominence and harmfulness of the problem; (2) The most frequent root causes inducing flakiness; and (3) The strategies applied by practitioners to deal with it in practice. An improved understanding of these matters may lead the software engineering research community to assess the need for tailoring existing instruments to the mobile context or for brand-new approaches that focus on the peculiarities identified. Methods:We address this gap of knowledge by means of an empirical study into the mobile developer’s perception of test flakiness. We first perform a systematic grey literature review to elicit how developers discuss and deal with the problem of test flakiness in the wild. Then, we complement the systematic review through a survey study that involves 130 mobile developers and that aims at analyzing their experience on the matter. Results:The results of the grey literature review indicate that developers are often concerned with flakiness connected to user interface elements. In addition, our survey study reveals that flaky tests are perceived as critical by mobile developers, who pointed out major production code- and source code design-related root causes of flakiness, other than the long-term effects of recurrent flaky tests. Furthermore, our study lets the diagnosing and fixing processes currently adopted by developers and their limitations emerge. Conclusion:We conclude by distilling lessons learned, implications, and future research directions.

Summary of the 1st Interpretability and Robustness in Neural Software Engineering (InteNSE 2023)

InteNSE is an interdisciplinary workshop for research at the intersection of Machine Learning (ML) and Software Engineering (SE) and would be a pioneer in emphasizing the implicit properties of neural software engineering and analysis. Due to recent computational advancements, ML has become an inseparable part of the SE research community. ML can indeed improve and revolutionize many SE tasks. However, most research in the AI and SE communities consider ML as a closed box, i.e., only considering the final performance of the developed models as an evaluation metric. Ignoring the implicit properties of neural models, such as interpretability and robustness, one cannot validate the model's actual performance, generalizability, and whether it is learning what it is supposed to do. Specifically, in the domain of SE, where the result of ML4SE tools is code synthesis, bug finding, or repair, interpretability and robustness are crucial to ensure the reliability of the products.

The anatomy of a vulnerability database: A systematic mapping study

Software vulnerabilities play a major role, as there are multiple risks associated, including loss and manipulation of private data. The software engineering research community has been contributing to the body of knowledge by proposing several empirical studies on vulnerabilities and automated techniques to detect and remove them from source code. The reliability and generalizability of the findings heavily depend on the quality of the information mineable from publicly available datasets of vulnerabilities as well as on the availability and suitability of those databases. In this paper, we seek to understand the anatomy of the currently available vulnerability databases through a systematic mapping study where we analyze (1) what are the popular vulnerability databases adopted; (2) what are the goals for adoption; (3) what are the other sources of information adopted; (4) what are the methods and techniques; (5) which tools are proposed. An improved understanding of these aspects might not only allow researchers to take informed decisions on the databases to consider when doing research but also practitioners to establish reliable sources of information to inform their security policies and standards.

Data Preparation for Software Vulnerability Prediction: A Systematic Literature Review

Software Vulnerability Prediction (SVP) is a data-driven technique for software quality assurance that has recently gained considerable attention in the Software Engineering research community. However, the difficulties of preparing Software Vulnerability (SV) related data is considered as the main barrier to industrial adoption of SVP approaches. Given the increasing, but dispersed, literature on this topic, it is needed and timely to systematically select, review, and synthesize the relevant peer-reviewed papers reporting the existing SV data preparation techniques and challenges. We have carried out a Systematic Literature Review (SLR) of SVP research in order to develop a systematized body of knowledge of the data preparation challenges, solutions, and the needed research. Our review of the 61 relevant papers has enabled us to develop a taxonomy of data preparation for SVP related challenges. We have analyzed the identified challenges and available solutions using the proposed taxonomy. Our analysis of the state of the art has enabled us identify the opportunities for future research. This review also provides a set of recommendations for researchers and practitioners of SVP approaches.

Pitfalls and guidelines for using time-based Git data

Many software engineering research papers rely on time-based data (e.g., commit timestamps, issue report creation/update/close dates, release dates). Like most real-world data however, time-based data is often dirty. To date, there are no studies that quantify how frequently such data is used by the software engineering research community, or investigate sources of and quantify how often such data is dirty. Depending on the research task and method used, including such dirty data could affect the research results. This paper presents an extended survey of papers that utilize time-based data, published in the Mining Software Repositories (MSR) conference series. Out of the 754 technical track and data papers published in MSR 2004–2021, we saw at least 290 (38%) papers utilized time-based data. We also observed that most time-based data used in research papers comes in the form of Git commits, often from GitHub. Based on those results, we then used the Boa and Software Heritage infrastructures to help identify and quantify several sources of dirty Git timestamp data. Finally we provide guidelines/best practices for researchers utilizing time-based data from Git repositories.

Static test flakiness prediction: How Far Can We Go?

Test flakiness is a phenomenon occurring when a test case is non-deterministic and exhibits both a passing and failing behavior when run against the same code. Over the last years, the problem has been closely investigated by researchers and practitioners, who all have shown its relevance in practice. The software engineering research community has been working toward defining approaches for detecting and addressing test flakiness. Despite being quite accurate, most of these approaches rely on expensive dynamic steps, e.g., the computation of code coverage information. Consequently, they might suffer from scalability issues that possibly preclude their practical use. This limitation has been recently targeted through machine learning solutions that could predict the flakiness of tests using various features, like source code vocabulary or a mixture of static and dynamic metrics computed on individual snapshots of the system. In this paper, we aim to perform a step forward and predict test flakiness only using static metrics. We propose a large-scale experiment on 70 Java projects coming from the iDFlakies and FlakeFlagger datasets. First, we statistically assess the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells, analyzing both their individual and combined effects. Based on the results achieved, we experiment with a machine learning approach that predicts test flakiness solely based on static features, comparing it with two state-of-the-art approaches. The key results of the study show that the static approach has performance comparable to those of the baselines. In addition, we found that the characteristics of the production code might impact the performance of the flaky test prediction models.

Socio-Technical Grounded Theory for Software Engineering

Grounded Theory (GT), a sociological research method designed to study social phenomena, is increasingly being used to investigate the human and social aspects of software engineering (SE). However, being written by and for sociologists, GT is often challenging for a majority of SE researchers to understand and apply. Additionally, SE researchers attempting ad hoc adaptations of traditional GT guidelines for modern socio-technical (ST) contexts often struggle in the absence of clear and relevant guidelines to do so, resulting in poor quality studies. To overcome these research community challenges and leverage modern research opportunities, this paper presents <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Socio-Technical Grounded Theory</i> (STGT) designed to ease application and achieve quality outcomes. It defines what exactly is meant by an ST research context and presents the STGT guidelines that expand GT's philosophical foundations, provide increased clarity and flexibility in its methodological steps and procedures, define possible scope and contexts of application, encourage frequent reporting of a variety of interim, preliminary, and mature outcomes, and introduce nuanced evaluation guidelines for different outcomes. It is hoped that the SE research community and related ST disciplines such as computer science, data science, artificial intelligence, information systems, human computer/robot/AI interaction, human-centered emerging technologies (and increasingly other disciplines being transformed by rapid digitalisation and AI-based augmentation), will benefit from applying STGT to conduct quality research studies and systematically produce rich findings and mature theories with confidence.

Determining a core view of research quality in empirical software engineering

Context:Research quality is intended to appraise the design and reporting of studies. It comprises a set of standards such as methodological rigor, practical relevance, and conformance to ethical standards. Depending on the perspective, different views of importance are given to the standards for research quality. Objective:To investigate the suitability of a conceptual model of research quality to Software Engineering (SE), from the perspective of researchers engaged in Empirical Software Engineering (ESE) research, in order to understand the core value of research quality. Method:We conducted a mixed-methods approach with two distinct group perspectives: (i) a research group; and (ii) the empirical SE research community. Our data collection approach comprised a questionnaire survey and a complementary focus group. We carried out a hierarchical voting prioritization to collect relative values for importance of standards for research quality. Results:In the context of this research, ‘internally valid’, ‘relevant research idea’, and ‘applicable results’ are perceived as the core standards for research quality in empirical SE. The alignment at the research group level was higher compared to that at the community level. Conclusion:The conceptual model was seen to express fairly the standards for research quality in the SE context. It presented limitations regarding its structure and components’ description, which resulted in an updated model.

A study on the use of vulnerabilities databases in software engineering domain

Over the last decade several software vulnerability databases have been introduced to guide researchers and developers in developing more secure and reliable software. While the Software Engineering research community is increasingly becoming aware of these vulnerabilities databases, no comprehensive literature survey exists that studies how they are used in software development. The objective of our survey is to provide insights on how the software vulnerability database (SVDBs) research landscape has evolved over the past 17 years and outline some open challenges associated with their use in non-security domain. More specifically, we introduce a semi-automated methodology based on topic modeling, to discover relevant topics from our dataset of 99 relevant SE research articles. We find 24 topics discussing the use of SVDBs in SE domain. The results shows that i) topics describing the use of SVDBs range from security empirical (case) studies to tools for generating security test cases; ii) the majority of the surveyed papers cover a limited number of software engineering contributions or activities (e.g., maintenance) and iii) that most of the surveyed articles rely on only one SVDB as their knowledge source. Dataset and results are available at https://github.com/isultane/svdbs_dataset

Modelling in low-code development: a multi-vocal systematic review

In 2014, a new software development approach started to get a foothold: low-code development. Already from its early days, practitioners in software engineering have been showing a rapidly growing interest in low-code development. In 2021 only, the revenue of low-code development technologies reached 13.8 billion USD. Moreover, the business success of low-code development has been sided by a growing interest from the software engineering research community. The model-driven engineering community has shown a particular interest in low-code development due to certain similarities between the two. In this article, we report on the planning, execution, and results of a multi-vocal systematic review on low-code development, with special focus to its relation to model-driven engineering. The review is intended to provide a structured and comprehensive snapshot of low-code development in its peak of inflated expectations technology adoption phase. From an initial set of potentially relevant 720 peer-reviewed publications and 199 grey literature sources, we selected 58 primary studies, which we analysed according to a meticulous data extraction, analysis, and synthesis process. Based on our results, we tend to frame low-code development as a set of methods and/or tools in the context of a broader methodology, often being identified as model-driven engineering.

Emerging Software Engineering Research Networks in (East) Africa

Software engineering (SE) researchers and research networks from emerging communities are often not visible in already established Software Engineering venues for a multitude of reasons. This limits the opportunities and mutual bene ts that can arise from collaborations between global and emerging Software Engineer- ing networks. This article focuses on a rst attempt to provide a map of the African software engineering research community with focus on the networks of two big East African Universities. We hope that this very initial mapping e ort will help to raise aware- ness in the international community about the variety of software engineering research in Africa. We formulate some suggestions for making our academic Software Engineering community more inclusive.

A critical review on the evaluation of automated program repair systems

Automated Program Repair (APR) has attracted significant attention from software engineering research and practice communities in the last decade. Several teams have recorded promising performance in fixing real bugs and there is a race in the literature to fix as many bugs as possible from established benchmarks. Gradually, repair performance of APR tools in the literature has gone from being evaluated with a metric on the number of generated plausible patches to the number of correct patches. This evolution is necessary after a study highlighting the overfitting issue in test suite-based automatic patch generation. Simultaneously, some researchers are also insisting on providing time cost in the repair scenario as a metric for comparing state-of-the-art systems.In this paper, we discuss how the latest evaluation metrics of APR systems could be biased. Since design decisions (both in approach and evaluation setup) are not always fully disclosed, the impact on repair performance is unknown and computed metrics are often misleading. To reduce notable biases of design decisions in program repair approaches, we conduct a critical review on the evaluation of patch generation systems and propose eight evaluation metrics for fairly assessing the performance of APR tools. Eventually, we show with experimental data on 11 baseline program repair systems that the proposed metrics allow to highlight some caveats in the literature. We expect wide adoption of these metrics in the community to contribute to boosting the development of practical, and reliably performable program repair tools.

Improved software defect prediction using Pruned Histogram-based isolation forest

Software defect prediction (SDP) is a hot topic in the modern software engineering research community. It has been used for evaluating software quality and reliability and allocating limited testing resources effectively. Based on analyzing the software source code and development process and extracting critical metrics, many data mining and machine learning methods have been used for SDP. However, these existing learning methods have difficulty with handling the imbalanced data distribution of accumulated training dataset. Isolation forest, an anomaly detection method based on the ensemble learning, has been studied to deal with the imbalanced data distribution issue for obtaining high prediction performance. However, the isolation forest method suffers from a main drawback of slow convergence, which is caused by selecting the feature value at random during the process of building isolation trees. To conquer this problem, in this paper histogram is constructed for the value set of selected isolation feature helping identify feature values preferable to build isolation trees. Motivated by the “many could be better than all” principle in the ensemble learning, the ensemble pruning strategy is further employed to optimize the obtained isolation forest, leading to a novel SDP method named PHIForest (Pruned Histogram-based Isolation Forest) in this work. The proposed method can provide fast convergence through the histogram-based splitting feature value selection, and decrease the ensemble scale and improve prediction performance through the ensemble pruning. Comprehensive experiments conducted on ten real datasets are performed to demonstrate effectiveness of the proposed SDP method.

The Symposium on Search-Based Software Engineering: Past, Present and Future

Abstract Context Search-Based Software Engineering (SBSE) is the research field where Software Engineering (SE) problems are modelled as search problems to be solved by search-based techniques. The Symposium on Search Based Software Engineering (SSBSE) is the premier event on SBSE, which had its 11th edition in 2019. Objective In order to better understand the characteristics and evolution of papers published at SSBSE, this work reports results from a mapping study targeting the proceedings of all SSBSE editions. Despite the existing mapping studies on SBSE, our contribution in this work is to provide information to researchers and practitioners willing to enter the SBSE field, being a source of information to strengthen the symposium, guide new studies, and motivate new collaboration among research groups. Method A systematic mapping study was conducted with a set of four research questions, in which 134 studies published in all editions of SSBSE, dated from 2009 to 2019, were evaluated. In a fifth question, 32 papers published in the challenge track were summarised. Results Throughout the years, 290 authors from 25 countries have contributed to the main track of the symposium, with the collaboration of at least two institutions in 46.3% of the papers. SSBSE papers have got substantial external visibility, as most citations are from different venues. The SE tasks addressed by SSBSE are mostly related to software testing, software debugging, software design, and maintenance. Evolutionary algorithms are present in 75% of the papers, being the most common search technique. The evaluation of the SBSE approaches usually includes industrial systems. Conclusions SSBSE has helped increase the popularity of SBSE in the SE research community and has played an important role on making SBSE mature. There are still problems and challenges to be addressed in the SBSE field, which can be tackled by SSBSE authors in further studies.

A practical guide on conducting eye tracking studies in software engineering

For several years, the software engineering research community used eye trackers to study program comprehension, bug localization, pair programming, and other software engineering tasks. Eye trackers provide researchers with insights on software engineers’ cognitive processes, data that can augment those acquired through other means, such as on-line surveys and questionnaires. While there are many ways to take advantage of eye trackers, advancing their use requires defining standards for experimental design, execution, and reporting. We begin by presenting the foundations of eye tracking to provide context and perspective. Based on previous surveys of eye tracking for programming and software engineering tasks and our collective, extensive experience with eye trackers, we discuss when and why researchers should use eye trackers as well as how they should use them. We compile a list of typical use cases—real and anticipated—of eye trackers, as well as metrics, visualizations, and statistical analyses to analyze and report eye-tracking data. We also discuss the pragmatics of eye tracking studies. Finally, we offer lessons learned about using eye trackers to study software engineering tasks. This paper is intended to be a one-stop resource for researchers interested in designing, executing, and reporting eye tracking studies of software engineering tasks.

Code smells and refactoring: A tertiary systematic review of challenges and observations

Refactoring and smells have been well researched by the software-engineering research community these past decades. Several secondary studies have been published on code smells, discussing their implications on software quality, their impact on maintenance and evolution, and existing tools for their detection. Other secondary studies addressed refactoring, discussing refactoring techniques, opportunities for refactoring, impact on quality, and tools support.In this paper, we present a tertiary systematic literature review of previous surveys, secondary systematic literature reviews, and systematic mappings. We identify the main observations (what we know) and challenges (what we do not know) on code smells and refactoring. We perform this tertiary review using eight scientific databases, based on a set of five research questions, identifying 40 secondary studies between 1992 and 2018.We organize the main observations and challenges about code smell and their refactoring into: smells definitions, most common code-smell detection approaches, code-smell detection tools, most common refactoring, and refactoring tools. We show that code smells and refactoring have a strong relationship with quality attributes, i.e., with understandability, maintainability, testability, complexity, functionality, and reusability. We argue that code smells and refactoring could be considered as the two faces of a same coin. Besides, we identify how refactoring affects quality attributes, more than code smells. We also discuss the implications of this work for practitioners, researchers, and instructors. We identify 13 open issues that could guide future research work.Thus, we want to highlight the gap between code smells and refactoring in the current state of software-engineering research. We wish that this work could help the software-engineering research community in collaborating on future work on code smells and refactoring.

Adopting the Appropriate Performance Measures for Soft Computing-based Estimation by Analogy

Soft Computing based estimation by analogy is a lucrative research domain for the software engineering research community. There are a considerable number of models proposed in this research area. Therefore, researchers are of interest to compare the models to identify the best one for software development effort estimation. This research showed that most of the studies used mean magnitude of relative error (MMRE) and percentage of prediction (PRED) for the comparison of their estimation models. Still, it was also found in this study that there are quite a number of criticisms done on accuracy statistics like MMRE and PRED by renowned authors. It was found that MMRE is an unbalanced, biased, and inappropriate performance measure for identifying the best among competing estimation models. The accuracy statistics, e.g., MMRE and PRED, are still adopted in the evaluation criteria by the domain researchers, stating the reason for “widely used,” which is not a valid reason. This research study identified that, since there is no practical solution provided so far, which could replace MMRE and PRED, the researchers are adopting these measures. The approach of partitioning the large dataset into subsamples was tried in this paper using estimation by analogy (EBA) model. One small and one large dataset were considered for it, such as Desharnais and ISBSG release 11. The ISBSG dataset is a large dataset concerning Desharnais. The ISBSG dataset was partitioned into subsamples. The results suggested that when the large datasets are partitioned, the MMRE produces the same or nearly the same results, which it produces for the small dataset. It is observed that the MMRE can be trusted as a performance metric if the large datasets are partitioned into subsamples.

Integrating Software Engineering Processes in the Development of Efficient Intrusion Detection Systems in Wireless Sensor Networks.

Applying Software Engineering (SE) processes is vital to critical and complex systems including security and networking systems. Nowadays, Wireless Sensor Networks (WSNs) and their applications are found in many military and civilian systems which make them attractive to security attackers. The increasing risks and system vulnerabilities of WSNs have encouraged researchers and developers to propose many security solutions including software-based Intrusion Detection Systems (IDSs). The main drawbacks of current IDSs are due to the lack of clear, structured software development processes. Unfortunately, a substantial gap has been observed between WSN and SE research communities. Integrating SE and WSNs is an emerging topic that will be expanded as technology evolves and spreads in all life aspects. Consequently, this paper highlighted the importance of Requirement Engineering, Software Design, and Testing when developing IDSs for WSNs. Three software IDS designs were proposed in this study: Scheduling, Broadcast, and Watchdog designs. The three designs were compared in terms of consumed energy and network lifetime. Although the same IDS approach was used, but, by highlighting the design phase and implementing different designs, the network lifetime was increased by 73.6% and the consumed energy was reduced by 20% in some of the designs. This is a clear indication of how following a proper SE process could affect the performance of the IDS in WSN. Moreover, conclusions were drawn in regard to applying software engineering processes to IDSs to deliver the required functionalities, with respect to operational constraints, with an improved performance, accuracy and reliability.

Need for Sleep: The Impact of a Night of Sleep Deprivation on Novice Developers’ Performance

We present a quasi-experiment to investigate whether, and to what extent, sleep deprivation impacts the performance of novice software developers using the agile practice of test-first development (TFD). We recruited 45 undergraduates, and asked them to tackle a programming task. Among the participants, 23 agreed to stay awake the night before carrying out the task, while 22 slept normally. We analyzed the quality (i.e., the functional correctness) of the implementations delivered by the participants in both groups, their engagement in writing source code (i.e., the amount of activities performed in the IDE while tackling the programming task) and ability to apply TFD (i.e., the extent to which a participant is able to apply this practice). By comparing the two groups of participants, we found that a single night of sleep deprivation leads to a reduction of 50 percent in the quality of the implementations. There is notable evidence that the developers’ engagement and their prowess to apply TFD are negatively impacted. Our results also show that sleep-deprived developers make more fixes to syntactic mistakes in the source code. We conclude that sleep deprivation has possibly disruptive effects on software development activities. The results open opportunities for improving developers’ performance by integrating the study of sleep with other psycho-physiological factors in which the software engineering research community has recently taken an interest in.

A Multi-Study Investigation into Dead Code

Dead code is a bad smell and it appears to be widespread in open-source and commercial software systems. Surprisingly, dead code has received very little empirical attention from the software engineering research community. In this paper, we present a multi-study investigation with an overarching goal to study, from the perspective of researchers and developers, when and why developers introduce dead code, how they perceive and cope with it, and whether dead code is harmful. To this end, we conducted semi-structured interviews with software professionals and four experiments at the University of Basilicata and the College of William & Mary. The results suggest that it is worth studying dead code not only in the maintenance and evolution phases, where our results suggest that dead code is harmful, but also in the design and implementation phases. Our results motivate future work to develop techniques for detecting and removing dead code and suggest that developers should avoid this smell.