Empirical Software Engineering Research Articles

Promoting open science in test-driven software experiments

A core principle of open science is the clear, concise and accessible publication of empirical data, including “raw” observational data as well as processed results. However, in empirical software engineering there are no established standards (de jure or de facto) for representing and “opening” observations collected in test-driven software experiments — that is, experiments involving the execution of software subjects in controlled scenarios. Execution data is therefore usually represented in ad hoc ways, often making it abstruse and difficult to access without significant manual effort. In this paper we present new data structures designed to address this problem by clearly defining, correlating and representing the stimuli and responses used to execute software subjects in test-driven experiments. To demonstrate their utility, we show how they can be used to promote the repetition, replication and reproduction of experimental evaluations of AI-based code completion tools. We also show how the proposed data structures facilitate the incremental expansion of execution data sets, and thus promote their repurposing for new experiments addressing new research questions.

A longitudinal study on the temporal validity of software samples

ContextIn Empirical Software Engineering, it is crucial to work with representative samples that reflect the current state of the software industry. An important consideration, especially in rapidly changing fields like software development, is that if we use a sample collected years ago, it should continue to represent the same population in the present day to produce generalizable results. However, it is seldom the case in which a software sample built several years ago accurately depicts the current state of the development industry. Nevertheless, many recent studies rely on rather old datasets (seven or more years of age) to conduct their investigations. ObjectiveTo analyze the evolution of a population of open-source projects, determine the likelihood of detecting significant differences over time, and study the activity history of the projects. MethodWe performed a longitudinal study with 72 snapshots of quality projects from Github, covering the period between July 1st 2017 and June 1st 2023. We recorded monthly values of seven repository metrics (contributors, commits, closed pull-requests, merged pull-requests, closed issues, number of stars and forks), encompassing data from a total of 1991 repositories. ResultsWe observed significant changes in all the metrics evaluated, with most cases showing negligible to small effect sizes. Notably, merged pull-requests registered medium effect sizes. The evolution was not equal in all the metrics, however, after five years it was unlikely that a sample of projects remained representative for any of the analyzed metrics, showing probabilities below 25%. ConclusionAlthough the temporal validity of a sample depends on the specific data being studied, employing datasets created several years ago does not appear to be a sound strategy if the aim is to produce results that can be extrapolated to the current state of the population.

Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories

Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories

Guest editorial: special issue on empirical software engineering and measurement

Guest editorial: special issue on empirical software engineering and measurement

Operationalizing validity of empirical software engineering studies

Operationalizing validity of empirical software engineering studies

Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions

There is abundant observational data in the software engineering domain, whereas running large-scale controlled experiments is often practically impossible. Thus, most empirical studies can only report statistical correlations —instead of potentially more insightful and robust causal relations. To support analyzing purely observational data for causal relations, and to assess any differences between purely predictive and causal models of the same data, this paper discusses some novel techniques based on structural causal models (such as directed acyclic graphs of causal Bayesian networks). Using these techniques, one can rigorously express, and partially validate, causal hypotheses; and then use the causal information to guide the construction of a statistical model that captures genuine causal relations—such that correlation does imply causation. We apply these ideas to analyzing public data about programmer performance in Code Jam, a large world-wide coding contest organized by Google every year. Specifically, we look at the impact of different programming languages on a participant’s performance in the contest. While the overall effect associated with programming languages is weak compared to other variables—regardless of whether we consider correlational or causal links—we found considerable differences between a purely associational and a causal analysis of the very same data. The takeaway message is that even an imperfect causal analysis of observational data can help answer the salient research questions more precisely and more robustly than with just purely predictive techniques—where genuine causal effects may be confounded.

“We do not appreciate being experimented on”: Developer and researcher views on the ethics of experiments on open-source projects

A tenet of open source software development is to accept contributions from users–developers (typically after appropriate vetting). But should this also include interventions done as part of research on open source development? Following an incident in which buggy code was submitted to the Linux kernel to see whether it would be caught, we conduct a survey among open source developers and empirical software engineering researchers to see what behaviors they think are acceptable. This covers two main issues: the use of publicly accessible information, and conducting active experimentation. The survey had 224 respondents. The results indicate that open-source developers are largely open to research, provided it is done transparently. In other words, many would agree to experiments on open-source projects if the subjects were notified and provided informed consent, and in special cases also if only the project leaders agree. While researchers generally hold similar opinions, they sometimes fail to appreciate certain nuances that are important to developers. Examples include observing license restrictions on publishing open-source code and safeguarding the code. Conversely, researchers seem to be more concerned than developers about privacy issues. Based on these results, it is recommended that open source repositories and projects address use for research in their access guidelines, and that researchers take care to ask permission also when not formally required to do so. We note too that the open source community wants to be heard, so professional societies and IRBs should consult with them when formulating ethics codes.

Reliability in software engineering qualitative research through Inter-Coder Agreement

The research on empirical software engineering that uses qualitative data analysis is increasing. However, most of them do not deepen into the validity of the findings, specifically in the reliability of coding in which these methodologies rely on. This paper aims to establish a novel theoretical framework that enables a methodological approach for conducting this validity analysis through Inter-Coder Agreement (ICA), based on the use of coefficients to measure the degree of agreement in collaborative coding. We systematically review several existing variants of Krippendorff’s α coefficients and provide a novel common mathematical framework to unify them. Finally, this paper illustrates the use of this theoretical framework in a large case study on DevOps culture. We expect that this work will help researchers who are committed to measuring consensus with quantitative techniques in collaborative coding, conducted as part of a qualitative research, to improve the rigor of their findings.

Framework for experimental evaluation of software solutions in a virtual environment

Objectives. Ready-made information technology solutions used when developing software have various characteristics depending on the objectives to be experimentally obtained. While the selection of appropriate technologies and software tools used in experimental software engineering can be time-consuming, experimental complexity can be reduced by providing the researcher with domain-specific tools. The aim of the study is to design and develop a domain-specific software framework for experimental evaluation of the characteristics of information technology solutions in a virtual environment.Methods. To determine the required characteristics of the software framework, an analysis of software tools for conducting experimental studies to evaluate the characteristics of information technology solutions in a virtual environment was conducted. Methods of decomposition, structural design, and software development were applied to design and develop the framework.Results. A software framework for conducting experimental research has been developed. The design results, key features of the framework and a description of the functionality are presented. The implementation of the framework comprises commands for managing virtual machines and commands for scaffolding. A technique for conducting experimental studies using the framework is proposed.Conclusions. The developed domain-specific software framework addresses shortcomings of existing tools to reduce labor costs when conducting experiments to evaluate information technology solutions. The developed framework and proposed methodology allows the number of programming and markup languages required for setting up a software experiment to be reduced from 3 to 1.

Applying Inter-Rater Reliability and Agreement in collaborative Grounded Theory studies in software engineering

Context:The qualitative research on empirical software engineering that uses Grounded Theory is increasing (GT). The trustworthiness, rigor, and transparency of GT qualitative data analysis can benefit, among others, when multiple analysts juxtapose diverse perspectives and collaborate to develop a common code frame based on a consensual and consistent interpretation. Inter-Rater Reliability (IRR) and/or Inter-Rater Agreement (IRA) are commonly used techniques to measure consensus, and thus develop a shared interpretation. However, minimal guidance is available about how and when to measure IRR/IRA during the iterative process of GT, so researchers have been using ad hoc methods for years. Objective:This paper presents a process for systematically measuring IRR/IRA in GT studies, when appropriate, which is grounded in a previous systematic mapping study on collaborative GT in the field of software engineering. Methods:Meta-science guided us to analyze the issues and challenges of collaborative GT and formalize a process to measure IRR/IRA in GT. Results:This process guides researchers to incrementally generate a theory while ensuring consensus on the constructs that support it, improving trustworthiness, rigor, and transparency, and promoting the communicability, reflexivity, and replicability of the research. Conclusion:The application of this process to a GT study seems to support its feasibility. In the absence of further confirmation, this would represent the first step in a de facto standard to be applied to those GT studies that may benefit from IRR/IRA techniques.

Comparing ϕ and the F-measure as performance metrics for software-related classifications

ContextThe F-measure has been widely used as a performance metric when selecting binary classifiers for prediction, but it has also been widely criticized, especially given the availability of alternatives such as ϕ (also known as Matthews Correlation Coefficient).ObjectivesOur goals are to (1) investigate possible issues related to the F-measure in depth and show how ϕ can address them, and (2) explore the relationships between the F-measure and ϕ.MethodBased on the definitions of ϕ and the F-measure, we derive a few mathematical properties of these two performance metrics and of the relationships between them. To demonstrate the practical effects of these mathematical properties, we illustrate the outcomes of an empirical study involving 70 Empirical Software Engineering datasets and 837 classifiers.ResultsWe show that ϕ can be defined as a function of Precision and Recall, which are the only two performance metrics used to define the F-measure, and the rate of actually positive software modules in a dataset. Also, ϕ can be expressed as a function of the F-measure and the rates of actual and estimated positive software modules. We derive the minimum and maximum value of ϕ for any given value of the F-measure, and the conditions under which both the F-measure and ϕ rank two classifiers in the same order.ConclusionsOur results show that ϕ is a sensible and useful metric for assessing the performance of binary classifiers. We also recommend that the F-measure should not be used by itself to assess the performance of a classifier, but that the rate of positives should always be specified as well, at least to assess if and to what extent a classifier performs better than random classification. The mathematical relationships described here can also be used to re-interpret the conclusions of previously published papers that relied mainly on the F-measure as a performance metric.

Developing research questions in conversation with the literature: operationalization & tool support

Empirical Software Engineering rests on the understanding of practical problems and their solution counterparts. Frequently, solutions are not absolute but relative to the context where the problem is observed. This tends to imply that the solution and the problem unveil gradually together, and hence, researchers are not always in the position to state the research question (RQ) at the onset. Like software engineers when facing blurred requirements, researchers might not be familiar enough with the problem in the early phases of a research to properly scope their RQs (hereafter referred to as RQ Scoping). Here, the literature may play the role of the stakeholders in Agile methods: keeping the focus on the aspects that are essential (vs. accidental) of the RQ. Informed by Inductive Top-Down Theorizing, this article acknowledges RQ Scoping as iterative and incremental, entailing a conversation between the experimental work and literature reviewing. Yet, for literature reviewing to become “Agile” it is not only required to be driven by the RQ but also to have tool support. Tools might bring transparency and traceability, both factors especially welcome in a scenario characterized by testing (is my RQ relevant?) and adjustment (how can I make my RQ relevant?). Specifically, the advent of the RQ in close relationship with the literature advises for “Agile” literature reviewing to be conducted at the place where the literature is naturally kept: the Reference Management System (e.g., Mendeley). This article introduces the theoretical underpinnings, design principles, proof of concept and evaluation for FRAMEndeley, a Mendeley-integrated utility for RQ Scoping.

Determining a core view of research quality in empirical software engineering

Context:Research quality is intended to appraise the design and reporting of studies. It comprises a set of standards such as methodological rigor, practical relevance, and conformance to ethical standards. Depending on the perspective, different views of importance are given to the standards for research quality. Objective:To investigate the suitability of a conceptual model of research quality to Software Engineering (SE), from the perspective of researchers engaged in Empirical Software Engineering (ESE) research, in order to understand the core value of research quality. Method:We conducted a mixed-methods approach with two distinct group perspectives: (i) a research group; and (ii) the empirical SE research community. Our data collection approach comprised a questionnaire survey and a complementary focus group. We carried out a hierarchical voting prioritization to collect relative values for importance of standards for research quality. Results:In the context of this research, ‘internally valid’, ‘relevant research idea’, and ‘applicable results’ are perceived as the core standards for research quality in empirical SE. The alignment at the research group level was higher compared to that at the community level. Conclusion:The conceptual model was seen to express fairly the standards for research quality in the SE context. It presented limitations regarding its structure and components’ description, which resulted in an updated model.

An interview with Xin Xia - 2022 SIGSOFT Awardee

Xin Xia received the 2022 SIGSOFT Early Career Researcher Award for his contributions to AI and SE, mining software repositories, and empirical software engineering. He is the Director of the Software Engineering Application Technology Lab at Huawei, China. He received the Ph.D. degree in Computer Science in 2014 from the College of Computer Science and Technology (Zhejiang University, China), and - prior to joining Huawei - he was an ARC DECRA Fellow and a lecturer at the Faculty of Information Technology, Monash University, Australia.

Investigating replication challenges through multiple replications of an experiment

As Empirical Software Engineering grows in maturity and number of publications, more replications are needed to provide a solid grounding to the evidence found through prior research. However, replication studies are scarce in general and some topics suffer more than others with such scarcity. On top, the challenges associated with replicating empirical studies are not well understood. In this study, we aim to fill this gap by investigating difficulties emerging when replicating an experiment. We used participants with distinct backgrounds to play the role of a research group attempting to replicate an experimental study addressing Highly-Configurable Systems. Seven external close replications in total were performed. After obtaining the quantitative replication results, a focus group session was applied to each group inquiring about the replication experience. We used the grounded theory’s constant comparison method for the qualitative analysis. We have seen in our study that, in the replications performed, most results hold when comparing them with the baseline. However, the participants reported many difficulties in replicating the original study, mostly related to the lack of clarity of the instructions and the presence of defects on replication artifacts. Based on our findings, we provide recommendations that can help mitigate the problems reported. The quality of replication artifacts and the lack of clear instructions might impact an experiment replication. We advocate having good quality replication instructions and well-prepared laboratory packages to foster and enable researchers to perform better replications.

Sampling in software engineering research: a critical review and guidelines

Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.

On the Frontiers of Software Science and Software Engineering

Advances in software engineering, software science, computational intelligence, and intelligent mathematics have led to the establishment of Frontiers in Computer Science—Software (FCSS). FCSS aims to promote transdisciplinary research on software science and engineering (SSE), autonomous systems, and computational intelligence. FCSS covers not only classical empirical software engineering and industrial processes, but also contemporary topics of software science, intelligent programming languages, autonomous code generation, mathematical foundations of software, and programming knowledge bases. FCSS reports empirical studies and emerging topics in software engineering including tools, development platforms, industrial processes, management infrastructures, quality assurance schemes, big data systems, and software migrations across languages and platforms.

Software Requirements Engineer’s Ability Assessment Method Based on Empirical Software Engineering

With the expansion of the scale and complexity of modern software systems, the failure rate of software projects remains high. One of the main reasons for the failure of software projects is the defects in processing software requirements. This paper proposes a software requirements engineer’s ability assessment method based on empirical software engineering to measure the matching degree between the software requirements engineer’s ability and industry expectations. First, collect the recruitment information of software requirements engineers from mainstream recruitment websites. Through natural language processing, the words related to the abilities of the software requirements engineer are counted. These words are summarized in the requirements acquisition, requirements analysis, and other SRE activities, then the industry expectations for various abilities are obtained. Later on, the authors collect the teaching settings of representative SRE courses, reflecting the software requirements engineer’s ability to learn the course. After that, this article defines the ratio of the industry expectation weight to the weight of each SRE activity in teaching as the software requirements engineer’s ability coefficient, which can intuitively reflect the matching degree between the software requirements engineer’s ability and industry expectations. Finally, take the national first-class undergraduate course “SRE” of Jinling Institute of Technology as an example to verify the method’s practicality to a certain extent.

Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality

Statistical analysis is the tool of choice to turn data into information and then information into empirical knowledge. However, the process that goes from data to knowledge is long, uncertain, and riddled with pitfalls. To be valid, it should be supported by detailed, rigorous guidelines that help ferret out issues with the data or model and lead to qualified results that strike a reasonable balance between generality and practical relevance. Such guidelines are being developed by statisticians to support the latest techniques for Bayesian data analysis. In this article, we frame these guidelines in a way that is apt to empirical research in software engineering. To demonstrate the guidelines in practice, we apply them to reanalyze a GitHub dataset about code quality in different programming languages. The dataset’s original analysis [Ray et al.  55 ] and a critical reanalysis [Berger et al.  6 ] have attracted considerable attention—in no small part because they target a topic (the impact of different programming languages) on which strong opinions abound. The goals of our reanalysis are largely orthogonal to this previous work, as we are concerned with demonstrating, on data in an interesting domain, how to build a principled Bayesian data analysis and to showcase its benefits. In the process, we will also shed light on some critical aspects of the analyzed data and of the relationship between programming languages and code quality—such as the impact of project-specific characteristics other than the used programming language. The high-level conclusions of our exercise will be that Bayesian statistical techniques can be applied to analyze software engineering data in a way that is principled, flexible, and leads to convincing results that inform the state-of-the-art while highlighting the boundaries of its validity. The guidelines can support building solid statistical analyses and connecting their results. Thus, they can help buttress continued progress in empirical software engineering research.

Empirical software engineering for evaluating adaptive task-oriented personal assistants: a case study in human/machine event extraction and coding

This paper concerns methodology for evaluating task-oriented personal assistants, where users perform a complex task that has objective success criteria, independent of personal preferences, and the agent provides suggestions to help users repeatedly perform the task consistently and accurately. We develop a systematic approach to evaluating task-oriented personal assistants in normal contexts of use through extending the methodology of empirical software engineering to evaluate effectiveness, efficiency and satisfaction. The approach allows the evaluation of both the human-agent system and of the personal assistant using data obtained by observations of user and system behaviour. A key element of our approach is to define empirically observable conditions that separate the learning period, when users and the agent are learning to perform the task, from the evaluation period, when performance benefits are measured. The methodology is illustrated using the example of a system for users to extract, annotate and code events from news articles.