Software Engineering Research Research Articles

Theories in Agile Software Development: Past, Present, and Future Introduction to the XP 2020 Special Section

Over the last two decades, agile software development has gained popularity among software engineering researchers and practitioners. However, the development and use of theories in agile research remain relatively low. While analyzing publications on agile software development in the Scopus database from the last decade, we found that only 7% of the papers used or developed a theory. This trend seems stable. However, it is promising that most theory-centric studies use or propose theories to address cognitive and behavioral aspects of people working in agile development. We argue that these aspects build fundamental pillars in agile software development. In this special section, we introduce extended versions of four papers selected from the XP2020 Conference. These papers make valuable contributions to aspects of learning and behavior in agile software development. We encourage researchers to be more theory-centric in their future empirical studies of agile methods and practices by familiarizing themselves with existing theories and applying and developing theories. This way, they can contribute to a reliable, evidence-based body of knowledge in our community.

A systematic literature review on trust in the software ecosystem

The worldwide software ecosystem is a trust-rich part of the world. Throughout the software life cycle, software engineers, end-users, and other stakeholders collaboratively place their trust in major hubs in the ecosystem, such as package managers, repository services, and software components. However, as our reliance on software grows, this trust is frequently violated by bad actors and crippling vulnerabilities in the software supply chain. This study aims to define software trust in the worldwide SECO, that is, to determine what signifies a trustworthy system, actor, or hub. We conduct a systematic literature review on the concept of trust in the software ecosystem. We acknowledge that trust is something between two actors in the software ecosystem, and we examine what role trust plays in the relationships between end-users and (1) software products, (2) package managers, (3) software producing organizations, and (4) software engineers. Two major findings emerged from the systematic literature review. To begin, we define trust in the software ecosystem by examining the definition and characteristics of trust. Second, we provide a list of trust factors that can be used to assemble an overview of software trust. Trust is critical in the communication between actors in the worldwide software ecosystem, particularly regarding software selection and evaluation. With this comprehensive overview of trust, software engineering researchers have a new foundation to understand and use trust to create a trustworthy software ecosystem.

Evaluating classifiers in SE research: the ECSER pipeline and two replication studies

ContextAutomated classifiers, often based on machine learning (ML), are increasingly used in software engineering (SE) for labelling previously unseen SE data. Researchers have proposed automated classifiers that predict if a code chunk is a clone, if a requirement is functional or non-functional, if the outcome of a test case is non-deterministic, etc.ObjectiveThe lack of guidelines for applying and reporting classification techniques for SE research leads to studies in which important research steps may be skipped, key findings might not be identified and shared, and the readers may find reported results (e.g., precision or recall above 90%) that are not a credible representation of the performance in operational contexts. The goal of this paper is to advance ML4SE research by proposing rigorous ways of conducting and reporting research.ResultsWe introduce the ECSER (Evaluating Classifiers in Software Engineering Research) pipeline, which includes a series of steps for conducting and evaluating automated classification research in SE. Then, we conduct two replication studies where we apply ECSER to recent research in requirements engineering and in software testing.ConclusionsIn addition to demonstrating the applicability of the pipeline, the replication studies demonstrate ECSER’s usefulness: not only do we confirm and strengthen some findings identified by the original authors, but we also discover additional ones. Some of these findings contradict the original ones.

Open science and Big Data in South Africa.

With the Square Kilometer Array (SKA) project and the new Multi-Purpose Reactor (MPR) soon coming on-line, South Africa and other collaborating countries in Africa will need to make the management, analysis, publication, and curation of "Big Scientific Data" a priority. In addition, the recent draft Open Science policy from the South African Department of Science and Innovation (DSI) requires both Open Access to scholarly publications and research outputs, and an Open Data policy that facilitates equal opportunity of access to research data. The policy also endorses the deposit, discovery and dissemination of data and metadata in a manner consistent with the FAIR principles - making data Findable, Accessible, Interoperable and Re-usable (FAIR). The challenge to achieve Open Science in Africa starts with open access for research publications and the provision of persistent links to the supporting data. With the deluge of research data expected from the new experimental facilities in South Africa, the problem of how to make such data FAIR takes center stage. One promising approach to make such scientific datasets more "Findable" and "Interoperable" is to rely on the Dataset representation of the Schema.org vocabulary which has been endorsed by all the major search engines. The approach adds some semantic markup to Web pages and makes scientific datasets more "Findable" by search engines. This paper does not address all aspects of the Open Science agenda but instead is focused on the management and analysis challenges of the "Big Scientific Data" that will be produced by the SKA project. The paper summarizes the role of the SKA Regional Centers (SRCs) and then discusses the goal of ensuring reproducibility for the SKA data products. Experiments at the new MPR neutron source will also have to conform to the DSI's Open Science policy. The Open Science and FAIR data practices used at the ISIS Neutron source at the Rutherford Appleton Laboratory in the UK are then briefly described. The paper concludes with some remarks about the important role of interdisciplinary teams of research software engineers, data engineers and research librarians in research data management.

For the Public Good: Connecting, Retaining, and Recognizing Current and Future RSEs at U.S. National Research Laboratories and Agencies

US national research laboratories and agencies play an integral role in advancing science and technology for the public good. The authors of this article, as Research Software Engineers (RSEs) and allies from eight unique national organizations, came together to explore RSE needs from the perspective of national research and development institutions. We identified three key areas of improvement for future RSEs to pursue science in the national interest: community establishment, hiring and retention, and recognition. To retain and cultivate this essential talent, US national institutions must evolve to support appropriate career pathways for RSEs and to recognize and reward RSE work.

Recruiting credible participants for field studies in software engineering research

Software practitioners are a primary provider of information for field studies in software engineering. Research typically recruits practitioners through some kind of sampling. But sampling may not in itself recruit the “right” participants. To assess existing guidance on participant recruitment, and to propose and illustrate a framework for recruiting professional practitioners as credible participants in field studies of software engineering. We review existing guidelines, checklists and other advisory sources on recruiting participants for field studies. We develop a framework, partly based on our prior research and on the research of others. We search for and select three exemplar studies (a case study, an interview study and a survey study) and use those to illustrate the framework. Whilst existing guidance recognises the importance of recruiting participants, there is limited guidance on how to recruit the “right” participants. The framework suggests the conceptualisation of participants as “research instruments” or, alternatively, as a sampling frame for items of interest. The exemplars suggest that at least some members of the research community are aware of the need to carefully recruit the “right” participants. The framework is intended to encourage researchers to think differently about the involvement of practitioners in field studies of software engineering. Also, the framework identifies a number of characteristics not explicitly addressed by existing guidelines.

Research Software Engineers: Career Entry Points and Training Gaps

Research Software Engineers: Career Entry Points and Training Gaps

Research Software Science: Expanding the Impact of Research Software Engineering

Software plays a central role in scientific discovery. Improving how we develop and use software for research can have both broad and deep impacts on a spectrum of challenges and opportunities society faces today. The emergence of the Research Software Engineer (RSE) as a role correlates with the growing complexity of scientific challenges and the diversity of software team skills. In this paper, we describe research software science (RSS), an idea related to RSE, and particularly suited to research software teams. RSS promotes the use of scientific methodologies to explore and establish broadly applicable knowledge. Using RSS, we can pursue sustainable, repeatable, and reproducible software improvements that positively impact research software toward improved scientific discovery.

What Makes Good Research in Software Engineering

What Makes Good Research in Software Engineering

Game Development Topics: A Tag-Based Investigation on Game Development Stack Exchange

Video-game development, despite being a multi-billion-dollar industry, has not attracted sustained attention from software engineering researchers and remains understudied from a software engineering perspective. We aim to uncover, from game developers’ perspectives, which video game development topics are the most asked about and which are the most supported, in order to provide insights about technological and conceptual challenges game developers and managers may face on their projects. To do so, we turned to the Game Development Stack Exchange (GDSE), a prominent Question and Answer forum dedicated to game development. On that forum, users ask questions and tag them with keywords recognized as important categories by the community. Our study relies on those tags, which we classify either as technology or concept topics. We then analysed these topics for their levels of community attention (number of questions, views, upvotes, etc.) and community support (whether their questions are answered and how long it takes). Related to community attention, we found that topics with the most questions include concepts such as 2D and collision detection and technologies such as Unity and C#, whereas questions touching on concepts such as video and augmented reality and technologies such as iOS, Unreal-4 and Three.js generally lack satisfactory answers. Moreover, by pairing topics, we uncovered early clues that, from a community support perspective, (i) the pairing of some technologies appear more challenging (e.g., questions mixing HLSL and MonoGame receive a relatively lower level of support); (ii) some concepts may be more difficult to handle conjointly (e.g., rotation and movement); and some technologies may prove more challenging to use to address a given concept (e.g., Java for 3D). Our findings provide insights to video game developers on the topics and challenges they might encounter and highlight tool selection and integration for video game development as a promising research direction.

The Pandora's box of social, process, and people debts in software engineering

AbstractIn software engineering, technical debt (TD) has been widely investigated, but debt regarding social issues, people, and processes has not been explored as much. It should be noted here that we use nontechnical debt (NTD) as an umbrella term to cover social, process, and people debts. Although the number of studies on NTD in software is increasing, the majority of them are descriptive rather than rigorous, and there is no systematic development of cumulative knowledge. As a result, identifying the fundamental causes of NTD and the associated mitigation techniques in software engineering is challenging. Therefore, this study investigates the scientific evidence regarding NTD till date by conducting a systematic mapping review of software engineering research between January 2000 and October 2021. The search strategy resulted in 175 studies, 17 of which were identified as unique and relevant primary papers. The primary studies show that NTD and TD are inextricably linked. In addition, this study also captured a plethora of causes and mitigation strategies for managing NTD and thus makes four important contributions: (i) highlighting state‐of‐the‐art NTD research; (ii) identification of the reported causes and mitigation strategies in the primary papers; and (iii) determination of opportunities for future NTD research.

Pitfalls and guidelines for using time-based Git data

Many software engineering research papers rely on time-based data (e.g., commit timestamps, issue report creation/update/close dates, release dates). Like most real-world data however, time-based data is often dirty. To date, there are no studies that quantify how frequently such data is used by the software engineering research community, or investigate sources of and quantify how often such data is dirty. Depending on the research task and method used, including such dirty data could affect the research results. This paper presents an extended survey of papers that utilize time-based data, published in the Mining Software Repositories (MSR) conference series. Out of the 754 technical track and data papers published in MSR 2004–2021, we saw at least 290 (38%) papers utilized time-based data. We also observed that most time-based data used in research papers comes in the form of Git commits, often from GitHub. Based on those results, we then used the Boa and Software Heritage infrastructures to help identify and quantify several sources of dirty Git timestamp data. Finally we provide guidelines/best practices for researchers utilizing time-based data from Git repositories.

Early career software developers and work preferences in software engineering

AbstractContext: The software engineering researchers and practitioners echoed the need for investigations to better understand the engineers developing software and services. In light of current studies, there are significant associations between the personalities of software engineers and their work preferences. However, limited studies are using psychometric measurements in software engineering.Objective: We aim to evaluate attitudes of early‐stage software engineers and investigate link between their personalities and work preferences.Method: We collected extensive psychometric data from 303 graduate‐level students in Computer Science programs at four Pakistani and one Swedish university using Five‐Factor Model. The statistical analysis investigated associations between various personality traits and work preferences.Results: The data support the existence of two clusters of software engineers, one of which is more highly rated across the board. Numerous correlations exist between personality qualities and the preferred types of employment for software developers. For instance, those who exhibit greater levels of emotional stability, agreeableness, extroversion, and conscientiousness like working on technical activities on a set timetable. Similar relationships between personalities and occupational choices are also evident in the earlier studies. More neuroticism is reported in female respondents than in male respondents. Higher intelligence was demonstrated by those who worked on the “entire development process” and “technical components of the project.”Conclusion: When assigning project tasks to software engineers, managers might use the statistically significant relationships that emerged from the analysis of personality attributes. It would be beneficial to construct effective teams by taking personality factors like extraversion and agreeableness into consideration. The study techniques and analytical tools we use may identify subtle relationships and reflect distinctions across various groups and populations, making them valuable resources for both future academic research and industrial practice.

Socio-Technical Grounded Theory for Software Engineering

Grounded Theory (GT), a sociological research method designed to study social phenomena, is increasingly being used to investigate the human and social aspects of software engineering (SE). However, being written by and for sociologists, GT is often challenging for a majority of SE researchers to understand and apply. Additionally, SE researchers attempting ad hoc adaptations of traditional GT guidelines for modern socio-technical (ST) contexts often struggle in the absence of clear and relevant guidelines to do so, resulting in poor quality studies. To overcome these research community challenges and leverage modern research opportunities, this paper presents <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Socio-Technical Grounded Theory</i> (STGT) designed to ease application and achieve quality outcomes. It defines what exactly is meant by an ST research context and presents the STGT guidelines that expand GT's philosophical foundations, provide increased clarity and flexibility in its methodological steps and procedures, define possible scope and contexts of application, encourage frequent reporting of a variety of interim, preliminary, and mature outcomes, and introduce nuanced evaluation guidelines for different outcomes. It is hoped that the SE research community and related ST disciplines such as computer science, data science, artificial intelligence, information systems, human computer/robot/AI interaction, human-centered emerging technologies (and increasingly other disciplines being transformed by rapid digitalisation and AI-based augmentation), will benefit from applying STGT to conduct quality research studies and systematically produce rich findings and mature theories with confidence.

Static test flakiness prediction: How Far Can We Go?

Test flakiness is a phenomenon occurring when a test case is non-deterministic and exhibits both a passing and failing behavior when run against the same code. Over the last years, the problem has been closely investigated by researchers and practitioners, who all have shown its relevance in practice. The software engineering research community has been working toward defining approaches for detecting and addressing test flakiness. Despite being quite accurate, most of these approaches rely on expensive dynamic steps, e.g., the computation of code coverage information. Consequently, they might suffer from scalability issues that possibly preclude their practical use. This limitation has been recently targeted through machine learning solutions that could predict the flakiness of tests using various features, like source code vocabulary or a mixture of static and dynamic metrics computed on individual snapshots of the system. In this paper, we aim to perform a step forward and predict test flakiness only using static metrics. We propose a large-scale experiment on 70 Java projects coming from the iDFlakies and FlakeFlagger datasets. First, we statistically assess the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells, analyzing both their individual and combined effects. Based on the results achieved, we experiment with a machine learning approach that predicts test flakiness solely based on static features, comparing it with two state-of-the-art approaches. The key results of the study show that the static approach has performance comparable to those of the baselines. In addition, we found that the characteristics of the production code might impact the performance of the flaky test prediction models.

Software Engineering After the COVID-19 Outbreak

Software Engineering (SE) has evolved over many decades and has led to many proven and well-established methods and tools that support the efficient development of software and IT systems in general. Although software development had often been performed by distributed teams even before the pandemic, the COVID-19 outbreak exacerbated the physical separation of teams, creating numerous challenges and issues for both organizations and developers. Therefore, in this new column "Software Engineering After the COVID-19 Outbreak", we aim at providing insights into the impact of the COVID-19 outbreak on the work of Software Engineering practitioners and researchers. We intend to collect past and current challenges, changes in structures and organizations, reports of experiences and, of course, proven solutions for solving problems that have arisen.

A decade of code comment quality assessment: A systematic literature review

Code comments are important artifacts in software systems and play a paramount role in many software engineering (SE) tasks related to maintenance and program comprehension. However, while it is widely accepted that high quality matters in code comments just as it matters in source code, assessing comment quality in practice is still an open problem. First and foremost, there is no unique definition of quality when it comes to evaluating code comments. The few existing studies on this topic rather focus on specific attributes of quality that can be easily quantified and measured. Existing techniques and corresponding tools may also focus on comments bound to a specific programming language, and may only deal with comments with specific scopes and clear goals (e.g., Javadoc comments at the method level, or in-body comments describing TODOs to be addressed).In this paper, we present a Systematic Literature Review (SLR) of the last decade of research in SE to answer the following research questions: (i) What types of comments do researchers focus on when assessing comment quality? (ii) What quality attributes (QAs) do they consider? (iii) Which tools and techniques do they use to assess comment quality?, and (iv) How do they evaluate their studies on comment quality assessment in general?Our evaluation, based on the analysis of 2353 papers and the actual review of 47 relevant ones, shows that (i) most studies and techniques focus on comments in Java code, thus may not be generalizable to other languages, and (ii) the analyzed studies focus on four main QAs of a total of 21 QAs identified in the literature, with a clear predominance of checking consistency between comments and the code. We observe that researchers rely on manual assessment and specific heuristics rather than the automated assessment of the comment quality attributes, with evaluations often involving surveys of students and the authors of the original studies but rarely professional developers.

The human experience of comprehending source code in virtual reality.

Virtual reality (VR) is an emerging technology used in various domains such as medicine, psychotherapy, architecture, and gaming. Recently, software engineering researchers have started to explore virtual reality as a tool for programmers. However, few studies examine source code comprehension in VR. This paper explores the human experience of comprehending source code in VR and compares it to source code comprehension in a desktop environment. We conducted a study with 26 graduate student programmers. We measured actual productivity, perceived productivity and used the NASA Task Load Index (TLX) survey to measure various factors such as mental demand, physical demand, temporal demand, performance, effort, and frustration. We found that the programmers experienced more physical demand, effort, and overall task load when reading and comprehending code in VR. However, we did not observe any statistically significant differences in the programmers’ measured productivity or perceived productivity between VR and desktop comprehension.

Philanthropic conference-based requirements engineering in time of pandemic and beyond.

As software engineering (SE) practitioners, we can help society by using our communities of experts to address a software need of a socially conscious organization. Doing so can benefit society in the locale of a SE conference and provide access to international experts for local organizations. Furthermore, established SE researchers as well as practitioners and students have the opportunity for a unique learning experience. While the SE community has already realized the importance of addressing human values and promoting social good objectives in software development, we are unaware of previous attempts to leverage SE conferences for this activity. Conferences present an opportunity to enjoy the assembly of SE practitioners, researchers, and students for the purpose of a philanthropic endeavor. Over the past four years of running a “Requirements Engineering for Social Good” event called RE Cares, co-located with the International Conference on Requirements Engineering, we worked with the stakeholders local to the conference venue. We selected stakeholders who would not necessarily have ready access to requirements engineering, software design, and development expertise otherwise, to build software targeting “good causes.” In the last two years, this event was altered to adapt to the constraints induced by COVID-19, moving to a hybrid mode and changing many of its practices accordingly. This paper summarizes and generalizes our experiences, discussing our lessons learned in the context of the pandemic and beyond and providing a framework for conducting similar social contribution in any SE conferences in general.

Time Pressure's Impact on Software Quality

In the software industry, time pressure is unavoidable, and it has an impact on software developers. It could aching programming quality or predisposition impression of execution on got done with responsibilities, making them contrast from actual implementation. Time pressure is frequent in the software industry, where shorter and shorter deadlines and high client demands lead to more tight deadlines. On the other hand, the effects of time constraints have received little attention in software engineering research. This study aims to survey final-year undergraduate students studying software engineering and computer science at Pakistani universities. We select the quality factors from the software quality model, i.e., McCall's classic model – 1977 and FRUP+ – 1992. This study considers these factors: performance, productivity, accuracy, creativity, reliability, efficiency, reusability, interoperability, flexibility, maintainability, integrity, testability, and portability. All selected factors for this study are affected by the time pressure software completed on time and delivered. Still, the quality of the software system is affected and sometimes cause error and issues.