Software Engineering Experiments Research Articles

Women and Software Engineering

My goal with this text is to provide some concepts and questions that can help the reader to reflect on and elevate the discussion about Women and Software Engineering by providing historical data and some reflection points for the future. Feminism constitutes both a theoretical perspective and a social movement aiming to diminish and ultimately eliminate gender-based inequality and oppression. Data feminism merges data science with feminist principles to examine and address biases and power dynamics in data and technology. Studies at the intersection between gender and software engineering cover gender representation, barriers, and experiences. The main questions I propose for future reflections in the community are: What are the benefits that women bring to software engineering? How does the career and the life of female software engineers unfold? How should software engineering research change so that feminist principles are incorporated? How can feminist knowledge and processes help to examine power structure in software engineering?

Hyperspectral confocal imaging for high-throughput readout and analysis of bio-integrated microlasers.

Integrating micro- and nanolasers into live cells, tissue cultures and small animals is an emerging and rapidly evolving technique that offers noninvasive interrogation and labeling with unprecedented information density. The bright and distinct spectra of such lasers make this approach particularly attractive for high-throughput applications requiring single-cell specificity, such as multiplexed cell tracking and intracellular biosensing. The implementation of these applications requires high-resolution, high-speed spectral readout and advanced analysis routines, which leads to unique technical challenges. Here, we present a modular approach consisting of two separate procedures. The first procedure instructs users on how to efficiently integrate different types of lasers into living cells, and the second procedure presents a workflow for obtaining intracellular lasing spectra with high spectral resolution and up to 125-kHz readout rate and starts from the construction of a custom hyperspectral confocal microscope. We provide guidance on running hyperspectral imaging routines for various experimental designs and recommend specific workflows for processing the resulting large data sets along with an open-source Python library of functions covering the analysis pipeline. We illustrate three applications including the rapid, large-volume mapping of absolute refractive index by using polystyrene microbead lasers, the intracellular sensing of cardiac contractility with polystyrene microbead lasers and long-term cell tracking by using semiconductor nanodisk lasers. Our sample preparation and imaging procedures require 2 days, and setting up the hyperspectral confocal microscope for microlaser characterization requires <2 weeks to complete for users with limited experience in optical and software engineering.

Fostering Collaboration and Advancing Research in Software Engineering and Game Development for Serious Contexts

The potential benefits of using the engaging and interactive nature of games to achieve specific objectives have been recognized by researchers and professionals from numerous domains. Serious games have been developed to impart knowledge, skills, and awareness in areas such as education, healthcare and the environment, while gamification has been applied to enhance the engagement, motivation, and participation of users in non-game activities such as sustainability and learning. As a result, the fields of game engineering, software engineering, and user experience are increasingly converging to create innovative solutions that blend the strengths of games with real-world applications.

Embedding Software Engineering in Mixed Methods Research: Computationally Enhanced Risk Communication

Mixed methods research ameliorates many convergent research challenges within the contemporary sociotechnical landscape. We suggest the integration of software engineering in mixed methods research is a critical step to address some of the remaining and persistent challenges. One such research challenge where computational science is particularly well suited is in hazard preparedness—in particular, the creation of risk communication messages to mitigate or to prevent harm. Computationally enhanced risk communication is convergent research that integrates software engineering and social science research for the benefit of protecting humans and infrastructure. To this end, we developed a mixed methods framework for the efficient construction of risk communication messages. We call this the Domain Agnostic Risk Communication (DARC) framework and present it here. The DARC framework formalizes connections between computational science and social science methods. It incorporates the best available science in risk communication research and a cadre of natural language processing techniques to impart validity, reliability, and precision into resultant messages. The DARC framework is highly modular owing to the incorporation of the software engineering principles of abstraction, extensibility, and encapsulation. Although the focus of this position article is on risk communication, we encourage the incorporation of software engineering into mixed methods research and the incorporation of mixed methods more broadly into software engineering experimentation.

Sentiment overflow in the testing stack: Analyzing software testing posts on Stack Overflow

Software testing is an integral part of modern software engineering practice. Past research has not only underlined its significance, but also revealed its multi-faceted nature. The practice of software testing and its adoption is influenced by many factors that go beyond tools or technology. This paper sets out to investigate the context of software testing from the practitioners’ point of view by mining and analyzing sentimental posts on the widely used question and answer website Stack Overflow. By qualitatively analyzing sentimental expressions of practitioners, which we extract from the Stack Overflow dataset using sentiment analysis tools, we discern factors that help us to better understand the lived experience of software engineers with regards to software testing. Grounded in the data that we have analyzed, we argue that sentiments like insecurity, despair and aspiration, have an impact on practitioners’ attitude towards testing. We suggest that they are connected to concrete factors like the level of complexity of projects in which software testing is practiced.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Characterization of continuous experimentation in software engineering: Expressions, models, and strategies

Continuous Experimentation (CE) has become increasingly popular across industry and academic communities. Major software organizations use CE to increase their revenue by adding value to end-users, and researchers are investigating the CE adoption process and usage to expand its success. Given this rapid evolution, observing a shared understanding of CE definitions, processes, and experiment strategies is difficult, potentially jeopardizing new implementations and focused research efforts. To characterize CE from the perspective of its definitions, processes, and strategies for experimentation available in the technical literature and to evolve the understanding perspectives for “continuous experimentation” and “data-driven development” definitions. To select and analyze sources of information in the technical literature dealing with different aspects of continuous experimentation through a Literature Study using an ad hoc search improved with snowballing (backward and forward). Organize the findings into new perspectives for CE definitions, processes, and experiment strategies. It was possible to identify many different definitions, processes, and experimental strategies used to describe CE in the 72 analyzed empirical papers, making it difficult to decide on their combination to be applied in a real software development project. Therefore, it has been proposed to evolve the CE understanding perspective, to categorize its experiment strategies, and to offer a combined development process for CE combining parts of other processes. Besides, conjectural requirements have been identified, which can contribute to better differentiating requirements and hypotheses in the CE context. Likely, a better understanding of CE is still missing. It can contribute towards organizing a common taxonomy to facilitate the possible choices for the experiment strategies. Therefore, there is space for more investigations on its applicability and value in different categories of software systems, despite all the advancements of CE and its promotion in developing modern software systems.

Discussing early innovation software by example of the PERPL software

Discussing early innovation software by example of the PERPL software Dr Joanna Leng, School of Computing, University of Leeds, UK, presents a project from the fellowship, on the PERPL (pattern extraction from relative positions of localisations) software which analyses super-resolution light microscopy (SRLM) data. Alistair developed the initial capabilities of PERPL with knowledge of the Python programming language. Still, with limited experience in software engineering, he realised it was becoming difficult to extend and for others to use. As the software and the underlying science were complex, the best way forward was for Alistair and me to work together. However, this close collaboration involving teaching software engineering skills is not a commonly recognised RSE practice.

Software Development Practice as Baradian Entanglement

Software development practice is a messy, complicated, and constantly shifting human endeavor. Barad’s concept of “entanglement” helps to theorize complex sociotechnical systems. We are testing the application of this theory to understand and explain software development practices, as our work appears to be the only ethnographic research using Barad in any technology industry. Our continual aim is to understand large-scale, collaborative software development more deeply in practice and to discover appropriate theories that describe our observations and insights. Both authors are experienced software engineers and researchers. Through an ongoing longitudinal ethnographic study at a large Australian software development company, we explore, support, and improve the lived experience and practice of the software developers that work there. Ethnographic insights and an appreciation of the mutual constitution of situated phenomena have expanded over several years into an elaboration of entanglement as a more insightful explanation of software development practice. This research is having a significant impact on the participant developers and organization, including changes in measurement practices, mentoring, knowledge management, and innovation.

Trustworthy agent-based simulation: the case for domain-specific modelling languages

Simulation is a key tool for researching complex system behaviour. Agent-based simulation has been applied across domains, such as biology, health, economics and urban sciences. However, engineering robust, efficient, maintainable, and reliable agent-based simulations is challenging. We present a vision for engineering agent simulations comprising a family of domain-specific modelling languages (DSMLs) that integrates core software engineering, validation and simulation experimentation. We relate the vision to examples of principled simulation, to show how the DSMLs would improve robustness, efficiency, and maintainability of simulations. Focusing on how to demonstrate the fitness for purpose of a simulator, the envisaged approach supports bi-directional transparency and traceability between the original domain understanding to the implementation, interpretation of results and evaluation of hypotheses.

A Systematic Mapping Study of Empirical Studies Performed with Collections of Software Projects

Context: software projects are commonresources in Software Engineering experiments,although these are often selected without following a specific strategy, which reduces the representativeness and replication of the results. An option is the use of preserved collections of software projects, but these must be current, with explicit guidelines that guarante etheir updating over a long period of time. Goal: to carry out a systematic secondary study about the strategies to select software projects in empirical studies to discover the guidelines taken into account, the degree of use of project collections, the meta-data extracted and the subsequent statistical analysis conducted. Method: A systematic mapping study to identify studies published from January 2013 to December 2020. Results: 122 studies were identified, of which the 72% used their own guidelines for project selection and the 27% used existent project collections. Likewise, there was no evidence of a standardized framework for the project selection process, nor the application of statistical methods that relates with the sample collection strategy.

Towards a model and methodology for evaluating data quality in software engineering experiments

Data collected during software engineering experiments might contain quality problems, leading to wrong experimental conclusions.

A Survey of Controlled Experiments in Software Engineering

A Survey of Controlled Experiments in Software Engineering

On researcher bias in Software Engineering experiments

Researcher bias occurs when researchers influence the results of an empirical study based on their expectations, either consciously or unconsciously. Researcher bias might be due to the use of Questionable Research Practices (QRPs). In research fields like medicine, blinding techniques have been applied to counteract researcher bias. In this paper, we present two studies to increase our body of knowledge on researcher bias in Software Engineering (SE) experiments, including: (i) QRPs potentially leading to researcher bias; (ii) causes behind researcher bias; and (iii) possible actions to counteract researcher bias with a focus on, but not limited to, blinding techniques. The former is an interview study, intended as an exploratory study, with nine experts of the empirical SE community. The latter is a quantitative survey with 51 respondents, who were experts of the above-mentioned community. The findings from the exploratory study represented the starting point to design the survey. In particular, we defined the questionnaire of this survey to support the findings from the exploratory study. From the interview study, it emerged that some QRPs (e.g., post-hoc outlier criteria) are acceptable in certain cases. Also, it appears that researcher bias is perceived in SE and, to counteract researcher bias, a number of solutions have been highlighted. For example, duplicating the data analysis in SE experiments or fostering open data policies in SE conferences/journals. The findings from the interview study are mostly confirmed by those from the survey, and allowed us to delineate recommendations to counteract researcher bias in SE experiments. Some recommendations are intended for SE researchers, while others are purposeful for the boards of SE research venues.

A study into the practice of reporting software engineering experiments

It has been argued that reporting software engineering experiments in a standardized way helps researchers find relevant information, understand how experiments were conducted and assess the validity of their results. Various guidelines have been proposed specifically for software engineering experiments. The benefits of such guidelines have often been emphasized, but the actual uptake and practice of reporting have not yet been investigated since the introduction of many of the more recent guidelines. In this research, we utilize a mixed-method study design including sequence analysis techniques for evaluating to which extent papers follow such guidelines. Our study focuses on the four most prominent software engineering journals and the time period from 2000 to 2020. Our results show that many experimental papers miss information suggested by guidelines, that no de facto standard sequence for reporting exists, and that many papers do not cite any guidelines. We discuss these findings and implications for the discipline of experimental software engineering focusing on the review process and the potential to refine and extend guidelines, among others, to account for theory explicitly.

The Importance of the Correlation in Crossover Experiments

<i>Context:</i> In empirical software engineering, crossover designs are popular for experiments comparing software engineering techniques that must be undertaken by human participants. However, their value depends on the correlation ( <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> ) between the outcome measures on the same participants. Software engineering theory emphasizes the importance of individual skill differences, so we would expect the values of <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> to be relatively high. However, few researchers have reported the values of <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> . <i>Goal:</i> To investigate the values of <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> found in software engineering experiments. <i>Method:</i> We undertook simulation studies to investigate the theoretical and empirical properties of <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> . Then we investigated the values of <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> observed in 35 software engineering crossover experiments. <i>Results:</i> The level of <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> obtained by analysing our 35 crossover experiments was small. Estimates based on means, medians, and random effect analysis disagreed but were all between 0.2 and 0.3. As expected, our analyses found large variability among the individual <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> estimates for small sample sizes, but no indication that <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> estimates were larger for the experiments with larger sample sizes that exhibited smaller variability. <i>Conclusions:</i> Low observed <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> values cast doubts on the validity of crossover designs for software engineering experiments. However, if the cause of low <inline-formula><tex-math notation="LaTeX">$r$</tex-math></inline-formula> values relates to training limitations or toy tasks, this affects <i>all</i> Software Engineering (SE) experiments involving human participants. For all human-intensive SE experiments, we recommend more intensive training and then tracking the improvement of participants as they practice using specific techniques, before formally testing the effectiveness of the techniques.

Developing a Theory-Informed Smartphone App for Early Psychosis: Learning Points From a Multidisciplinary Collaboration.

Background: Actissist is a smartphone app designed to deliver an intervention grounded in cognitive behavior therapy for early psychosis. Actissist was developed by a multidisciplinary team of academics, clinicians, experts by experience and software engineers. Actissist has been tested in two trials, the first a proof-of-concept trial where Actissist was safe, acceptable and feasible, the second, a powered randomized controlled trial.Objective: This article describes how our multidisciplinary team designed and developed Actissist. This article describes: (i) how Actissist was informed by initial qualitative interviews and focus groups and an expert reference group; (ii) refinements made to the app based on ongoing user feedback; (iii) successes and challenges encountered; and (iv) learning points and recommendations for involving stakeholders in digital health interventions.Methods: Expert reference group meetings informed the development of Actissist and design of subsequent trials, which included individuals with lived experience of psychosis, clinicians, academics, computer scientists and software engineers. Person-centered stakeholder involvement was promoted using focus groups and qualitative interviews prior to the development of the app, which informed version one of Actissist. Interviews were carried out with participants who had used Actissist. Two further versions of Actissist were developed following additional rounds of testing.Results: Multidisciplinary working throughout the Actissist project led to the development, inclusion and improvement of the app design and content. These changes and features included non-directive and compassionate content, co-designed recovery videos, relaxation exercises, psychoeducation material, ability to “favorite” areas of the app that users found helpful, and goal-setting. Challenges to collaborative working included discrepancies between what stakeholders want and what is technically possible to deliver, resource pressures, trying to deliver desired features within the boundaries of fundamental trial design considerations, and power imbalances associated with multidisciplinary working.Conclusions: The involvement of stakeholders in the design and development and delivery of Actissist has been fundamental to our development approach. Through this collaborative process, we have identified different perspectives and ideas that would have not been generated by the research team alone.Clinical Trial Registrations: Proof-of-concept trial: http://www.isrctn.com/ISRCTN34966555Fully-powered randomized controlled trial: https://www.isrctn.com/ISRCTN76986679

Design and Implementation of Welding Procedure Qualification Expert System

Welding process evaluation is an important link to ensure the quality of welding engineering and determine the welding process specification. It is also the main procedure to evaluate the welder’s examination results. In the traditional way of processing WPS documents, the test results are usually judged manually by experts based on experience. However, the evaluation of welding process is often very complex, involving the base material, sample type, process parameters and a large number of international standards. Therefore, how to utilize the software engineering method and experience, make the welding process evaluation program accurate and efficient information, make the welding process evaluation program easier and more efficient, is very important to the whole field of welding.

Understanding Hypotheses Engineering in Software Startups through a Gray Literature Review

Abstract Context The higher availability of software usage data and the influence of the Lean Startup led to the rise of experimentation in software engineering, a new approach for development based on experiments to understand the user needs. In the models proposed to guide this approach, the first step is generally to identify, prioritize, and specify the hypotheses that will be tested through experimentation. However, although practitioners have proposed several techniques to handle hypotheses, the scientific literature is still scarce. Objective The goal of this study is to understand what activities, as proposed in industry, are entailed to handle hypotheses, facilitating the comparison, creation, and evaluation of relevant techniques. Methods We performed a gray literature review (GLR) on the practices proposed by practitioners to handle hypotheses in the context of software startups. We analyzed the identified documents using thematic synthesis. Results The analysis revealed that techniques proposed for software startups in practice compress five different activities: elicitation, prioritization, specification, analysis, and management. It also showed that practitioners often classify hypotheses in types and which qualities they aim for these statements. Conclusion Our results represent the first description for hypotheses engineering grounded in practice data. This mapping of the state-of-practice indicates how research could go forward in investigating hypotheses for experimentation in the context of software startups. For practitioners, they represent a catalog of available practices to be used in this context.

An Expert Judgment in Source Code Quality Research Domain—A Comparative Study between Professionals and Students

In scientific research, evidence is often based on empirical data. Scholars tend to rely on students as participants in experiments in order to validate their thesis. They are an obvious choice when it comes to scientific research: They are usually willing to participate and are often themselves pursuing an education in the experiment’s domain. The software engineering domain is no exception. However, readers, authors, and reviewers do sometimes question the validity of experimental data that is gathered in controlled experiments from students. This is why we will address this difficult-to-answer question: Are students a proper substitute for experienced professional engineers while performing experiments in a typical software engineering experiment. As we demonstrate in this paper, it is not a “yes or no” answer. In some aspects, students were not outperformed by professionals, but in others, students would not only give different answers compared to professionals, but their answers would also diverge. In this paper we will show and analyze the results of a controlled experiment in the source code quality domain in terms of comparing student and professional responses. We will show that authors have to be careful when employing students in experiments, especially when complex and advanced domains are addressed. However, they may be a proper substitution in cases, where non-advanced aspects are required.

ПІДГОТОВКА МАЙБУТНІХ ІНЖЕНЕРІВ-ПРОГРАМІСТІВ ДО ТЕСТУВАННЯ ПРОГРАМНОГО ЗАБЕЗПЕЧЕННЯ В УМОВАХ НЕФОРМАЛЬНОЇ ОСВІТИ: ПРОБЛЕМИ ТА ШЛЯХИ УСПІШНОЇ РЕАЛІЗАЦІЇ

This article discusses the problem of preparing student programmers for the process of software testing in real production conditions. It reveals the relevance of the field of software development in general and the testing process in particular, gives a characteristic of the latter. An analysis of the requirements put forward by employers to young specialists-testers is carried out, the main ones are highlighted: knowledge of specific theoretical aspects of activities, the availability of project experience, a high level of development of soft skills. The paper examines the international experience of training software engineers, which highlights special attention of foreign programs to the practices of non-formal education, in particular, to conducting industrial practice at the expense of the resources and programs of partner enterprises, as well as taking into account international certificates of training programs in the assessment of disciplines. In the final part of the work, a description of the concept of non-formal education is given, features of its legal status in Ukraine are highlighted, characteristics of its main features are given, such as the use of innovative teaching methods, flexibility and modifiability, implementation on the basis of specialized or independent institutions. The final part of the article examines the issue of the relevance of non-formal forms of education in the context of training future software engineers for software testing in a real project environment. An analysis of the existing commercial practices present in the educational space of Ukraine is carried out, conclusions are drawn regarding the possibility of using informal training practices within the formal programs of domestic universities.