Exploring the Essence of Continuous Experimentation

Abstract

Context: Today's software industry is faced with rapidly changing requirements and high expectations from customers and users. Agile software development with a focus on rapid and frequent product deployment to end-users calls for equally continuous feedback acquisition, guiding software design and supporting evidencebased product development decisions. For keeping pace and succeeding on the highly competitive software market it is essential for companies to quickly and reliably evaluate their ideas based on real usage experiences in a natural context. This can be enabled using Continuous Experimentation (CE) - an approach to close the feedback cycle and tightly integrate the empirical evaluation of new features into the software development process. Objectives: With this project I aim at creating an understanding of CE as practiced and theorised today. Furthermore, my goal is to identify the smallest set of prerequisites for successfully implementing systematic experimentation practices into companies' software development process. I want to find solutions for companies to adopt CE as most suitable to their product stage, process maturity, and available resources; and enable organisations to systematically run experiments for data-driven decision-making. Methods: In addition to a currently on-going multivocal literature review, the project will involve empirical studies following a design science approach. Starting with a practitioner survey or interview, and a pilot case study for finding both, the typical hurdles encountered and the minimum requirements needed to apply CE in any company, the project will transition into an iterative multiple-case study, where proposed solutions are to be evaluated. Conclusions: The proposed doctoral research project will contribute to an increased understanding of systematic experimentation practices in different organisational settings, and strives for providing actionable solutions for enabling more flexible software development which can quickly react to feedback gained through experimentation, as well as an evaluation of these suggested solutions. Especially, this project aims at the creation of an easy-to-adopt approach called light-weight CE, which allows also companies with few resources or projects involving an earlystage product to benefit from the advantages of CE.