Software-intensive Products Research Articles

Adversarial Machine Learning in Industry: A Systematic Literature Review

Adversarial Machine Learning (AML) discusses the act of attacking and defending Machine Learning (ML) Models, an essential building block of Artificial Intelligence (AI). ML is applied in many software-intensive products and services and introduces new opportunities and security challenges. AI and ML will gain even more attention from the industry in the future, but threats caused by already-discovered attacks specifically targeting ML models are either overseen, ignored, or mishandled. Current AML research investigates attack and defense scenarios for ML in different industrial settings with a varying degree of maturity with regard to academic rigor and practical relevance. However, to the best of our knowledge, a synthesis of the state of academic rigor and practical relevance is missing. This literature study reviews studies in the area of AML in the context of industry, measuring and analyzing each study’s rigor and relevance scores. Overall, all studies scored a high rigor score and a low relevance score, indicating that the studies are thoroughly designed and documented but miss the opportunity to include touch points relatable for practitioners.

SmartDelta project: Automated quality assurance and optimization across product versions and variants

Software systems are often built in increments with additional features or enhancements on top of existing products. This incremental development may result in the deterioration of certain quality aspects. In other words, the software can be considered an evolving entity emanating different quality characteristics as it gets updated over time with new features or deployed in different operational environments. Approaching software development with this mindset and awareness regarding quality evolution over time can be a key factor for the long-term success of a company in today’s highly competitive market of industrial software-intensive products. Therefore, it is important to be able to accurately analyze and determine the quality implications of each change and increment to a software system. To address this challenge, the multinational SmartDelta project develops automated solutions for the quality assessment of product deltas in a continuous engineering environment. The project provides smart analytics from development artifacts and system executions, offering insights into quality degradation or improvements across different product versions, and providing recommendations for the next builds. This paper presents the challenges in incremental software development tackled in the scope of the SmartDelta project, and the solutions that are produced and planned in the project, along with the industrial impact of the project for software-intensive industrial systems.

A taxonomy of assets for the development of software-intensive products and services

Context:Developing software-intensive products or services usually involves a plethora of software artefacts. Assets are artefacts intended to be used more than once and have value for organisations; examples include test cases, code, requirements, and documentation. During the development process, assets might degrade, affecting the effectiveness and efficiency of the development process. Therefore, assets are an investment that requires continuous management.Identifying assets is the first step for their effective management. However, there is a lack of awareness of what assets and types of assets are common in software-developing organisations. Most types of assets are understudied, and their state of quality and how they degrade over time have not been well-understood. Methods:We performed an analysis of secondary literature and a field study at five companies to investigate and identify assets to fill the gap in research. The results were analysed qualitatively and summarised in a taxonomy. Results:We present the first comprehensive, structured, yet extendable taxonomy of assets, containing 57 types of assets. Conclusions:The taxonomy serves as a foundation for identifying assets that are relevant for an organisation and enables the study of asset management and asset degradation concepts.

HyMap: Eliciting hypotheses in early-stage software startups using cognitive mapping

Context: Software startups develop innovative, software-intensive products. Given the uncertainty associated with such an innovative context, experimentation is a valuable approach for these companies, especially in the early stages of the development, when implementing unnecessary features represents a higher risk for companies' survival. Nevertheless, researchers have argued that the lack of clearly defined practices led to limited adoption of experimentation. In this regard, the first step is to define the hypotheses based on which teams will create experiments. Objective: We aim to develop a systematic technique to identify hypotheses for early-stage software startups. Methods: We followed a Design Science approach consisted of three cycles in the construction phase, that involved seven startups in total, and an evaluation of the final artifact within three startups. Results: We developed the HyMap, a hypotheses elicitation technique based on cognitive mapping. It consists of a visual language to depict a cognitive map representing the founder's understanding of the product, and a process to elicit this map consisted of a series of questions the founder must answer. Our evaluation showed that the artifacts are clear, easy to use, and useful leading to hypotheses and facilitating founders to visualize their idea. Conclusion: Our study contributes to both descriptive and prescriptive bodies of knowledge. Regarding the first, it provides a better understanding of the guidance founders use to develop their startups and, for the latter, a technique to identify hypotheses in early-stage software startups.

A Taxonomy of Assets for the Development of Software-Intensive Products and Services

Context: Developing software-intensive products or services involves utilising many artefacts that are either part of the offering or part of enabling their development. These artefacts, if valuable and used more than once by the development organisation, can be seen as assets such as test cases, code, requirements, and documentation. As such, assets are often reused, updated, and become a base or even necessity for product development and evolution over time, constituting an investment. Assets are not well understood beyond code-based ones in the area of technical debt. Thus most asset types are often overlooked, and their state, which is subject to degradation over time, has not been much studied. Method: To address the problem, we conducted industrial focus groups with five companies and a literature review. The results were analysed qualitatively and summarised in a taxonomy. Results: We created a structured, extendable taxonomy of assets. The taxonomy contains 57 assets. Conclusions: The taxonomy serves as foundations for defining and identifying assets as a concept. It provides the foundation for studies into asset degradation and subsequent asset management. The taxonomy also includes code-based assets and thus can be helpful in further research into investigating degradation and debt concepts.

The entrepreneurial logic of startup software development: A study of 40 software startups

Software startups are an essential source of innovation and software-intensive products. The need to understand product development in startups and to provide relevant support are highlighted in software research. While state-of-the-art literature reveals how startups develop their software, the reasons why they adopt these activities are underexplored. This study investigates the tactics behind software engineering (SE) activities by analyzing key engineering events during startup journeys. We explore how entrepreneurial mindsets may be associated with SE knowledge areas and with each startup case. Our theoretical foundation is based on causation and effectuation models. We conducted semi-structured interviews with 40 software startups. We used two-round open coding and thematic analysis to describe and identify entrepreneurial software development patterns. Additionally, we calculated an effectuation index for each startup case. We identified 621 events merged into 32 codes of entrepreneurial logic in SE from the sample. We found a systemic occurrence of the logic in all areas of SE activities. Minimum Viable Product (MVP), Technical Debt (TD), and Customer Involvement (CI) tend to be associated with effectual logic, while testing activities at different levels are associated with causal logic. The effectuation index revealed that startups are either effectuation-driven or mixed-logics-driven. Software startups fall into two types that differentiate between how traditional SE approaches may apply to them. Effectuation seems the most relevant and essential model for explaining and developing suitable SE practices for software startups.

XPro: A Model to Explain the Limited Adoption and Implementation of Experimentation in Software Startups

Software startups develop innovative, software-intensive products or services. Such innovativeness translates into uncertainty regarding a matching need for a product from potential customers, representing a possible determinant reason for startup failure. Research has shown that experimentation, an approach based on the use of experiments to guide several aspects of software development, could improve these companies’ success rate by fostering the evaluation of assumptions about customers’ needs before developing a full-fledged product. Nevertheless, software startups are not using experimentation as expected. In this study, we investigated the reasons behind such a mismatch between theory and practice. To achieve it, we performed a qualitative survey study of 106 failed software startups. We built the eXperimentation Progression model (XPro), demonstrating that the effective adoption and implementation of experimentation is a staged process: first, teams should be aware of experimentation, then they need to develop an intention to experiment, perform the experiments, analyze the results, and finally act based on the obtained learning. Based on the XPro model, we further identified 25 inhibitors that prevent a team from progressing along the stages properly. Our findings inform researchers of how to develop practices and techniques to improve experimentation adoption in software startups. Practitioners could learn various factors that could lead to their startup failure so they could take action to avoid them.

Fast and curious: A model for building efficient monitoring- and decision-making frameworks based on quantitative data

Nowadays, the hype around artificial intelligence is at its absolute peak. Large amounts of data are collected every second of the day and a variety of tools exists to enable easy analysis of data. In practice, however, making meaningful use of it is way more challenging. For instance, affected stakeholders often struggle to specify their information needs and to interpret the results of such analyses. In this study we investigate how to enable continuous monitoring of information needs, and the generation of knowledge and insights for various stakeholders involved in the lifecycle of software-intensive products. The overarching goal is to support their decision making by providing relevant insights related to their area of responsibility. We implement multiple monitoring- and decision-making frameworks for six individual, real-world cases selected from three different platforms and covering four types of stakeholders. We compare the individual procedures to derive a generic process for instantiating such frameworks as well as a model to scale it up for multiple stakeholders. For one, we discovered that information needs of stakeholders are often related to a limited subset of data sources and should be specified in stages. For another, stakeholders often benefit from sharing and reusing existing components among themselves in later phases. Specifically, we identify three types of reuse: (1) Data and knowledge, (2) tools and methods, and (3) concepts. As a result, key aspects of our model are iterative feedback and specification cycles as well as the reuse of appropriate components to speed up the instantiation process and maximize the efficiency of the model. Our results indicate that knowledge and insights can be generated much faster and stakeholders feel the benefits of the analysis very early on by iteratively specifying information needs and by systematically sharing and reusing knowledge, tools and concepts.

Using Bayesian Network to Estimate the Value of Decisions within the Context of Value-Based Software Engineering: A Multiple Case Study

Context: Companies must make a paradigm shift in which both short- and long-term value aspects are employed to guide their decision-making. Such need is pressing in innovative industries, such as ICT, and is the core of Value-based Software Engineering (VBSE). Objective: This paper details three case studies where value estimation models using Bayesian Network (BN) were built and validated. These estimation models were based upon value-based decisions made by key stakeholders in the contexts of feature selection, test cases execution prioritization, and user interfaces design selection. Methods: All three case studies were carried out according to a Framework called VALUE — improVing decision-mAking reLating to software-intensive prodUcts and sErvices development. This framework includes a mixed-methods approach, comprising several steps to build and validate company-specific value estimation models. Such a building process uses as input data key stakeholders’ decisions (gathered using the Value tool), plus additional input from key stakeholders. Results: Three value estimation BN models were built and validated, and the feedback received from the participating stakeholders was very positive. Conclusions: We detail the building and validation of three value estimation BN models, using a combination of data from past decision-making meetings and also input from key stakeholders.

Minimum Viable Products for Internet of Things Applications: Common Pitfalls and Practices

Internet of Things applications are not only the new opportunity for digital businesses but also a major driving force for the modification and creation of software systems in all industries and businesses. Compared to other types of software-intensive products, the development of Internet of Things applications lacks a systematic approach and guidelines. This paper aims at understanding the common practices and challenges among start-up companies who are developing Internet of Things products. A qualitative research is conducted with data from twelve semi-structured interviews. A thematic analysis reveals common types of Minimum Viable Products, prototyping techniques and production concerns among early stage hardware start-ups. We found that hardware start-ups go through an incremental prototyping process toward production. The progress associates with the transition from speed-focus to quality-focus. Hardware start-ups heavily rely on third-party vendors in term of development speed and final product quality. We identified 24 challenges related to management, requirement, design, implementation and testing. Internet of Things entrepreneurs should be aware of relevant pitfalls and managing both internal and external risks.

A Progression Model of Software Engineering Goals, Challenges, and Practices in Start-Ups

Context: Software start-ups are emerging as suppliers of innovation and software-intensive products. However, traditional software engineering practices are not evaluated in the context, nor adopted to goals and challenges of start-ups. As a result, there is insufficient support for software engineering in the start-up context. Objective: We aim to collect data related to engineering goals, challenges, and practices in start-up companies to ascertain trends and patterns characterizing engineering work in start-ups. Such data allows researchers to understand better how goals and challenges are related to practices. This understanding can then inform future studies aimed at designing solutions addressing those goals and challenges. Besides, these trends and patterns can be useful for practitioners to make more informed decisions in their engineering practice. Method: We use a case survey method to gather first-hand, in-depth experiences from a large sample of software start-ups. We use open coding and cross-case analysis to describe and identify patterns, and corroborate the findings with statistical analysis. Results: We analyze 84 start-up cases and identify 16 goals, 9 challenges, and 16 engineering practices that are common among start-ups. We have mapped these goals, challenges, and practices to start-up life-cycle stages (inception, stabilization, growth, and maturity). Thus, creating the progression model guiding software engineering efforts in start-ups. Conclusions: We conclude that start-ups to a large extent face the same challenges and use the same practices as established companies. However, the primary software engineering challenge in start-ups is to evolve multiple process areas at once, with a little margin for serious errors.

Software engineering in start-up companies: An analysis of 88 experience reports

ContextStart-up companies have become an important supplier of innovation and software-intensive products. The flexibility and reactiveness of start-ups enables fast development and launch of innovative products. However, a majority of software start-up companies fail before achieving any success. Among other factors, poor software engineering could be a significant contributor to the challenges experienced by start-ups. However, the state-of-practice of software engineering in start-ups, as well as the utilization of state-of-the-art is largely an unexplored area.ObjectiveIn this study we investigate how software engineering is applied in start-up context with a focus to identify key knowledge areas and opportunities for further research.MethodWe perform a multi-vocal exploratory study of 88 start-up experience reports. We develop a custom taxonomy to categorize the reported software engineering practices and their interrelation with business aspects, and apply qualitative data analysis to explore influences and dependencies between the knowledge areas.ResultsWe identify the most frequently reported software engineering (requirements engineering, software design and quality) and business aspect (vision and strategy development) knowledge areas, and illustrate their relationships. We also present a summary of how relevant software engineering knowledge areas are implemented in start-ups and identify potentially useful practices for adoption in start-ups.ConclusionsThe results enable a more focused research on engineering practices in start-ups. We conclude that most engineering challenges in start-ups stem from inadequacies in requirements engineering. Many promising practices to address specific engineering challenges exists, however more research on adaptation of established practices, and validation of new start-up specific practices is needed.

Make Up Your Mind

Today, connected software-intensive products permeate virtually every aspect of our lives and the amount of customer and product data that is collected by companies across domains is exploding. In revealing what products we use, when and how we use them and how the product performs, this data has the potential to help companies optimize existing products, prioritize among features and evaluate new innovations. However, despite advanced data collection and analysis techniques, companies struggle with how to effectively extract value from the data they collect and they experience difficulties in defining what values to optimize for. As a result, the impact of data is low and companies run the risk of sub-optimization due to misalignment of the values they optimize for. In this paper, and based on multi-case study research in embedded systems and online companies, we explore data collection and analysis practices in companies in the embedded systems and in the online domain. In particular, we look into how the value that is delivered to customers can be expressed as a value function that combines different factors that are of importance to customers. By expressing customer value as a value function, companies have the opportunity to increase their awareness of key value factors and they can establish an agreement on what to optimize for. Based on our findings, we see that companies in the embedded systems domain suffer from vague and confusing value functions while companies in the online domain use simple and straightforward value functions to inform development. Ideally, and as proposed in this paper, companies should strive for a comprehensive value function that includes all relevant factors without being vague or too simple as is the case in the companies we studied. To achieve this, and to address the difficulties many companies experience, we present a systematic approach to value modelling in which we provide detailed guidance for how to quantify feature value in such a way that it can be systematically validated over time to help avoid sub-optimization that will harm the company in the long run.

Key Stakeholders’ Value Propositions for Feature Selection in Software-Intensive Products: An Industrial Case Study

Numerous software companies are adopting value-based decision making. However, what does value mean for key stakeholders making decisions? How do different stakeholder groups understand value? Without an explicit understanding of what value means, decisions are subject to ambiguity and vagueness, which are likely to bias them. This case study provides an in-depth analysis of key stakeholders’ value propositions when selecting features for a large telecommunications company's software-intensive product. Stakeholders’ value propositions were elicited via interviews, which were analyzed using Grounded Theory coding techniques (open and selective coding). Thirty-six value propositions were identified and classified into six dimensions: customer value, market competitiveness, economic value/profitability, cost efficiency, technology & architecture, and company strategy. Our results show that although propositions in the customer value dimension were those mentioned the most, the concept of value for feature selection encompasses a wide range of value propositions. Moreover, stakeholder groups focused on different and complementary value dimensions, calling to the importance of involving all key stakeholders in the decision making process. Although our results are particularly relevant to companies similar to the one described herein, they aim to generate a learning process on value-based feature selection for practitioners and researchers in general.

A theory of openness for software engineering tools in software organizations

ContextThe increased use of Open Source Software (OSS) affects how software-intensive product development organizations (SIPDO) innovate and compete, moving them towards Open Innovation (OI). Specifically, software engineering tools have the potential for OI, but require better understanding regarding what to develop internally and what to acquire from outside the organization, and how to cooperate with potential competitors. AimThis paper aims at synthesizing a theory of openness for software engineering tools in SIPDOs, that can be utilized by managers in defining more efficient strategies towards OSS communities. MethodWe synthesize empirical evidence from a systematic mapping study, a case study, and a survey, using a narrative method. The synthesis method entails four steps: (1) Developing a preliminary synthesis, (2) Exploring the relationship between studies, (3) Assessing the validity of the synthesis, and (4) Theory formation. ResultWe present a theory of openness for OSS tools in software engineering in relation to four constructs: (1) Strategy, (2) Triggers, (3) Outcomes, and (4) Level of openness. ConclusionThe theory reasons that openness provides opportunities to reduce the development cost and development time. Furthermore, OI positively impacts on the process and product innovation, but it requires investment by organizations in OSS communities. By betting on openness, organizations may be able to significantly increase their competitiveness.

Introducing continuous experimentation in large software-intensive product and service organisations

Abstract Software development in highly dynamic environments imposes high risks to development organizations. One such risk is that the developed software may be of only little or no value to customers, wasting the invested development efforts. Continuous experimentation, as an experiment-driven development approach, may reduce such development risks by iteratively testing product and service assumptions that are critical to the success of the software. Although several experiment-driven development approaches are available, there is little guidance available on how to introduce continuous experimentation into an organization. This article presents a multiple-case study that aims at better understanding the process of introducing continuous experimentation into an organization with an already established development process. The results from the study show that companies are open to adopting such an approach and learning throughout the introduction process. Several benefits were obtained, such as reduced development efforts, deeper customer insights, and better support for development decisions. Challenges included complex stakeholder structures, difficulties in defining success criteria, and building experimentation skills. Our findings indicate that organizational factors may limit the benefits of experimentation. Moreover, introducing continuous experimentation requires fundamental changes in how companies operate, and a systematic introduction process can increase the chances of a successful start.

An imperfect software debugging model considering irregular fluctuation of fault introduction rate

ABSTRACTIn general, software testing is a complicated and uncertain process. New faults can be introduced into the software during each fault removal. This process is called imperfect debugging. For simplicity, fault introduction rates are generally assumed to be constant. However, software debugging can be affected by many factors, such as subjective and objective influences, the difficulty and complexity of fault removal, the dependent relationships among faults, the changes in different phases of software testing, and the test schedules. Thus, the rate of fault introduction is not a constant, but is an irregularly fluctuating variable in software debugging. In this article, we propose a model with imperfect software debugging considering the irregular changes in fault introduction rates during software debugging. Experimental results reveal that our proposed model has good fitting capability and considerably stronger forecasting performance than that of the other models, and that the proposed model assumptions are close to the actual software debugging situation. Moreover, research on the irregular fluctuation of the fault introduction rate in software debugging has a certain reference value and important significance for software-intensive product testing, for instance, cloud computing.

Decision-Tree Models for Predicting Time Performance in Software-Intensive Projects

Initial requirements, new requirements and technical issues are the main factors that have a great effect over the software product development process. Difficulties resulting from incomprehensibility of initial requirements indicate two sub-factors: Deviations determined during analysis of initial requirements and deviations resulting from interpretation of requirements inaccurate and incomplete. New requirements being received from customers or end users during the development process affect the project performance. There can be problems during the implementation of product specifications, inaccurate formation of architectural design and technical solutions, incorrect coding of functions, or wrong realization of interfaces. The general technical problems cover the all problems arising from technical reasons and the negative situations they create on the project. During the design and implementation activities of software intensive projects, these tree main factors can be affected by other sub-factors. The aim of this study is to examine the factor classes which influence these three problem domains with CHAID (Chi-squared Automatic Interaction Detection) technique. Time deviations caused by initial requirements, new requirements and general technical problems are selected as target variables. In this research, 75 projects that develop software intensive products are studied to formalize the most accurate decision mechanism.

Towards improving decision making and estimating the value of decisions in value-based software engineering: the VALUE framework

To sustain growth, maintain competitive advantage and to innovate, companies must make a paradigm shift in which both short- and long-term value aspects are employed to guide their decision-making. Such need is clearly pressing in innovative industries, such as ICT, and is also the core of Value-based Software Engineering (VBSE). The goal of this paper is to detail a framework called VALUE—improving decision-making relating to software-intensive products and services development—and to show its application in practice to a large ICT company in Finland. The VALUE framework includes a mixed-methods approach, as follows: to elicit key stakeholders’ tacit knowledge regarding factors used during a decision-making process, either transcripts from interviews with key stakeholders are analysed and validated in focus group meetings or focus-group meeting(s) are directly applied. These value factors are later used as input to a Web-based tool (Value tool) employed to support decision making. This tool was co-created with four industrial partners in this research via a design science approach that includes several case studies and focus-group meetings. Later, data on key stakeholders’ decisions gathered using the Value tool, plus additional input from key stakeholders, are used, in combination with the Expert-based Knowledge Engineering of Bayesian Network (EKEBN) process, coupled with the weighed sum algorithm (WSA) method, to build and validate a company-specific value estimation model. The application of our proposed framework to a real case, as part of an ongoing collaboration with a large software company (company A), is presented herein. Further, we also provide a detailed example, partially using real data on decisions, of a value estimation Bayesian network (BN) model for company A. This paper presents some empirical results from applying the VALUE Framework to a large ICT company; those relate to eliciting key stakeholders’ tacit knowledge, which is later used as input to a pilot study where these stakeholders employ the Value tool to select features for one of their company’s chief products. The data on decisions obtained from this pilot study is later applied to a detailed example on building a value estimation BN model for company A. We detail a framework—VALUE framework—to be used to help companies improve their value-based decisions and to go a step further and also estimate the overall value of each decision.

Case Study of Agile Security Engineering

Security concerns are increasingly guiding both the design and processes of software-intensive product development. In certain environments, the development of the product requires special security arrangements for development processes, product release, maintenance and hosting, and specific security-oriented processes and governance. Integrating the security engineering processes into agile development methods can have the effect of mitigating the agile methods' intended benefits. This article describes a case of a large ICT service provider building a secure identity management system for a sizable government agency. The project was a subject to strict security regulations due to the end product's critical role. The project was a multi-team, multi-site, standard-regulated security engineering and development work executed following the Scrum framework. The study reports the difficulties in combining security engineering with agile development, provides propositions to enhance Scrum for security engineering activities. Also, an evaluation of the effects of the security work on project cost presented.