Variability-intensive Systems Research Articles

VaryMinions: leveraging RNNs to identify variants in variability-intensive systems’ logs

From business processes to course management, variability-intensive software systems (VIS) are now ubiquitous. One can configure these systems’ behaviour by activating options, e.g., to derive variants handling building permits across municipalities or implementing different functionalities (quizzes, forums) for a given course. These customisation facilities allow VIS to support distinct relevant customer requirements while taking advantage of reuse for common parts. Customisation thus allows realising both scope and scale economies. Behavioural differences amongst variants manifest themselves in event logs. To re-engineer this kind of system, one must know which variant(s) have produced which behaviour. Since variant information is barely present in logs, this paper supports this task by employing machine learning techniques to classify behaviours (event sequences) among variants. Specifically, we train Long Short Term Memory (LSTMs) and Gated Recurrent Units (GRUs) recurrent neural networks to relate event sequences with the variants they belong to on six different datasets issued from the configurable process and VIS domains. After having evaluated 20 different architectures of LSTM/GRU, our results demonstrate that it is possible to effectively learn the trace-to-variant mapping with high accuracy (at least 80%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$80\\%$$\\end{document} and up to 99%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$99\\%$$\\end{document}) and at scale, i.e., identifying 50 variants using 5000+ traces for each variant.

Reviewing Automated Analysis of Feature Model Solutions for the Product Configuration

Feature models (FMs) appeared more than 30 years ago, and they are valuable tools for modeling the functional variability of systems. The automated analysis of feature models (AAFM) is currently a thriving, motivating, and active research area. The product configuration of FMs is a relevant and helpful operation, a crucial activity overall with large-scale feature models. The minimal conflict detection, the diagnosis of in-conflict configuration, and the product completion of consistent partial configuration are significant operations for obtaining consistent and well-defined products. Overall, configuring products for large-scale variability intensive systems (VIS) asks for efficient automated solutions for minimal conflict, diagnosis, and product configuration. Given the relevance of minimal conflict, diagnosis, and product configuration, and the current application of large-scale configuration and FMs for representing those systems and products, the main goals of this research paper are to establish the fundaments of the product configuration of feature models and systematically review existing solutions for the conflict detection, diagnosis, and product completion in FMs from 2010 to 2019. We can perceive that even though modern computing approaches exist for AAFM operations, no solutions exist for assisting the product configurations before 2020. This article reports that in 2020, new solutions appear regarding applying parallel computing for those goals. This research highlights research opportunities for developing new and more efficient solutions for conflict detection, diagnosis, and product completion of large-scale configurations.

Statistical model checking for variability-intensive systems: applications to bug detection and minimization

We propose a new Statistical Model Checking (SMC) method toidentify bugs in variability-intensive systems (VIS). Thestate-space of such systems is exponential in the number ofvariants, which makes the verification problem harder than forclassical systems. To reduce verification time, we propose tocombine SMC with featured transition systems (FTS)—amodel that represents jointly the state spaces of all variants. Ournew methods allow the sampling of executions from one or more(potentially all) variants. We investigate their utility in twocomplementary use cases. The first case considers the problem offinding all variants that violate a given property expressed inLinear-Time Logic (LTL) within a given simulation budget. To achievethis, we perform random walks in the featured transition systemseeking accepting lassos. We show that our method allows us to findbugs much faster (up to 16 times according to our experiments) thanexhaustive methods. As any simulation-based approach, however, therisk of Type-1 error exists. We provide a lower bound and an upperbound for the number of simulations to perform to achieve thedesired level of confidence. Our empirical study involving 59properties over three case studies reveals that our method managesto discover all variants violating 41 of the properties.This indicates that SMC can act as a coarse-grainedanalysis method to quickly identify the set of buggy variants.The second case complements the first one. In case thecoarse-grained analysis reveals that no variant can guarantee tosatisfy an intended property in all their executions, one shouldidentify the variant that minimizes the probability of violatingthis property. Thus, we propose a fine-grained SMC method thatquickly identifies promising variants and accurately estimates theirviolation probability. We evaluate different selection strategiesand reveal that a genetic algorithm combined with elitist selectionyields the best results.

Verification and abstraction of real-time variability-intensive systems

Featured timed automaton (FTA) is a concise formalism to model the real-time behaviour of variability-intensive systems. FTA extends the timed automaton by allowing optional transitions and clock constraints that are relevant only for a subset of the system variants. Then, one can verify a variant individually by deriving the corresponding TA from the FTA and using established tools like UPPAAL or apply family-based algorithms to verify all variants at once. These latter algorithms consist of computing the reachability relation in FTA as an antichain. Yet, they suffer from a three-source complexity: the number of states, the number of time clocks and the number of variants. This motivates the design of abstraction refinement heuristics to reduce verification effort. In this paper, we present the syntax and semantics of FTA, efficient algorithms to compute their reachability relations, and discuss how abstraction methods can be applied.

Automated analysis of two-layered feature models with feature attributes

Abstract The proliferation of features and platforms in variability intensive systems, coupled with substantial technological progress, imposes several challenges for software developers and equipment manufacturers—in some cases referred as technical sustainability. For instance, in the mobile application domain, developers often need to know the requirements and limitations of their applications to be supported on a specific platform. Conversely, an equipment manufacturer is interested in knowing what additional features become accessible on the application layer when the or platform is being upgraded. To date, analyzing such interdependencies between specific feature and platform combinations is a tough problem, but important to solve. There are well-established approaches in the literature to analyze variability–intensive systems using feature models. However, there is a lack of approaches to analyze application and platform features in multiple layers. In this paper we present a framework towards the analysis of multi-layered feature models. First, modeling the two layers including their respective interdependencies. Second, a definition of operations that can be imposed on such models. We also provide a reference implementation for analysis of multiple layers. Finally, we present two empirical evaluations demonstrating the feasibility of the approach in practice.

Functional feasibility analysis of variability-intensive data flow-oriented applications over highly-configurable platforms

Data-flow oriented embedded systems, such as automotive systems used to render HMI (e.g., instrument clusters, infotainments), are increasingly built from highly variable specifications while targeting different constrained hardware platforms configurable in a fine-grained way. These variabilities at two different levels lead to a huge number of possible embedded system solutions, which functional feasibility is extremely complex and tedious to predetermine. In this paper, we propose a tooled approach that capture high level specifications as variable dataflows, and targeted platforms as variable component models. Dataflows can then be mapped onto platforms to express a specification of such variability-intensive systems. The proposed solution transforms this specification into structural and behavioral variability models and reuses automated reasoning techniques to explore and assess the functional feasibility of all variants in a single run. We also report on the validation of the proposed approach. A qualitative evaluation has been conducted on an industrial case study of automotive instrument cluster, while a quantitative one is reported on large generated datasets.

Automated analysis of feature models: Quo vadis?

Feature models have been used since the 90s to describe software product lines as a way of reusing common parts in a family of software systems. In 2010, a systematic literature review was published summarizing the advances and settling the basis of the area of automated analysis of feature models (AAFM). From then on, different studies have applied the AAFM in different domains. In this paper, we provide an overview of the evolution of this field since 2010 by performing a systematic mapping study considering 423 primary sources. We found six different variability facets where the AAFM is being applied that define the tendencies: product configuration and derivation; testing and evolution; reverse engineering; multi-model variability-analysis; variability modelling and variability-intensive systems. We also confirmed that there is a lack of industrial evidence in most of the cases. Finally, we present where and when the papers have been published and who are the authors and institutions that are contributing to the field. We observed that the maturity is proven by the increment in the number of journals published along the years as well as the diversity of conferences and workshops where papers are published. We also suggest some synergies with other areas such as cloud or mobile computing among others that can motivate further research in the future.

Teaching Software Product Lines

Software Product Line (SPL) engineering has emerged to provide the means to efficiently model, produce, and maintain multiple similar software variants, exploiting their common properties, and managing their variabilities (differences). With over two decades of existence, the community of SPL researchers and practitioners is thriving, as can be attested by the extensive research output and the numerous successful industrial projects. Education has a key role to support the next generation of practitioners to build highly complex, variability-intensive systems. Yet, it is unclear how the concepts of variability and SPLs are taught, what are the possible missing gaps and difficulties faced, what are the benefits, and what is the material available. Also, it remains unclear whether scholars teach what is actually needed by industry. In this article, we report on three initiatives we have conducted with scholars, educators, industry practitioners, and students to further understand the connection between SPLs and education, that is, an online survey on teaching SPLs we performed with 35 scholars, another survey on learning SPLs we conducted with 25 students, as well as two workshops held at the International Software Product Line Conference in 2014 and 2015 with both researchers and industry practitioners participating. We build upon the two surveys and the workshops to derive recommendations for educators to continue improving the state of practice of teaching SPLs, aimed at both individual educators as well as the wider community.

Variability testing in the wild: the Drupal case study

Variability testing techniques search for effective and manageable test suites that lead to the rapid detection of faults in systems with high variability. Evaluating the effectiveness of these techniques in realistic settings is a must, but challenging due to the lack of variability-intensive systems with available code, automated tests and fault reports. In this article, we propose using the Drupal framework as a case study to evaluate variability testing techniques. First, we represent the framework variability using a feature model. Then, we report on extensive non-functional data extracted from the Drupal Git repository and the Drupal issue tracking system. Among other results, we identified 3392 faults in single features and 160 faults triggered by the interaction of up to four features in Drupal v7.23. We also found positive correlations relating the number of bugs in Drupal features to their size, cyclomatic complexity, number of changes and fault history. To show the feasibility of our work, we evaluated the effectiveness of non-functional data for test case prioritization in Drupal. Results show that non-functional attributes are effective at accelerating the detection of faults, outperforming related prioritization criteria as test case similarity.

Automated configuration support for infrastructure migration to the cloud

With an increasing number of cloud computing offerings in the market, migrating an existing computational infrastructure to the cloud requires comparison of different offers in order to find the most suitable configuration. Cloud providers offer many configuration options, such as location, purchasing mode, redundancy, and extra storage. Often, the information about such options is not well organised. This leads to large and unstructured configuration spaces, and turns the comparison into a tedious, error-prone search problem for the customers. In this work we focus on supporting customer decision making for selecting the most suitable cloud configuration—in terms of infrastructural requirements and cost. We achieve this by means of variability modelling and analysis techniques. Firstly, we structure the configuration space of an IaaS using feature models, usually employed for the modelling of variability-intensive systems, and present the case study of the Amazon EC2. Secondly, we assist the configuration search process. Feature models enable the use of different analysis operations that, among others, automate the search of optimal configurations. Results of our analysis show how our approach, with a negligible analysis time, outperforms commercial approaches in terms of expressiveness and accuracy.

Testing variability-intensive systems using automated analysis: an application to Android

Software product lines are used to develop a set of software products that, while being different, share a common set of features. Feature models are used as a compact representation of all the products (e.g., possible configurations) of the product line. The number of products that a feature model encodes may grow exponentially with the number of features. This increases the cost of testing the products within a product line. Some proposals deal with this problem by reducing the testing space using different techniques. However, a daunting challenge is to explore how the cost and value of test cases can be modeled and optimized in order to have lower-cost testing processes. In this paper, we present TESting vAriAbiLity Intensive Systems (TESALIA), an approach that uses automated analysis of feature models to optimize the testing of variability-intensive systems. We model test value and cost as feature attributes, and then we use a constraint satisfaction solver to prune, prioritize and package product line tests complementing prior work in the software product line testing literature. A prototype implementation of TESALIA is used for validation in an Android example showing the benefits of maximizing the mobile market share (the value function) while meeting a budgetary constraint.

Featured Transition Systems: Foundations for Verifying Variability-Intensive Systems and Their Application to LTL Model Checking

The premise of variability-intensive systems, specifically in software product line engineering, is the ability to produce a large family of different systems efficiently. Many such systems are critical. Thorough quality assurance techniques are thus required. Unfortunately, most quality assurance techniques were not designed with variability in mind. They work for single systems, and are too costly to apply to the whole system family. In this paper, we propose an efficient automata-based approach to linear time logic (LTL) model checking of variability-intensive systems. We build on earlier work in which we proposed featured transitions systems (FTSs), a compact mathematical model for representing the behaviors of a variability-intensive system. The FTS model checking algorithms verify all products of a family at once and pinpoint those that are faulty. This paper complements our earlier work, covering important theoretical aspects such as expressiveness and parallel composition as well as more practical things like vacuity detection and our logic feature LTL. Furthermore, we provide an in-depth treatment of the FTS model checking algorithm. Finally, we present SNIP, a new model checker for variability-intensive systems. The benchmarks conducted with SNIP confirm the speedups reported previously.

VAST 2011 workshop summary

The purpose of this article is to report about the first international workshop on Variability-intensive Systems Testing, Validation & Verification (VAST) which was held as part of the IEEE International Conference on Software Testing, Verification & Validation in Berlin on March 21st, 2011.