Software Product Line Testing Research Articles

Software product line testing: a systematic literature review

A Software Product Line (SPL) is a software development paradigm in which a family of software products shares a set of core assets. Testing has a vital role in both single-system development and SPL development in identifying potential faults by examining the behavior of a product or products, but it is especially challenging in SPL. There have been many research contributions in the SPL testing field; therefore, assessing the current state of research and practice is necessary to understand the progress in testing practices and to identify the gap between required techniques and existing approaches. This paper aims to survey existing research on SPL testing to provide researchers and practitioners with up-to-date evidence and issues that enable further development of the field. To this end, we conducted a Systematic Literature Review (SLR) with seven research questions in which we identified and analyzed 118 studies dating from 2003 to 2022. The results indicate that the literature proposes many techniques for specific aspects (e.g., controlling cost/effort in SPL testing); however, other elements (e.g., regression testing and non-functional testing) still need to be covered by existing research. Furthermore, most approaches are evaluated by only one empirical method, most of which are academic evaluations. This may jeopardize the adoption of approaches in industry. The results of this study can help identify gaps in SPL testing since specific points of SPL Engineering still need to be addressed entirely.

A Comparative Study on Testing Optimization Techniques with Combinatorial Interaction Testing for Optimizing Software Product Line Testing

A Comparative Study on Testing Optimization Techniques with Combinatorial Interaction Testing for Optimizing Software Product Line Testing

Test Case Prioritization for Software Product Line: A Systematic Mapping Study

Combinatorial explosion remains a common issue in testing. Due to the vast number of product variants, the number of test cases required for comprehensive coverage has significantly increased. One of the techniques to efficiently tackle this problem is prioritizing the test suites using a regression testing method. However, there is a lack of comprehensive reviews focusing on test case prioritization in SPLs. To address this research gap, this paper proposed a systematic mapping study to observe the extent of test case prioritization usage in Software Product Line Testing. The study aims to classify various aspects of SPL-TCP (Software Product Line – Test Case Prioritization), including methods, criteria, measurements, constraints, empirical studies, and domains. Over the last ten years, a thorough investigation uncovered twenty-four primary studies, consisting of 12 journal articles and 12 conference papers, all related to Test Case Prioritization for SPLs. This systematic mapping study presents a comprehensive classification of the different approaches to test case prioritization for Software Product Lines. This classification can be valuable in identifying the most suitable strategies to address specific challenges and serves as a guide for future research works. In conclusion, this mapping study systematically classifies different approaches to test case prioritization in Software Product Lines. The results of this study can serve as a valuable resource for addressing challenges in SPL testing and provide insights for future research.

Automated code-based test case reuse for software product line testing

ContextA software product line (SPL) grows in size as a new product is developed. A new product in an SPL should be tested extensively for quality assurance. For the efficient testing, previous studies suggested reusing the existing test cases of a product family. However, either their methods were not efficient because interventions from human experts, specifications, architecture and/or traceabilities for test cases were required. ObjectiveTo address these limitations, we propose an Automated Code-based Test case reuse for SPLs (ActSPL). ActSPL automatically identifies reusable test cases for new products of a product family using source code and test cases. MethodActSPL automatically constructs a hash-based traceability links between test cases and source code of a product family. Using the traceability links, ActSPL selects reusable test cases for a given new product from existing test cases of the product family. ResultsWe evaluated ActSPL in terms of the effectiveness and cost reduction of reusing test cases with five open-source SPLs. The evaluation results showed that ActSPL, on average, achieved 100 % precision and 62 % recall. In addition, ActSPL, on average, saved 47.5 % of time required for testing a new product from scratch. ConclusionOur study shows the feasibility of ActSPL reusing SPL test cases based on source code and test cases. Our results can be a basis for successive studies for automated code-based SPL testing.

Unifying Behavioral and Feature Modeling for Testing of Software Product Lines

Existing software product line (SPL) engineering testing approaches generally provide positive testing that validates the SPL’s functionality. Negative testing is commonly neglected. This research aims to unify behavioral and feature models of an SPL, enable testing before and after variability binding for domain-centric and product-centric testing, and combine positive and negative testing for a holistic testing view. This study suggests behavioral modeling with event sequence graphs (ESGs). This heterogeneous modeling strategy supports bottom-up domain testing and top-down product testing with the feature model. This new feature-oriented ESG test creation method generates shorter test sequences than the original ESG optimum test sequences. Statechart and original ESG test-generating methods are compared. Positive testing findings are similar. The Statechart technique generated 12 test cases with 59 events, whereas the ESG technique created six test cases with 60 events. The ESG technique generated 205 negative test cases with 858 events with the Test Suite Designer tool. However, the Conformiq Designer tool for the Statechart technique does not have a negative test case generation capability. It is shown that the proposed ESG-based holistic approach confirms not only the desirable (positive) properties but also the undesirable (negative) ones. As an additional research, the traditional ESG test-generating approach is compared to the new feature-oriented method on six SPLs of different sizes and features. Our case study results show that the traditional ESG test generation approach demonstrated higher positive test generation scores compare to the proposed feature-oriented test generation approach. However, our proposed feature-oriented test generation approach is capable of generating shorter test sequences, which could be beneficial for reducing the execution time of test cases compared to traditional ESG approach. Finally, our case study has also shown that regardless of the test generation approach, there has been found no significant difference between the Bottom-up and Top-down test strategies with respect to their positive test generation scores.

Testing of highly configurable cyber–physical systems — Results from a two-phase multiple case study

Cyber–physical systems are commonly highly configurable. Testing such systems is particularly challenging because they comprise numerous heterogeneous components that can be configured and combined in different ways. Despite a plethora of work investigating software testing in general and software product line testing in particular, variability in tests and how it is applied in industry with cyber–physical systems is not well understood. In this paper, we report on a multiple case study with four companies maintaining highly configurable cyber–physical systems focusing on their testing practices, with a particular focus on variability. Based on the results of the multiple case study, we conducted an interactive survey with experienced engineers from eight companies, including the initial four. We reflect on the lessons learned from the multiple case study. We conclude that experience-based selection of configurations for testing is currently predominant. We learned that variability modeling techniques and tools are not utilized and the dependencies between configuration options are only partially modeled at best using custom artifacts such as spreadsheets or configuration files. Another finding is that companies have the need and desire to cover more configuration combinations by automated tests. Our findings raise many questions interesting to the scientific community and motivating future research.

Variability testing of software product line: A preference-based dimensionality reduction approach

Multi- and many-evolutionary algorithms have been applied to derive products for the variability testing of Software Product Lines (SPLs). This problem refers to the selection of an adequate product set to test a SPL by optimizing some objectives related to the number of products to be tested, testing criteria to be satisfied, and revealed faults. However, some problems emerge when the number of objectives to be optimized increases, for example: the solutions generated by the optimization algorithms become incomparable, designing a Pareto-front in this context requires a large number of solutions, and the visualization of such solutions requires special techniques. Several techniques are proposed to tackle this problem, such as decomposition and algorithms based on indicators. Among them, algorithms based on dimensionality reduction and user preferences are widely used, but there are no studies in the literature investigating the usage of both in a combined way. In light of this, we introduce COR-NSGA-II (Confidence-based Objective Reduction NSGA-II). COR-NSGA-II defines for each objective a confidence-level calculated with the user preferences provided interactively. The objectives with higher values of confidence are removed from the next algorithm execution. For assessing the feasibility of COR-NSGA-II, experiments were conducted by using six different SPLs, seven objectives, two types of reference points representing the user preferences, and two scenarios to simulate different user profiles. COR-NSGA-II is evaluated against four algorithms explored in the literature for the problem, and outperforms most of them according to R-HV and R-IGD. It takes less time to execute and generates a reduced number of solutions, all of them satisfying the user preferences. A qualitative analysis performed with 12 potential users shows that the task of selecting a solution generated by COR-NSGA-II is easier than selecting a solution generated by the other algorithms.

Cost-effective test case generation with the hyper-heuristic for software product line testing

Implementation of optimization algorithm for test case generation in Model-Based Testing (MBT) for Software Product Line (SPL) has been increasing, due to the demand for optimal test case results with a balanced trade-off between cost and effectiveness measure. This paper proposed a hyper-heuristic test cases generation approach in MBT for SPL called Improvement Selection Rules-Modified Choice Function (ISR-MCF). ISR-MCF is implemented with three search operators which are Non-Dominated Sorting Genetic Algorithm II with low-level heuristic (NSGA-II-LLH), Strength Pareto Evolutionary with Low-Level Heuristic (SPEA 2-LLH) and Particle Swarm Optimization with Low-Level Heuristic (PSO-LLH). The approach was evaluated with a test model and the result shows that the proposed ISR-MCF with NSGA-II-LLH outperforms other existing rules for minimization measure (size of a test suite and execution time and maximization measure (coverage criteria).

Model-based Testing of Software Product Lines - Table F.1

Model-based Testing of Software Product Lines - Table F.1

Looking For Novelty in Search-Based Software Product Line Testing

Testing software product lines (SPLs) is difficult due to a huge number of possible products to be tested. Recently, there has been a growing interest in similarity-based testing of SPLs, where similarity is used as a surrogate metric for the <inline-formula><tex-math notation="LaTeX">$t$</tex-math></inline-formula> -wise coverage. In this context, one of the primary goals is to sample, by optimizing similarity metrics using search-based algorithms, a small subset of test cases (i.e., products) as dissimilar as possible, thus potentially making more <inline-formula><tex-math notation="LaTeX">$t$</tex-math></inline-formula> -wise combinations covered. Prior work has shown, by means of empirical studies, the great potential of current similarity-based testing approaches. However, the rationale of this testing technique deserves a more rigorous exploration. To this end, we perform correlation analyses to investigate how similarity metrics are correlated with the <inline-formula><tex-math notation="LaTeX">$t$</tex-math></inline-formula> -wise coverage. We find that similarity metrics generally have significantly positive correlations with the <inline-formula><tex-math notation="LaTeX">$t$</tex-math></inline-formula> -wise coverage. This well explains why similarity-based testing works, as the improvement on similarity metrics will potentially increase the <inline-formula><tex-math notation="LaTeX">$t$</tex-math></inline-formula> -wise coverage. Moreover, we explore, for the first time, the use of the novelty search (NS) algorithm for similarity-based SPL testing. The algorithm rewards “novel” individuals, i.e., those being different from individuals discovered previously, and this well matches the goal of similarity-based SPL testing. We find that the novelty score used in NS has (much) stronger positive correlations with the <inline-formula><tex-math notation="LaTeX">$t$</tex-math></inline-formula> -wise coverage than previous approaches relying on a genetic algorithm (GA) with a similarity-based fitness function. Experimental results on 31 software product lines validate the superiority of NS over GA, as well as other state-of-the-art approaches, concerning both <inline-formula><tex-math notation="LaTeX">$t$</tex-math></inline-formula> -wise coverage and fault detection capacity. Finally, we investigate whether it is useful to combine two satisfiability solvers when generating new individuals in NS, and how the performance of NS is affected by its key parameters. In summary, looking for novelty provides a promising way of sampling diverse test cases for SPLs.

Reducing Redundant Test Executions in Software Product Line Testing—A Case Study

In the context of software product line (SPL) engineering, test cases can be reused for testing a family of products that share common parts of source code. An approach to test the products of a product family is to exhaustively execute each test case on all the products. However, such an approach would be very inefficient because the common parts of source code will be tested multiple times unnecessarily. To reduce unnecessary repetition of testing, we previously proposed a method to avoid equivalent test executions of a product line in the context of regression testing. However, it turns out that the same approach can be used in a broader context than just regression testing of product families. In this paper, we argue the generality of the method in the sense that it can be used for testing of the first version of a product family as well as regression testing of its subsequent versions. In addition, in this paper, in order to make the method practically usable for users, we propose a process for applying it to SPL testing. We demonstrate the generality of our method and the practical applicability of the proposed process for the method by conducting a case study.

A SURVEY ON SOFTWARE PRODUCT-LINE TESTING

SPL (Software Product Line) is known as a set of software systems that share a mutual set of features. It is a powerful concept to achieve more efficient software system development. One of the necessary steps in software development processes is Testing. It consumes typically more than 50% of the whole development costs. Testing SPLs is challenging due to the exponential number of products in the number of features. Several approaches have been proposed to reduce the number of products to be tested. However, the testing aspect of SPL is still underdeveloped. This study aims at surveying the latest research on SPL testing to identify useful approaches and needs for future research. seven papers are classified concerning the following: the used Approach, the algorithms, and the type of testing that the research focuses on. The survey found that more validation and evaluation research is required to produce a more robust foundation for SPL testing. Finally, directions for future software product line testing recommendations.

Cost-effective product prioritisation technique for software product line testing

Software product line testing (SPLT) is a tedious task due to the generation of large number of products. Products testing sequence plays a significant role so that the testing resources could not be exhausted until all the products are tested. This problem can be overcome using the test case prioritisation (TCP) techniques. Many TCP techniques have been proposed to test SPL products in order, but they have not focused on product cost. The product with the highest cost should be tested first, because testing resources should not be exhausted to reach the main product with higher cost. In this paper, a cost-effective product prioritisation technique for SPLT is proposed in which products are prioritised on the basis of products cost. The rigorous evaluation results show that cost-effective prioritisation order is better than random order and similarity-based order.

Cost-effective product prioritisation technique for software product line testing

Software product line testing (SPLT) is a tedious task due to the generation of large number of products. Products testing sequence plays a significant role so that the testing resources could not be exhausted until all the products are tested. This problem can be overcome using the test case prioritisation (TCP) techniques. Many TCP techniques have been proposed to test SPL products in order, but they have not focused on product cost. The product with the highest cost should be tested first, because testing resources should not be exhausted to reach the main product with higher cost. In this paper, a cost-effective product prioritisation technique for SPLT is proposed in which products are prioritised on the basis of products cost. The rigorous evaluation results show that cost-effective prioritisation order is better than random order and similarity-based order.

Prediction of Euclidean distance between existing and target product for software product line testing using FeatureIDE

Prediction of Euclidean distance between existing and target product for software product line testing using FeatureIDE

Prediction of Euclidean distance between existing and target product for software product line testing using FeatureIDE

Prediction of Euclidean distance between existing and target product for software product line testing using FeatureIDE

Efficient Regression Testing of Software Product Lines by Reducing Redundant Test Executions

Regression testing for software product lines (SPLs) is challenging because it must ensure that all the products of a product family work correctly whenever changes are made. One approach to reducing the cost of regression testing is the regression test selection (RTS), which selects a subset of regression test cases. However, even when RTS is applied, SPL regression testing can still be expensive because, in the product line context, each test case can be executed on more than one product that reuses the test case, which would typically result in a large number of test executions. A promising direction is to eliminate redundant test executions of test cases. We propose a method that, given a test case, identifies a set of products, on which the test case will cover the same sequence of source code statements and produce the same testing results, and then excludes these products from products to apply the test case to. The evaluation results showed that when the full selection approach and the approach of repetitively applying an RTS method for a single software system are used for test selection, our method reduced, respectively, 59.3% and 40.0% of the numbers of test executions of the approaches.

Model-based testing of software product lines: Mapping study and research roadmap

Model-Based Testing (MBT) has been successfully applied to Software Product Lines (SPL). This paper provides a panorama of state-of-the-art on MBT of SPLs. We performed a systematic mapping for answering questions related with domains, approaches, solution types, variability, test case automation, artifacts, and evaluation. We built a roadmap from 44 selected studies. Main obtained results are: Software and Automotive domains are most considered; Black-box testing is widely performed; most studies have fully-automated support; variability is considered in most studies; Finite State Machines is the most used model to test SPLs; Behavioral-based and Scenario-based are the most used models; Case Studies and Experiments are used to evaluate MBT solutions and the majority is performed in industrial environments; traceability is not widely explored for MBT solutions. Furthermore, we provide a roadmap synthesizing studies based on used models, more formal artifacts, supporting tools, variability management, (semi-)automation, and traceability. The roadmap contributes to identify related primary studies based on given artifacts, variability management, tools, automation, and traceability techniques and to identify, from a given primary study, which artifacts, tools, variability management, automation and traceability techniques are related. Therefore, the roadmap serves as a guide to researchers and practitioners on how to model-based test SPLs.

Abstract Test Case Prioritization Using Repeated Small-Strength Level-Combination Coverage

Abstract test cases (ATCs) have been widely used in practice, including in combinatorial testing and in software product line testing. When constructing a set of ATCs, due to limited testing resources in practice (e.g., in regression testing), test case prioritization (TCP) has been proposed to improve the testing quality, aiming at ordering test cases to increase the speed with which faults are detected. One intuitive and extensively studied TCP technique for ATCs is $\lambda$ -wise Level-combination Coverage based Prioritization ( $\lambda$ LCP), a static, black-box prioritization technique that only uses the ATC information to guide the prioritization process. A challenge facing $\lambda$ LCP, however, is the necessity for the selection of the fixed prioritization strength $\lambda$ before testing—testers need to choose an appropriate $\lambda$ value before testing begins. Choosing higher $\lambda$ values may improve the testing effectiveness of $\lambda$ LCP (e.g., by finding faults faster), but may reduce the testing efficiency (by incurring additional prioritization costs). Conversely, choosing lower $\lambda$ values may improve the efficiency, but may also reduce the effectiveness. In this paper, we propose a new family of $\lambda$ LCP techniques, Repeated Small-strength Level-combination Coverage-based Prioritization (RSLCP), that repeatedly achieves the full combination coverage at lower strengths. RSLCP maintains $\lambda$ LCP's advantages of being static and black box, but avoids the challenge of prioritization strength selection. We have performed an empirical study involving five different versions of each of five C programs. Compared with $\lambda$ LCP, and Incremental-strength LCP (ILCP), our results show that RSLCP could provide a good tradeoff between testing effectiveness and efficiency. Our results also show that RSLCP is more effective and efficient than two popular techniques of Similarity-based Prioritization (SP). In addition, the results of empirical studies also show that RSLCP can remain robust over multiple system releases.

Test coverage criteria for software product line testing: Systematic literature review

Abstract Context In software product line testing (SPLT), test coverage criterion is an important concept, as it provides a means of measuring the extent to which domain testing has been performed and redundant application testing can be avoided based on the test coverage level achieved in domain testing. However, no previous literature reviews on SPLT have addressed test coverage criterion in SPLT. Objective The objectives of this paper are as follows: (1) to clarify the notions of test basis and test coverage criterion for SPLT; (2) to identify the test coverage criteria currently used for SPLT; (3) to investigate how various SPLT aspects, such as the SPLT method, variability implementation mechanism, and variability management approach, affect the choice of test coverage criterion for SPLT; and (4) to analyze the limitations of test coverage criteria currently used for SPLT. Method This paper conducts a systematic review of test coverage criteria in SPLT with 78 selected studies. Results We have several findings that can guide the future research on SPLT. One important finding is that choice of test coverage criterion in SPLT is independent from variability implementation mechanism, variability management, SPL approach, and binding time but is dependent on the variability representation used in development artifacts. Another that is easily overlooked is that SPL test coverage criteria with the same test coverage criterion names of single system testing neither adequately convey what should be covered by the test methods applying them, nor can they be more generally regarded as extensions or generalizations for SPLT of their corresponding test coverage criteria of single system testing. Conclusion This study showed that SPL test coverage criteria should be defined or redefined so that they can clearly deliver the target properties to be satisfied by SPLT.