Test Suite Augmentation Research Articles

Analyzing the impact of API changes on Android apps

The continuous evolution of Android mobile operating system leads to regular updates to its APIs, which may compromise the functionality of Android apps. Given the high frequency of Android API updates, analyzing the impact of API changes is vital to ensure the high reliability of Android apps. This paper introduces Apicia, a novel approach to analyzing the impact of API changes on Android apps. Apicia investigates the impact of changing the target API and identifies the affected program elements (i.e., classes, methods, and statements), the affected tests whose executions may exhibit changed behaviors as a result of the API update, as well as the app code that is not covered by the existing tests. We evaluate Apicia on 219 real-world Android apps. According to the results, API changes impact 46.30% of tests per app on average, and regression test selection based on Apicia can be cost effective. Moreover, many affected statements are not covered by existing tests, which indicates Apicia can help with test suite augmentation to achieve better coverage. These findings suggest that Apicia is a promising approach for assisting Android developers with understanding, testing, and debugging Android apps that are subject to rapid API updates.

ConTesa: Directed Test Suite Augmentation for Concurrent Software

As software evolves, test suite augmentation techniques may be used to identify which part of the program needs to be tested due to code changes and how to generate these new test cases for regression testing. However, existing techniques focus exclusively on sequential software, without considering concurrent software in which multiple threads may interleave with each other during the execution and thus lead to a combinatorial explosion. To fill the gap, we propose ConTesa , the first test suite augmentation tool for concurrent software. The goal is to generate new test cases capable of exercising both code changes and the thread interleavings affected by these code changes. At the center of ConTesa is a two-pronged approach. First, it judiciously reuses the current test inputs while amplifying their interleaving coverage using random thread schedules. Then, it leverages an incremental symbolic execution technique to generate more test inputs and interleavings, to cover the new concurrency-related program behaviors. We have implemented ConTesa and evaluated it on a set of real-world multithreaded Linux applications. Our results show that it can achieve a significantly high interleaving coverage and reveal more bugs than state-of-the-art testing techniques.

Test Suite Augmentation via Integrating Black- and White-Box Testing Techniques

Test Suite Augmentation via Integrating Black- and White-Box Testing Techniques

Directed test suite augmentation: an empirical investigation

SummaryTest suite augmentation techniques are used in regression testing to identify code elements in a modified program that are not adequately tested and to generate test cases to cover those elements. A defining feature of test suite augmentation techniques is the potential for reusing existing regression test suites. Our preliminary work suggests that several factors influence the efficiency and effectiveness of augmentation techniques that perform such reuse. These include the order in which target code elements are considered while generating test cases, the manner in which existing regression test cases and newly generated test cases are used, and the algorithm used to generate test cases. In this work, we present the results of two empirical studies examining these factors, considering two test case generation algorithms (concolic and genetic). The results of our studies show that the primary factor affecting augmentation using these approaches is the test case generation algorithm utilized; this affects both cost and effectiveness. The manner in which existing and newly generated test cases are utilized also has a substantial effect on efficiency and in some cases a substantial effect on effectiveness. The order in which target code elements are considered turns out to have relatively few effects when using concolic test case generation but in some cases influences the efficiency of genetic test case generation. The results of our first study, on four relatively small programs using a large number of test suites, are supported by our second study of a much larger program available in multiple versions. Together, the studies reveal a potential opportunity for creating a more cost‐effective hybrid augmentation approach leveraging both concolic and genetic test case generation techniques, while appropriately utilizing our understanding of the factors that affect them. Copyright © 2014 John Wiley & Sons, Ltd.

Assuring the model evolution of protocol software specifications by regression testing process improvement

SUMMARYModel‐based testing helps test engineers automate their testing tasks so that they are more cost‐effective. When the model is changed because of the evolution of the specification, it is important to maintain the test suites up to date for regression testing. A complete regeneration of the whole test suite from the new model, although inefficient, is still frequently used in the industry, including Microsoft. To handle specification evolution effectively, we propose a test case reusability analysis technique to identify reusable test cases of the original test suite based on graph analysis. We also develop a test suite augmentation technique to generate new test cases to cover the change‐related parts of the new model. The experiment on four large protocol document testing projects shows that our technique can successfully identify a high percentage of reusable test cases and generate low‐redundancy new test cases. When compared with a complete regeneration of the whole test suite, our technique significantly reduces regression testing time while maintaining the stability of requirement coverage over the evolution of requirements specifications. Copyright © 2011 John Wiley & Sons, Ltd.