Test Suite Execution Research Articles

Quantification of Regression Test Suite Execution Time in Parallel Execution Setup with Weighted Test Suite Split Algorithm

Regression test suite execution time study focus is essentially on two aspects. They are execution time reduction and making effective use of available hardware resources and manpower. This paper investigates how the regression test suite can be split into subsets to make use of parallel execution across several machines with identical execution speeds and asymmetrical execution speeds. In the symmetrical execution speed setup, long test execution time test cases are evenly distributed across all the hardware machines. However, in asymmetrical execution speed machines, more test cases are distributed to speed machines to make efficient use of hardware resources. In all the situations where there is automation tool to execute the individual test cases of test suite this approach can be employed to make effective use of hardware resources and to keep the execution time within bounds. The Algorithms can also be used in situations where there are queues involved and serial, fixed time service takes place for each of the entity being served.

Quantification of Regression Test Suite Execution Time in Parallel Execution Setup with Weighted Test Suite Split Algorithm

Regression test suite execution time study focus is essentially on two aspects. They are execution time reduction and making effective use of available hardware resources and manpower. This paper investigates how the regression test suite can be split into subsets to make use of parallel execution across several machines with identical execution speeds and asymmetrical execution speeds. In the symmetrical execution speed setup, long test execution time test cases are evenly distributed across all the hardware machines. However, in asymmetrical execution speed machines, more test cases are distributed to speed machines to make efficient use of hardware resources. In all the situations where there is automation tool to execute the individual test cases of test suite this approach can be employed to make effective use of hardware resources and to keep the execution time within bounds. The Algorithms can also be used in situations where there are queues involved and serial, fixed time service takes place for each of the entity being served.

Underpinning Quality Assurance: Identifying Core Testing Strategies for Multiple Layers of Internet-of-Things-Based Applications

The Internet of Things (IoT) constitutes a digitally integrated network of intelligent devices equipped with sensors, software, and communication capabilities, facilitating data exchange among a multitude of digital systems via the Internet. Despite its pivotal role in the software development life-cycle (SDLC) for ensuring software quality in terms of both functional and non-functional aspects, testing within this intricate software–hardware ecosystem has been somewhat overlooked. To address this, various testing techniques are applied for real-time minimization of failure rates in IoT applications. However, the execution of a comprehensive test suite for specific IoT software remains a complex undertaking. This paper proposes a holistic framework aimed at aiding quality assurance engineers in delineating essential testing methods across different testing levels within the IoT. This delineation is crucial for effective quality assurance, ultimately reducing failure rates in real-time scenarios. Furthermore, the paper offers a mapping of these identified tests to each layer within the layered framework of the IoT. This comprehensive approach seeks to enhance the reliability and performance of IoT-based applications.

PUGH Decision Trapezoidal Fuzzy and Gradient Reinforce Deep Learning for Large Scale Requirement Prioritization

Objective: To prioritize requirements for large scale software projects within time involving uncertainty in the opinions among different stakeholders. Methods: We propose Pugh Trapezoidal Fuzzy and Gradient Reinforce Learning (PTF-GRL) methods for large scale software requirement prioritization. A Pugh Decision-based Trapezoidal Fuzzy Requirement Selection model is designed, inputting the functional and non-functional requirements of the corresponding stakeholders. With the assistance of Trapezoidal Fuzzy Inference, the qualitative factors are mapped with the corresponding numeric factors, which increases the computational efficiency. Findings: Performance is analyzed based on four parameters: The first parameter is accuracy and our method showed improvement of 4%, 7% and 3% compared to JRD-SCRUM, IFS and SRPTackle respectively. The second parameter is prioritization time and found that our method had reduced time of 30%, 37% and 39% compared with existing methods. The third parameter is precision and it was found that our method improves precision by 6%, 10% and 5% compared with the other two methods. The final parameter we consider is the test suite execution and our method showed improvement of 12%, 19% and 5% compared with the existing two methods. Novelty/Applications: The originality of this work indicates the better performance along with the optimal test suite execution even considering the uncertainty factor in the proposed method compared with existing similar methods. Keywords: Software Project; Pugh Decision Matrix; Trapezoidal Fuzzy Inference; Gradient Orientation; Reinforce Learning; Requirement Prioritization

Comparison of Multi-Objective Evolutionary Algorithms to Prioritize Regression Test Cases

Regression testing is one of the most critical testing activities among software product verification activities. Nevertheless, resources and time constraints could inhibit the execution of a full regression test suite, hence leaving us in confusion on what test cases to run to preserve the high quality of software products. Different techniques can be applied to prioritize test cases in resource-constrained environments, such as manual selection, automated selection, or hybrid approaches. Different Multi-Objective Evolutionary Algorithms (MOEAs) have been used in this domain to find an optimal solution to minimize the cost of executing a regression test suite while obtaining maximum fault detection coverage as if the entire test suite was executed. MOEAs achieve this by selecting set of test cases and determining the order of their execution. In this paper, three Multi Objective Evolutionary Algorithms, namely, NSGA-II, IBEA and MoCell are used to solve test case prioritization problems using the fault detection rate and branch coverage of each test case. The paper intends to find out what’s the most effective algorithm to be used in test cases prioritization problems, and which algorithm is the most efficient one, and finally we examined if changing the fitness function would impose a change in results. Our experiment revealed that NSGA-II is the most effective and efficient MOEA; moreover, we found that changing the fitness function caused a significant reduction in evolution time, although it did not affect the coverage metric.

Intelligent evaluation of test suites for developing efficient and reliable software

ABSTRACT Test suites play an important role in developing reliable software applications. Generally, the behaviour of software applications is verified by executing test suites to find defects. The quality of a test suite needs to be evaluated and enriched (if needed) especially for testing critical systems, such as plane-navigation system. This paper presents a novel method for comparing concrete and executable test suites using equivalence classes. This comparison identifies gaps in test suites with respect to each other. These gaps indicate potential weaknesses in the test suites. Furthermore, this method provides a mechanism to enrich the test suites using these gaps. In this method, we devise equivalence classes, and associate each test case to an equivalence class. We, then, simulate the comparison of test suites by comparing sets of equivalence classes. The method compares test suites in a platform independent manner. The test suites, which are compared, are smaller than the original test suites because the redundant test cases are removed from the test suites, which makes it efficient. We exercise our method over three case studies to demonstrate its viability and effectiveness. The first case study illustrates the application of the method and evaluates its effectiveness using a mutation analysis. The second case study evaluates its effectiveness using mutation and coverage analyses. The final case study evaluates it on a real case study, which is Lucene search engine.

Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity

Test suite reduction is an important approach that decreases the cost of regression testing. A test suite reduction technique operates based on the relationship between the test cases in the regression test suite and the test requirements in the program under test. Thus, its effectiveness should be closely related to the complexity of a regression test suite – the product of the number of test cases and the number of test requirements. Our previous work has shown that cost-aware techniques (i.e., the test suite reduction techniques that aim to decrease the regression test suite's execution cost) generally outperform the others in terms of decreasing the cost of running the regression test suite. However, the previous empirical studies that evaluated cost-aware techniques did not take into account test suite complexity. That is, prior experiments do not reveal if the cost-aware techniques scale and work effectively on test suites with more test cases and more test requirements. This means that prior experiments do not appropriately shed light on how well test suite reduction methods work with large programs or test suites. Therefore, this paper focuses on the Greedy-based techniques and empirically evaluates the additional Greedy and two cost-aware Greedy techniques – at different levels of test suite complexity – from various standpoints including the cost taken to run the regression test suite, the time taken to reduce the test suites, the total regression testing costs, the fault detection capability, the fault detection efficiency, and the common rates of the representative sets. To the best of our knowledge, none of the previous empirical studies classify a considerable number of test suites according to their complexity. Nor do any prior experiments evaluate the test suite reduction techniques, in terms of the aforementioned criteria, at different levels of test suite complexity. This paper represents the first such attempt to carry out this important task. Based on the empirical results, we confirm the strengths and weaknesses of the cost-aware techniques and develop insights into how the cost-aware techniques' effectiveness varies as the test suite complexity increases. • We evaluate test reduction methods at various levels of test suite complexity. • Few previous studies evaluated the test reduction methods in the way that we did. • The cost-aware methods generally attain the lowest total regression testing costs. • The cost-aware methods generally realize higher fault detection efficiency. • The benefits of using cost-aware methods increase as test suite complexity grows.

Enhancing Efficiency of the Test Case Prioritization Technique by Improving the Rate of Fault Detection

Test case prioritization techniques organize test cases for implementation in a manner that enhance their efficacy in accordance with some performance goal. The main aim of regression testing is to test the amended software to assure that the amendments performed in software are correct. It is not always feasible to retest entire test cases in a test suite due to limited resources. Therefore, it is necessary to develop some effective techniques that can enhance the regression testing effectiveness by organizing the test cases in an order following some testing criterion. One possible criterion of such prioritization is to enhance a test suite’s fault detection rate. It aspires to arrange test cases in an order that higher priority test cases run earlier than lower ones. This paper proposed a methodology for prioritizing regression test cases based on four factors namely the rate of fault detection, the number of faults detected, the test case ability of risk detection and the test case effectiveness. The proposed approach is implemented on two projects. The resultant test case order is analyzed with other prioritization techniques such as no prioritization, random prioritization, reverse prioritization, optimal prioritization and along with previous works for project 1. We have applied our proposed approach for prioritizing test cases in an order that maximize fault coverage with least test suite execution and compared its effectiveness with other orderings. The result of proposed approach shows higher average percentage of fault detected value and outperforms all other approaches.

Automatic Business Process Test Case Selection: Coverage Metrics, Algorithms, and Performance Optimizations

Business processes describe and implement the business logic of companies, control human interaction, and invoke heterogeneous services during runtime. Therefore, ensuring the correct execution of processes is crucial. Existing work is addressing this challenge through process verification. However, the highly dynamic aspects of the current processes and the deep integration and frequent invocation of third party services limit the use of static verification approaches. Today, one frequently utilized approach to address this limitation is to apply process tests. However, the complexity of process models is steadily increasing. So, more and more test cases are required to assure process model correctness and stability during design and maintenance. But executing hundreds or even thousands of process model test cases lead to excessive test suite execution times and, therefore, high costs. Hence, this paper presents novel coverage metrics along with a genetic test case selection algorithm. Both enable the incorporation of user-driven test case selection requirements and the integration of different knowledge sources. In addition, techniques for test case selection computation performance optimization are provided and evaluated. The effectiveness of the presented genetic test case selection algorithm is evaluated against five alternative test case selection algorithms.

A case study of black box fail-safe testing in web applications

Abstract External failures like network changes can affect system operation negatively. Mitigation requirements try to prevent or reduce their effects. This paper presents a black box testing approach that tests fail-safe behavior in web applications. Failure mitigation tests are built based on a functional test suite. Mitigation requirements are used to build mitigation models and mitigation tests paths through them. A genetic algorithm is used to generate failure scenarios (failure mitigation test requirements). Weaving rules describe how mitigation test paths are combined with behavioral test paths to create failure mitigation test paths. These are then transformed into an executable test suite. The paper also evaluates the genetic algorithm used to generate test requirements with respect to efficiency and effectiveness. A large case study, a commercial mortgage lending system, is used to explore applicability, scalability, and effectiveness of the approach.

Locating one pairwise interaction: Three recursive constructions

In a complex component-based system, choices (levels) for components (factors) may interact to cause faults in the system behaviour. When faults may be caused by interactions among few factors at specific levels, covering arrays provide a combinatorial test suite for discovering the presence of faults. While well studied, covering arrays do not enable one to determine the specific levels of factors causing the faults; locating arrays ensure that the results from test suite execution suffice to determine the precise levels and factors causing faults, when the number of such causes is small. Constructions for locating arrays are at present limited to heuristic computational methods and quite specific direct constructions. In this paper three recursive constructions are developed for locating arrays to locate one pairwise interaction causing a fault.

Neuro-Fuzzy Modeling for Multi-Objective Test Suite Optimization

AbstractRegression testing is a type of testing activity, which ensures that source code changes do not affect the unmodified portions of the software adversely. This testing activity may be very expensive in, some cases, due to the required time to execute the test suite. In order to execute the regression tests in a cost-effective manner, the optimization of regression test suite is crucial. This optimization can be achieved by applying test suite reduction (TSR), regression test selection (RTS), or test case prioritization (TCP) techniques. In this paper, we designed and implemented an expert system for TSR problem by using neuro-fuzzy modeling-based approaches known as “adaptive neuro-fuzzy inference system with grid partitioning” (ANFIS-GP) and “adaptive neuro-fuzzy inference system with subtractive clustering” (ANFIS-SC). Two case studies were performed to validate the model and fuzzy logic, multi-objective genetic algorithms (MOGAs), non-dominated sorting genetic algorithm (NSGA-II) and multi-objective particle swarm optimization (MOPSO) algorithms were used for benchmarking. The performance of the models were evaluated in terms of reduction of test suite size, reduction in fault detection rate, reduction in test suite execution time, and reduction in requirement coverage. The experimental results showed that our ANFIS-based optimization system is very effective to optimize the regression test suite and provides better performance than the other approaches evaluated in this study. Size and execution time of the test suite is reduced up to 50%, whereas loss in fault detection rate is between 0% and 25%.

A Novel Testing Model for SOA based Services

SOA (Service-Oriented Architecture) filled the gap between software and commercial enterprise. SOA integrates multiple web services. We bear to secure the caliber of web services that gives guarantee about what network services work and their output results. There is close to work has to be performed for an automatic test case generation for SOA based services. But, full coverage of XML elements is missing. To the best of our knowledge this all works do not attempt to cover all possible elements of the XML schema presents in the WSDL file. There is also a need to apply different assertions on each service operation for generating the test cases. To overcome this problem we proposed a novel testing model for SOA based application. This new testing model helps us to get the automatic test cases of SOA based application. We build up our new test model with the aid of our proposed test case generation algorithm and test case selection algorithm. In the end, we generate the test suite execution results and find the coverage of XML schema elements present in the WSDL file.

Residual Investigation

We introduce the concept of residual investigation for program analysis. A residual investigation is a dynamic check installed as a result of running a static analysis that reports a possible program error. The purpose is to observe conditions that indicate whether the statically predicted program fault is likely to be realizable and relevant. The key feature of a residual investigation is that it has to be much more precise (i.e., with fewer false warnings) than the static analysis alone, yet significantly more general (i.e., reporting more errors) than the dynamic tests in the program's test suite that are pertinent to the statically reported error. That is, good residual investigations encode dynamic conditions that, when considered in conjunction with the static error report, increase confidence in the existence or severity of an error without needing to directly observe a fault resulting from the error. We enhance the static analyzer FindBugs with several residual investigations appropriately tuned to the static error patterns in FindBugs, and apply it to nine large open-source systems and their native test suites. The result is an analysis with a low occurrence of false warnings (false positives) while reporting several actual errors that would not have been detected by mere execution of a program's test suite.

Testing iOS Apps with HadoopUnit: Rapid Distributed GUI Testing

Smartphone users have come to expect high-quality apps. This has increased the importance of software testing in mobile software development. Unfortunately, testing apps—particularly the GUI—can be very time-consuming. Exercising every user interface element and verifying transitions between different views of the app under test quickly becomes problematic. For example, execution of iOS GUI test suites using Apple’s UI Automation framework can take an hour or more if the app’s interface is complicated. The longer it takes to run a test, the less frequently the test can be run, which in turn reduces software quality. This book describes how to accelerate the testing process for iOS apps using HadoopUnit, a distributed test execution environment that leverages the parallelism inherent in the Hadoop platform. HadoopUnit was previously used to run unit and system tests in the cloud. It has been modified to perform GUI testing of iOS apps on a small-scale cluste —a modest computing infrastructure available to almost every developer. Experimental results have shown that distributed test execution with HadoopUnit can significantly outperform the test execution on a single machine, even if the size of the cluster used for the execution is as small as two nodes. This means that the approach described in this book could be adopted without a huge investment in IT resources. HadoopUnit is a cost-effective solution for reducing lengthy test execution times of system-level GUI testing of iOS apps.

Design of WSN Identifier Encoding Conformance Test System Based on Web

Accurate and unique sensor node identifier encoding will be an important factor to ensure the application of Wireless Sensor Network (WSN) identifier. Therefore, in order to detect the conformity of the WSN identifier encoding, the design and implementation of a WSN identifier encoding conformance test system based on web are presented. The WSN identifier encoding conformance test system consists of test method, system function, and test architecture. System function includes executable test suite (ETS) generation function, the test preparation function, the test execution function, the test response information analysis function and the test analysis function. For rapid conformance and credibility of the test system, we designed the conformance test software which consists of test server and test router. The design of test server includes man-machine interface, application service components, supporting service components and system management. The design of test router includes data communication layer management, test execution management. The implementation of test system mainly including the dataflow of test system and implementation process of test server. Verification of the test system comprises ETS verification and the test execution result verification. Additionally, the WSN identifier encoding test specifications was formulated according to the design of test system. The test results demonstrate the effectiveness of the test system by testing the sensor nodes in WSN.

Exception handling analysis and transformation using fault injection: Study of resilience against unanticipated exceptions

ContextIn software, there are the error cases that are anticipated at specification and design time, those encountered at development and testing time, and those that were never anticipated before happening in production. Is it possible to learn from the anticipated errors during design to analyze and improve the resilience against the unanticipated ones in production? ObjectiveIn this paper, we aim at analyzing and improving how software handles unanticipated exceptions. The first objective is to set up contracts about exception handling and a way to assess them automatically. The second one is to improve the resilience capabilities of software by transforming the source code. MethodWe devise an algorithm, called short-circuit testing, which injects exceptions during test suite execution so as to simulate unanticipated errors. It is a kind of fault-injection techniques dedicated to exception-handling. This algorithm collects data that is used for verifying two formal contracts that capture two resilience properties w.r.t. exceptions: the source-independence and pure-resilience contracts. Then we propose a code modification technique, called “catch-stretching” which allows error-recovery code (of the form of catch blocks) to be more resilient. ResultsOur evaluation is performed on 9 open-source software applications and consists in analyzing 241 catch blocks executed during test suite execution. Our results show that 101/214 of them (47%) expose resilience properties as defined by our exception contracts and that 84/214 of them (39%) can be transformed to be more resilient. ConclusionOur work shows that it is possible to reason on software resilience by injecting exceptions during test suite execution. The collected information allows us to apply one source code transformation that improves the resilience against unanticipated exceptions. This works best if the test suite exercises the exceptional programming language constructs in many different scenarios.

An approach to creating concretized test scenarios within test automation technology for industrial software projects

The paper describes an approach for creation of effective test automation technology for industrial software projects based on a formal model of the system under test, performing its automatic symbolic verification of the model and generation of symbolic traces, then their automatic concretization, generation and execution of test suites. The technology includes means for test results analysis and tests correction and update. High quality of software products is ensured by a balance between static (verification) and dynamic (testing) analysis techniques starting from the earlier development phases and through automation of all phases of software development. Special emphasis is made on presentation of the algorithm of concretization and configuring test scenarios.

Testing of PolPA-based usage control systems

The implementation of an authorization system is a critical and error-prone activity that requires a careful verification and testing process. As a matter of fact, errors in the authorization system code could grant accesses that should instead be denied, thus jeopardizing the security of the protected system. In this paper, we address the testing of the implementation of the Policy Decision Point (PDP) within the PolPA authorization system that enables history-based and usage-based control of accesses. Accordingly, we propose two testing strategies specifically conceived for validating the history-based access control and the usage control functionalities of the PolPA PDP. The former is based on a fault model able to highlight the problems and vulnerabilities that could occur during the PDP implementation. The latter combines the standard technique for conditions coverage with a methodology for simulating the continuous control of the PDP during the runtime execution. Both strategies are implemented within a testing framework supporting the automatic generation and execution of security test suites. Results produced by the application of this testing framework to a real case study are presented.

New test case prioritization strategies for regression testing of web applications

A test case prioritization technique reschedules test cases for regression testing in an order to achieve specific goals like early fault detection. Currently, black box testing of web applications requires manual prioritization of test cases which is very time consuming. We propose a new two level prioritization approach to prioritize test cases for web applications as a whole. Our approach selects modified functionalities in a web application and automatically executes test cases on the basis of the impact of modified functionalities. We suggest several new prioritization strategies for web applications and examine whether these prioritization strategies improve the rate of fault detection for web applications. We propose a new test suite prioritization model for web applications that detects modified functionalities, selects test cases related to those functionalities and reschedules them to detect faults early in test suite execution.