Complete, Sound, and Scalable Identification of Minimal Failure-Causing Schema

Combinatorial testing is a widely used technique to detect failures caused by interactions of system under test (SUT) parameters. Faulty interaction localization (FIL) is a problem to locate parameter-value combinations that trigger failures from combinatorial test cases and their testing results. FIL is important for debugging, but is expensive for large test suites and SUTs since the number of candidates of faulty interactions increases exponentially with the number of parameters and the size of interactions. To address this problem, this paper proposes a method employing logistic regression. The proposed FIL based on Regression coefficients Of loGistic regression analysis (called FROG) computes the suspiciousness of each parameter-value combination to be included in a faulty interaction from its corresponding regression coefficient. We evaluate the proposed method by applying FROG to combinatorial t-way test cases (2 ≤ t ≤ 4) for real application SUT models, e.g. TCAS, GCC, and Apache. Our experiment results show that FROG can effectively locate faulty interactions injected while efficiently reducing the number of candidates of potential faulty interactions to be checked.

Combinatorial testing (CT) is an important black-box testing method. In CT, the behavior of the system under test (SUT) is affected by several parameters/components. Then CT generates a combinatorial test suite. After the user executes a test suite and starts debugging, some test cases fail and some pass. From the perspective of a black box, the failures are caused by interaction of several parameters. It will be helpful if we can identify a small set of interacting parameters that caused the failures. This paper proposes a new automatic approach to identifying faulty interactions. It uses (pseudo-Boolean) constraint solving and optimization techniques to analyze the execution results of the combinatorial test suite. Experimental results show that the method is quite efficient and it can find faulty combinatorial interactions quickly. They also shed some light on the relation between the size of test suite and the ability of fault localization.

In this paper, the Test Pattern Generation (TPG) with a new simple hybrid (dynamic and static) compaction technique for combinational logic circuits and systems is presented. Digital systems are implemented nowadays on an advance VLSI technology, which is called Field Programmable Gate Array (FPGA). The test procedure requires a deliberate introduction of faults in the System Under Test (SUT). FPGA circuits and systems are developed and written in Hardware Description Languages (HDLs). In addition, a novel method is also proposed and implemented in order to intentionally inject faults in the Verilog HDL code design of the SUT. It covers all possible fault locations in the SUT. This approach provides high quality compact test vectors and efficient memory utilization in comparison with other state-of-the-art methods. Not only are the hardness of individual faults and the sensitiveness of fault locations determined, but also their effectiveness is proposed in the fault tolerance strategy.

The purpose of this study is to describe the new paradigms in front-end web application testing and how combinatorial testing (CT) fits in this modern development environment to create cost effective, highly fault-detecting automated tests. The system under test (SUT) is a cloud based application for monitoring and controlling building operations from Siemens Building Technologies, currently under development. Three examples for the applications of combinatorial testing in the front-end are analyzed, incorporating the CT model into automation using Protractor test framework's end-to-end (e2e) UI tests in behavioral driven development (BDD) style. The manner in which the CT model translates automation code is studied. A scenario where a sequence of actions are incorporated into a CT model is illustrated; with a focus on veri?cation of these sequences, compositions of the actions and streamlining the expected assertions per the test oracle.

Combinatorial testing (CT) has been proven effective in revealing the failures caused by the interaction of factors that affect the behavior of a system. The theory of Minimal Failure-Causing Schema (MFS) has been proposed to isolate the cause of a failure after CT. Most algorithms that aim to identify MFS focus on handling a single fault in the System Under Test (SUT). However, we argue that multiple faults are more common in practice, under which masking effects may be triggered so that some failures cannot be observed. The traditional MFS theory lacks a mechanism to handle such effects; hence, they may incorrectly isolate the MFS. To address this problem, we propose a new MFS model that takes into account multiple faults. We first formally analyze the impact of the multiple faults on existing MFS identifying algorithms, especially in situations where masking effects are triggered by multiple faults. We then develop an approach that can assist traditional algorithms to better handle multiple faults. Empirical studies were conducted using several kinds of open-source software, which showed that multiple faults with masking effects do negatively affect traditional MFS identifying approaches and that our approach can help to alleviate these effects.

This paper explores an effective solution for a test configuration to perform system-level manufacturing tests without use of system management/user interface software of system under test (SUT). A generic, computer aided and modular test configuration, which acts in combination and in an integrated way with the management unit(s) of the SUT over Ethernet connectivity is proposed as a solution. Using enough number of software modules to be installed on the management unit(s), the proposed test configuration provides a number of improvements compared to the existing test configurations. Parallel testing capability is one of these major improvements meaning that test scenarios may be applied to a certain number of SUTs at the same time independently. This feature accelerates the system-level manufacturing test activities both during the planned test operations and during fault localization cases. The proposed configuration is implemented and used on various systems based on various platforms such as; sheltered C4I platforms, tracked observer or gun platforms, wheeled ground surveying platforms etc.

Extracting Combinatorial Test parameters and their values using model checking and evolutionary algorithms

Even for small programs, the input space is huge - often unbounded. Partition testing divides the input space into disjoint equivalence classes and combinatorial testing selects a subset of all possible input class combinations, according to criteria such as pairwise coverage. The down side of this approach is that the partitioning of the input space into equivalence classes (input classification) is done manually. It is expensive and requires deep domain and implementation understanding. In this paper, we propose a novel approach to classify test inputs and their dependencies automatically. Firstly, random (or automatically generated) input vectors are sent to the system under test (SUT). For each input vector, an observed vector is produced by monitoring the execution of the SUT. Secondly, hit vectors are grouped into clusters using machine learning. Each cluster contains similar hit vectors, i.e., similar behaviors, and from them we obtain corresponding clusters of input vectors. Input classes are then extracted for each input parameter straightforwardly. Our experiments with a number of subjects show good results as the automatically generated classifications are the same or very close to the expected ones.

Security testing of web applications remains a major problem of software engineering. In order to reveal vulnerabilities, manual and automatic testing approaches use different strategies for detection of certain kinds of inputs that might lead to a security breach. In this paper we compared a state-of-the-art manual testing tool with an automated one that is based on model-based testing. The first tool requires user input from the tester whereas the second one reduces the necessary amount of manual manipulation. Both approaches depend on the corresponding test case generation technique and its produced inputs are executed against the system under test (SUT). For this case we enhance a novel technique, which combines a combinatorial testing technique for input generation and a model-based technique for test execution. In this work the input parameter modelling is improved by adding constraints to generate more comprehensive and sophisticated testing inputs. The evaluated results indicate that both techniques succeed in detecting security leaks in web applications with different results, depending on the background logic of the testing approach. Last but not least, we claim that attack pattern-based combinatorial testing with constraints can be an alternative method for web application security testing, especially when we compare our method to other test generation techniques like fuzz testing.

Advancement in software development has resulted in complex software applications that encompass various functional and non-functional requirements. Such a complex system usually consists of many inputs either directly from users or from other connected systems or devices. Here, there is a potential for the system to go wrong due to certain combinations of inputs. Combinatorial Testing (or T-Way Testing) is effective in tackling the issue. Numerous studies have proposed strategies in generating T-Way test suite, and current trend indicates that researchers often incorporate metaheuristic algorithms in their proposed strategies. Many recent studies employ parameter optimization algorithms such as (Whale Optimization Algorithm, Particle Swarm Optimization, Gravitational Optimization Algorithm) in generating an optimized T-Way test suite. Often, researchers need to tune the parameters involved in the algorithm before the algorithm can be used for test suite generation. Since the system under test (SUT) can come in various numbers of input, it is impossible to find a single best value for every algorithm parameter. As a result, this paper proposed a T-Way Test suite generator utilizing Wingsuit Optimization Algorithm (a parameter free optimization algorithm) for combinatorial test suite generation. The algorithm learnt by itself as the optimization process progresses and hence eliminates the need for control parameters. Statistical analysis shows that WFS produces a smaller test suite compares to most T-Way strategies and in some cases, the difference between test suite size produce by WFS and other T-Way strategies are insignificant.

Currently, software product becomes an essential component in running many stakeholders' activities. For instance, the industries mostly use cloud services to execute their important business functionality. However, by a few input's parameter interacting, this functionality can be pended. Such constraint poses challenging to cover various features of failure especially in ensuring cloud application. One way is to devise a strategy to cover input parameters' characteristics based on Combinatorial testing approach. This technique includes all possible combinations of test inputs for detecting bugs on the System Under Test (SUT). The paper explains the Combinatorial covering arrays to generate relatively exhaustive testing by modeling features of sample services using Feature IDE plugin in Eclipse IDE. This way, we build the input domain model to represent coverage of the existing mobility service running on NEMo Mobility cloud platform. Using this model, covering arrays is applied to generate t-way test cases by leveraging IPOg algorithm, which is implemented in a CiTLab. As a test case management, the JUnit testing framework uses test stubs to validate the test methods of generated test cases on the specified service (SUT).

At present, programming item turns into a fundamental segment in running numerous partners' exercises. For example, the enterprises for the most part use cloud administrations to execute their significant business usefulness. Be that as it may, by a couple of info's parameter interfacing, this usefulness can be pended. Such requirement postures testing to cover different highlights of disappointment particularly in guaranteeing cloud application. One path is to devise a technique to cover input parameters' qualities dependent on combinatorial testing approach. This method incorporates every single imaginable blend of test contributions for identifying bugs on the system under test (SUT). The paper clarifies the combinatorial covering exhibits to create generally comprehensive testing by demonstrating highlights of test administrations utilising feature IDE module in eclipse IDE. Along these lines, we fabricate the information area model to speak to inclusion of the current portability administration running on NEMo mobility cloud stage. Utilising this model, covering exhibits is connected to create t-way experiments by utilising IPOg calculation, which is executed in a CiTLab. As an experiment the executives, the JUnit testing structure, uses test stubs to approve the test techniques for produced experiments on the predefined administration (SUT).

Reflections on Surrogate-Assisted Search-Based Testing: A Taxonomy and Two Replication Studies based on Industrial ADAS and Simulink Models

In its simplest form, software testing consists of creating test cases from a defined input space, running them in the system-under-test (SUT), and evaluating the outputs with a mechanism for determining success or failure (i.e. an oracle). Metamorphic testing (MT) provides powerful concepts for alleviating the problem of a lack of oracles. To increase the adoption of MT among industry practitioners, approaches and tools that lower the effort to identify potential metamorphic relations (MRs) are very much in demand. As such, we propose a learning-based approach to MR discovery and exploration using concepts of metamorphic testing, association rule learning, and combinatorial testing. The results have implications for numerous applications including software testing and program comprehension, among others. These implications set a strong foundation for a future, extensible metamorphic exploration framework.

Detecting Arrays (DAs) are mathematical objects that enable fault localization in combinatorial interaction testing. Each row of a DA serves as a test case, whereas a whole DA is treated as a test suite. In real-world testing problems, it is often the case that some constraints exist among test parameters. In this paper, we show that it may be impossible to construct a DA using only constraint-satisfying test cases. The reason for this is that a set of some faulty interactions may always mask the effect of other faulty interactions in the presence of constraints. Based on this observation, we propose the notion of Constrained Detecting Arrays (CDAs) to adapt DAs to practical situations. The definition of CDAs requires that all rows of a CDA must satisfy the constraints and the same fault localization capability as the DA must hold except for such inherently undetectable faults. We then propose a computational method for constructing CDAs. Experimental results obtained by using a program that implements the method show that the method was able to produce CDAs within a reasonable time for practical problem instances.