Test Case Generation Techniques Research Articles

Investigating the robustness of locators in template-based Web application testing using a GUI change classification model

GUI-based test-cases generated by Capture and Replay tools suffer from the well-known fragility problem: they may break even if small layout changes are operated in a Web application, without modifying the app functionality. An approach based on the automatic injection of HTML tag attributes named hooks in the source code of Web templates has been recently proposed to solve this problem. Such hooks allow the unique identification of GUI items to be located during test case execution. This technique showed its effectiveness in a preliminary validation study, where it allowed to significantly reduce the number of test case locator breakages in regression testing of student-made Web applications. This paper presents a further validation study where we compared the robustness of hook-based test cases against state-of-the-art and state-of-the-practice techniques for locating GUI objects. We proposed a three dimensional model for classifying different types of layout changes and used it to define a benchmark of realistic changes. Thanks to the model, we systematically compared the robustness of test cases generated by different techniques with respect to specific types of changes and studied the relationship between fragility issues and types of changes in different test case generation techniques.

The optimization path of core literacy of innovative talents in universities based on the improved Markov model

Abstract As we enter the information technology era of the 21st century, universities are the main place for talent cultivation, but as an important part of higher Education, they are not sufficiently aware of innovative talent cultivation, and their own development is immature, which leads to the unsound innovative talent cultivation model. Therefore, in this paper, we use the Markov model-based test case generation technique to explore in depth the techniques of transfer probability acquisition, random selection, and adequacy determination and improve the Markov model with linear regression to further strengthen the robustness of the prediction algorithm, reduce the error of prediction and improve the accuracy of prediction. Through the processing of student data, the results show that the standard deviation of the improved Markov model is 0.06903 and the processing time is 87 seconds, the standard deviation of the original model is 0.07255 and the processing time is 133 seconds, and the larger the standard deviation of the experimental sample the less accurate the experiment is. The improved Markov model has a good experimental effect with little error, which can effectively optimize the training mode of innovative talents in universities.

Software Test Case Generation Tools and Techniques: A Review

Software Industry is evolving at a very fast pace since last two decades. Many software developments, testing and test case generation approaches have evolved in last two decades to deliver quality products and services. Testing plays a vital role to ensure the quality and reliability of software products. In this paper authors attempted to conduct a systematic study of testing tools and techniques. Six most popular e-resources called IEEE, Springer, Association for Computing Machinery (ACM), Elsevier, Wiley and Google Scholar to download 738 manuscripts out of which 125 were selected to conduct the study. Out of 125 manuscripts selected, a good number approx. 79% are from reputed journals and around 21% are from good conference of repute. Testing tools discussed in this paper have broadly been divided into five different categories: open source, academic and research, commercial, academic and open source, and commercial & open source. The paper also discusses several benchmarked datasets viz. Evosuite 10, SF100 Corpus, Defects4J repository, Neo4j, JSON, Mocha JS, and Node JS to name a few. Aim of this paper is to make the researchers aware of the various test case generation tools and techniques introduced in the last 11 years with their salient features.

Adaptive Emulation Framework for Multi-Architecture IoT Firmware Testing

Internet of things (IoT) devices are being increasingly used in numerous areas. However, the low priority on security and various IoT types have made these devices vulnerable to attacks. To prevent this, recent studies have analyzed firmware in an emulation environment that does not require actual devices and is efficient for repeated experiments. However, these studies focused only on major firmware architectures and rarely considered exotic firmware. In addition, because of the diversity of firmware, the emulation success rate is not high in terms of large-scale analyses. In this study, we propose the adaptive emulation framework for multi-architecture (AEMA). In the field of automated emulation frameworks for IoT firmware testing, AEMA considers the following issues: (1) limited compatibility for exotic firmware architectures, (2) emulation instability when configuring an automated environment, and (3) shallow testing range resulting from structured inputs. To tackle these problems, AEMA can emulate not only major firmware architectures but also exotic firmware architectures not previously considered, such as Xtensa, ColdFire, and reduced instruction set computer (RISC) version five, by implementing a minority emulator. Moreover, we applied the emulation arbitration technique and input keyword extraction technique for emulation stability and efficient test case generation. We compared AEMA with other existing frameworks in terms of emulation success rates and fuzz testing. As a result, AEMA succeeded in emulating 864 out of 1,083 overall experimental firmware and detected vulnerabilities at least twice as fast as the experimental group. Furthermore, AEMA found a 0-day vulnerability in real-world IoT devices within 24 h.

A Systematic Mapping Study on UML Model based Test Case Generation and Optimization Techniques

A Systematic Mapping Study on UML Model based Test Case Generation and Optimization Techniques

Combinatorial Test Case Generation Using Q-Value Based Particle Swarm Optimization

Combinatorial testing is an effective method for generating test cases. Pairwise testing is a combinatorial approach that evaluates the interactions between the input test parameters while reducing test case size by selecting a broader search area. Most combinatorial testing research focuses on developing novel approaches for generating an optimal number of test cases that cover pairwise combinations of input test parameters. Using existing test case generation techniques, optimal or near-optimal combinatorial test cases are generated in polynomial time. The authors presented the Q-value-based Particle Swarm Optimization (Q-PSO) technique for efficiently and effectively generating an optimal number of test cases. The primary goals of the proposed technique are to generate test cases using a Q-value based PSO, which is easier to build and has fewer parameters to define than other meta-heuristic search methodologies and to put the proposed technique into practice and report on an empirical study that examines and verifies the significant impact factors in the proposed approach. Q-value is used to evaluate the particles (referred to as test cases) in the Q-PSO. The reward is totalled in the Q-value, which serves as the fitness function for PSO evolution. The Q-value of each particle determines its performance and indicates how quickly the particle can lead the system's state to the set of objective states. The authors used the Q-PSO technique to validate the efficiency and efficacy of the proposed approach. The Q-PSO technique's results are compared to existing metaheuristics and computation-based techniques. In most inputs based on the development environment, meta-heuristic search techniques take significantly longer than other greedy techniques. For some inputs, the proposed Q-PSO technique outperforms existing meta-heuristics techniques. Q-PSO results are also compared to IPOG, ITCH, Jenny, TConfig, TVG, and other well-known computational-based techniques. The goal of the comparison is to examine how the size of the test cases generated has grown over time.

Automatic Generation and Optimization of Combinatorial Test Cases from UML Activity Diagram Using Particle Swarm Optimization

Generation of test cases is one of the essential activities of the software testing process. The process of executing a programme to identify defects to improve the system's quality is known as software testing. Manually writing test cases takes time, effort, and money. On the other hand, generating test cases automatically is the solution to this problem. For this automation process, a model-based test case generation technique would be acceptable. A model is usually required to generate test cases in the model-based testing technique. Nowadays, researchers have relied on the activity diagram to generate test cases. Test cases for combinatorial logic systems are required. Combinatorial testing is essential for producing a small number of test cases and identifying errors occurred by interactions between system input parameters. Information about constraints, parameters and its values are required for generation of test cases. It is difficult to extract information regarding constraints, parameters, its values, and interactions between parameters from an Activity Diagram. A novel approach is proposed to extract this information from an Activity Diagram. The authors created a tool that automatically generates combinatorial test cases using UML Activity Diagrams. The proposed tool has two main parts. First, the combinatorial test design model is developed for extraction of input parameters. Second part is generation of optimized number of combinatorial test cases using Particle Swarm Optimization algorithm. Finally, the authors experimented on a real-world case study namely viz. Railway Reservation using the proposed tool, and it is shown that the proposed tool generated optimum number of combinatorial test cases.

BOOMPizer: Minimization and prioritization of CONCOLIC based boosted MC/DC test cases

Recent research evidence indicates that the powerful testing tools, even though generate test inputs automatically for coverage measures, but not up to satisfaction. These tools sometimes achieve high structural coverage, which do not guarantee to have high fault detection ability. These findings lead us to a decisive point that code coverage is merely one factor towards effective test data generation. Thus, we discuss our findings and proposed work on Modified Condition/ Decision Coverage (MC/DC) test case generation and prioritization techniques. This work aims to generate, minimize, and prioritize MC/DC test cases obtained through concolic testing process. This work presents three technical contributions. The first contribution is to propose a greedy algorithm to increase the number of effective test cases for improving MC/DC scores. The second contribution is to minimize the updated test suite size to have only the optimal number of contributing test cases towards forming MC/DC pairs. The third contribution is to prioritize these test cases by considering both their Contribution Index (CI) values and Fault Exposing Potential (FEP) values. The proposed approach is validated by experimenting on eighteen Java programs and achieved on an average ×1.67 times increase in the number of effective test cases that lead to an average increase in MC/DC score by 41.08%. We also achieved on an average 49.00% reduction rate to minimize the test suite size and finally prioritized the test cases, based on their prioritization index values.

Exploring Software Security Test Generation Techniques: Challenges and Opportunities

Ensuring the security of the software has raised concerns from the research community which triggered numerous approaches that tend to eliminate it. The process of ensuring the security of software includes the introduction of processes in the Software Development Life Cycle where one of them is testing after the software is developed. Manually testing software for security is a labor-intensive task. Therefore, it is required to automate the process of testing by generating test cases by automated techniques. In this paper, we review various software security test case generation approaches and techniques. We try to explore and classify the most eminent techniques for test case generation. The techniques are summarized and presented briefly to covers all researches work that has been done in the targeted classification. Moreover, this paper aims to depict the sound of security in the current state of the art of test case generation. The findings are summarized and discussed where the opportunities and challenges are revealed narratively. Although the paper intends to provide a comprehensive view of the research in test case generation, there was a noticeable lack in the test case generation from the security perspectives

Enabledness-based Testing of Object Protocols

A significant proportion of classes in modern software introduce or use object protocols, prescriptions on the temporal orderings of method calls on objects. This article studies search-based test generation techniques that aim to exploit a particular abstraction of object protocols (enabledness preserving abstractions (EPAs)) to find failures. We define coverage criteria over an extension of EPAs that includes abnormal method termination and define a search-based test case generation technique aimed at achieving high coverage. Results suggest that the proposed case generation technique with a fitness function that aims at combined structural and extended EPA coverage can provide better failure-detection capabilities not only for protocol failures but also for general failures when compared to random testing and search-based test generation for standard structural coverage.

Survey on Prioritizing Test Cases in Various Levels of the Software Development Life Cycle

Test case prioritization is a technical method to reorder the execution of test cases to reduce regression testing costs. This paper has examined various existing techniques that are widely used and suggests improving test case prioritization process after finding many research gaps. These research gaps are collected after doing a thorough study on 206 papers after surfing 310 papers on test case generation and prioritization techniques. These papers are collected from different electronic databases such as IEEE Explore, Science Direct, ACM Library, Springer, Wiley, and Elsevier. The authors have targeted to make a statistical record to show research contribution on test case prioritization at three levels of software development life cycle. This survey shows that 20.87% of papers are contributing for TCP at the requirement phase, 38.83% of papers are contributing for TCP at the design phase, 40.29% of papers are contributing to TCP at the coding phase. The inference of this study cites many future recommendations for the current researchers in the conclusion section.

Proposal of Iterative Genetic Algorithm for Test Suite Generation

Huge and reputed software industries are expected to deliver quality products. However, industry suffers from a loss of approximately $500 billion due to shoddy software quality. The quality of the product in terms of its accuracy, efficiency, and reliability can be revamped through testing by focusing attention on testing the product through effective test case generation and prioritization. The authors have proposed a test-case generation technique based on iterative listener genetic algorithm that generates test cases automatically. The proposed technique uses its adaptive nature and solves the issues like redundant test cases, inefficient test coverage percentage, high execution time, and increased computation complexity by maintaining the diversity of the population which will decrease the redundancy in test cases. The performance of the technique is compared with four existing test-case generation algorithms in terms of computational complexity, execution time, coverage, and it is observed that the proposed technique outperformed.

An Enhanced Multiview Test Case Generation Technique for Object-oriented Software using Class and Activity Diagrams

Software testing is one of the vital steps in software development life cycle. Test case generation is the first process in software testing which takes a lot of time, cost and effort to build an effective product from the start. Automatic test case generation is the best way to address this issue and model-based test case generation approach would be suitable for this automation process. One way to generate test cases automatically is by generating test cases from Unified Modeling Language (UML) models. The challenge with the existing test case generation techniques using UML models is that they provide a single view, meaning that the techniques capture a single aspect of the system, such as structural or behavioral but not both. In this paper, we have successfully developed a technique that automatically generates test cases which capture both structural and behavioral views of the system. These test cases can help to discover software faults early in the software development cycle. Finally, we conducted an experiment by comparing our technique with a manual process. The results show that the proposed technique can produce same test cases as manually writing test cases of the same system model but this technique saves a lot of time, effort and cost as well.

An evaluation of Monte Carlo-based hyper-heuristic for interaction testing of industrial embedded software applications

Hyper-heuristic is a new methodology for the adaptive hybridization of meta-heuristic algorithms to derive a general algorithm for solving optimization problems. This work focuses on the selection type of hyper-heuristic, called the exponential Monte Carlo with counter (EMCQ). Current implementations rely on the memory-less selection that can be counterproductive as the selected search operator may not (historically) be the best performing operator for the current search instance. Addressing this issue, we propose to integrate the memory into EMCQ for combinatorial t-wise test suite generation using reinforcement learning based on the Q-learning mechanism, called Q-EMCQ. The limited application of combinatorial test generation on industrial programs can impact the use of such techniques as Q-EMCQ. Thus, there is a need to evaluate this kind of approach against relevant industrial software, with a purpose to show the degree of interaction required to cover the code as well as finding faults. We applied Q-EMCQ on 37 real-world industrial programs written in Function Block Diagram (FBD) language, which is used for developing a train control management system at Bombardier Transportation Sweden AB. The results show that Q-EMCQ is an efficient technique for test case generation. Addition- ally, unlike the t-wise test suite generation, which deals with the minimization problem, we have also subjected Q-EMCQ to a maximization problem involving the general module clustering to demonstrate the effectiveness of our approach. The results show the Q-EMCQ is also capable of outperforming the original EMCQ as well as several recent meta/hyper-heuristic including modified choice function, Tabu high-level hyper-heuristic, teaching learning-based optimization, sine cosine algorithm, and symbiotic optimization search in clustering quality within comparable execution time.

Test Case Understandability Model

Several automated test case generation techniques have been proposed to date, although the adoption of such techniques in the industry remains low. A key factor that has contributed to this low adoption rate is the difficulty experienced by the developer in terms of reading and understanding automatically generated test cases. For this reason, it is essential to construct a test case understandability model for improving the generated test case. In the present paper, we extracted 20 test case metrics, six developer related metrics and two understandability proxies from a white-box test case classification experiment. Based on these metrics, we employed classification and regression algorithms to build test case understandability model. From the experiment, we can conclude that combined metrics always exhibit better discriminatory performance in classification models as well as a higher correlation in regression models when compared to a model that involved only test case metrics or developer metrics.

Automated optimal test data generation for OCL specification using harmony search algorithm

Exploring software testing possibilities at an early software life cycle is increasingly necessary to avoid the propagation of defects to the subsequent phases. This requirement demands technique that can generate automated test cases at the initial phases of software development. Thus, we propose a novel framework for automated test data generation using formal specifications written in object constraint language (OCL). We also defined a novel fitness function named exit-predicate-wise branch coverage (EPWBC) to evaluate the generated test data. Another focus of the proposed approach is to optimise the test case generation process by applying, harmony search (HS) algorithm. The experimental results indicate that the proposed framework outperforms the other OCL-based test case generation techniques. Furthermore, it has been inferred that OCL based testing adopting HS algorithm forms an excellent combination to produce more test coverage and an optimal test suite thereby improving the quality of a system.

A systematic mapping of test case generation techniques using UML interaction diagrams

AbstractModel‐based test case generation techniques provide a mechanism to derive tests systematically. This study provides a systematic mapping of test case generation techniques based on UML interaction diagrams. The study compares the test case generation techniques regarding their capabilities and limitations, and it also assesses the reporting quality of the primary studies. We can conclude that the studies presenting test case generation techniques using UML interaction diagrams were not following the guidelines for research methods (eg, case studies or experiments). Solutions were not empirically evaluated in industrial contexts. Our study revealed that better tool support is needed to introduce the UML interaction diagram–based test case generation techniques in the industry.

Test Case Generation Process using Soft Computing Techniques

Software testing is the SDLC's important and most expensive step. Software testing is difficult and time-consuming work requiring a great deal of money for software development. Testing is both an operation that is static and adaptive. Software testing process deals with the creation of test cases, checking and validating either passed or failed test cases. It is unidealistic to check only the discerning parts of the material as a whole at once. It is not possible to test the whole system once, so selected parts of the code are considered for analysis. Since the input space of the Product Under Test (PUT) can be very large, it is important to analyze a representative subset of test cases. During software testing, the most important task is to build appropriate test cases. An effective set of test cases can detect more errors. Software testing always requires high deficiencies. Test cases are constructed using the test data. In the automation of software testing, the important task is to generate test data according to a given level of competence. The improved test data are determined using the test case development methodology and the test data adequacy criterion being applied. For increase the level of automation and performance, these aspects of test case development need to be studied. This paper studies the various test case generation techniques using soft computing techniques like Genetic Algorithm, Artificial Bee colony methods. Further an evaluation criterion for the test case generation process, empirical study of Code Coverage and its importance is discussed.

A Systematic Study on Factors Impacting GUI Traversal-Based Test Case Generation Techniques for Android Applications

Many test case generation algorithms have been proposed to test Android apps through their graphical user interfaces. However, no systematic study on the impact of the core design elements in these algorithms on effectiveness and efficiency has been reported. This paper presents the first controlled experiment to examine three key design factors, each of which is popularly used in GUI traversal-based test case generation techniques. These three major factors are definition of GUI state equivalence, state search strategy, and waiting time strategy in between input events. The empirical results on 33 Android apps with real faults revealed interesting results. First, different choices of GUI state equivalence led to significant difference on failure detection rate and extent of code coverage. Second, searching the GUI state hierarchy randomly is as effective as searching it systematically. Last but not the least, the choices on when to fire the next input event to the app under test is immaterial so long as the length of the test session is practically long enough such as 1 h. We also found two new GUI state equivalence definitions that are statistically as effective as the existing best strategy for GUI state equivalence.

Test Case Minimization Applying Firefly Algorithm

The objective of this study is to propose a test case minimization method performed on UML statechart to produce test cases that are optimal while considering higher coverage criteria. Current test case generation techniques consume a large amount of time and cost with less testing coverage, while generating large number or test cases, what results in difficulties in detecting the faults and errors in the tested systems. Many approaches on test case minimization use genetic algorithms, model checking, or graph search algorithms to reduce the number of generated test cases, also the using of UML diagrams to test the system requirements and design before implementing it in the coding phase. However, these studies lack concentration in achieving higher coverage criteria and minimization in the generated test cases in the same time. The proposed test case minimization method has the following steps: provide weight to the paths, calculate path coverage for each path, transform an immediate graph into an adjacency matrix, which later is used to apply firefly algorithm and generate optimal test cases. A review on of the previous similar research in this domain has been presented and analysed to identify the issues and gaps in this domain. The steps use to perform the test case minimization have been presented together with some example and formula used. Findings from this study shows that this test case minimization has successful covered more types of test coverage which are all state, all transition, all transition pairs and all-one-loop paths. At the same time, it is capable of minimizing the number of test cases.