Test Case Generation Process Research Articles

Optimized automated testing: test case generation and maintenance using latent semantic analysis-based TextRank and particle swarm optimization algorithms

Software development would have to include automated testing to ensure the finished product and performs as intended. However, the process of Test Case Generation and Maintenance can be time-consuming and error-prone, especially when manual methods are used. This research proposes a new approach to improve the efficiency and accuracy of automated testing using latent semantic analysis (LSA)-based TextRank (TR) and particle swarm optimization (PSO) algorithms. The study aims to evaluate the effectiveness of these algorithms in generating and optimizing test cases based on requirements analysis. To retrieve key information from the criteria, methods including text classification (TC), named entity recognition (NER), and sentiment analysis (SA) are used to evaluate the text. Test cases are then generated using LSA-based TR for text summarization and PSO for optimization. The aim of this work is to identify any limitations that need to be addressed and to evaluate the overall efficiency and accuracy of automated testing (AT) using proposed algorithms. The results of this research are expected to have important implications for the software industry, helps to improve the overall efficiency and accuracy of AT. The findings could guide future research that led to the creation of more advanced and effective tools for AT.

ARTE: Automated Generation of Realistic Test Inputs for Web APIs

Automated test case generation for web APIs is a thriving research topic, where test cases are frequently derived from the API specification. However, this process is only partially automated since testers are usually obliged to manually set meaningful valid test inputs for each input parameter. In this article, we present ARTE, an approach for the automated extraction of realistic test data for web APIs from knowledge bases like DBpedia. Specifically, ARTE leverages the specification of the API parameters to automatically search for realistic test inputs using natural language processing, search-based, and knowledge extraction techniques. ARTE has been integrated into RESTest, an open-source testing framework for RESTful APIs, fully automating the test case generation process. Evaluation results on 140 operations from 48 real-world web APIs show that ARTE can efficiently generate realistic test inputs for 64.9% of the target parameters, outperforming the state-of-the-art approach SAIGEN (31.8%). More importantly, ARTE supported the generation of over twice as many valid API calls (57.3%) as random generation (20%) and SAIGEN (26%), leading to a higher failure detection capability and uncovering several real-world bugs. These results show the potential of ARTE for enhancing existing web API testing tools, achieving an unprecedented level of automation.

A deep learning-based automated framework for functional User Interface testing

The use of automation tools in software testing helps keep pace with the timeline of the deliverables. Over time with the inclusion of continuous integration/continuous delivery (CI/CD) pipelines, automation tools are becoming less effective. The testing community is turning to AI to help keep the pace. We study the use of transformers to automate the process of test case generation directly from the User Interface (UI) element description instead of relying on the test specification document from which test cases are extracted manually. We also demonstrate the capability of the proposed approach in repairing flaky tests. We employ object detection algorithms EfficientDet and DEtectionTRansformer for detecting the elements from an application UI automatically without requiring a tester to locate complex-scripted UI elements. We also use Tesseract to automatically identify the text present on the UI elements. We transform the generated UI element description to actual test designer-written test cases using text-generation transformers like GPT-2 and T5 . The generated test cases are then translated into executable test scripts using a simple parser. We carry out our cases study on 30 e-commerce applications. The percentage of correct executable test cases generated by the framework employing EfficientDet is 93.82% and employing DEtectionTRansformer is 98.08% . The framework eliminates an average of 96.05% flakiness across the applications selected for the study. It is concluded that the proposed approach can be used with current automation tools in the industry to enhance their capability in generating test cases and repairing the flaky tests.

Automated Web application testing driven by pre-recorded test cases

There are fully automated approaches proposed for Web application testing. These approaches mainly rely on tools that explore an application by crawling it. The crawling process results in a state transition model, which is used for generating test cases. Although these approaches are fully automated, they consume too much time and they usually require manual configuration. This is due to the lack of insight and domain knowledge of crawling tools regarding the application under test . We propose a semi-automated approach instead. We introduce a tool that takes a set of recorded event sequences as input. These sequences can be captured during exploratory tests. They are replayed as pre-recorded test cases. They are also exploited for steering the crawling and test case generation process. We performed a case study with 5 Web applications. These applications were randomly tested with state-of-the-art tools. Our approach can reduce the crawling time by hours, while compromising the coverage achieved by 0.2% to 7.43%. In addition, our tool does not require manual configuration before crawling. The input for the tool was created within 15 min of exploratory testing. • Exploratory testing is used for improving automated Web application testing. • Recorded test cases are used for configuring and steering the crawling process. • Application specific configurations are possible without technical expertise. • 15 min of exploratory testing can reduce the crawling time by hours.

Effective Covering Array Generation Using an Improved Particle Swarm Optimization

In the test case generation process of combinatorial testing, particle swarm optimization (PSO) is widely concerned for its simple implementation and fast convergence rate; however, it often falls into local optimum due to premature convergence. To attack this problem, a novel adaptive value measurement strategy is adopted by weighing the relationship between current test cases and historical test cases. The test case with the minimum average hamming distance is selected as the optimal test case, and the inertial weight linear differential decrease strategy is developed to ensure better inertial weight in different search stages, further to improve the capability of generating smaller covering arrays. In addition, we integrate the simulated annealing strategy into the improved PSO to improve the ability of particles jumping out of the local optimum, and an innovative approach for generating a better covering array is proposed. Experiments on 16 classical random strength covering arrays suggest that our approach outperforms six other techniques in terms of effectiveness.

MT-ART: A Test Case Generation Method Based on Adaptive Random Testing and Metamorphic Relation

Most of metamorphic testing (MT) research works focused on the generation and application of metamorphic relations (MRs). There is no clear conclusion about the relationship between test case generation methods and performance of MT. In this article, we introduce a novel method based on adaptive random testing (ART) and MR for MT test case generation. It proposes a family of algorithms for MT test cases generation, named as MT based ART (MT-ART). Three distances are measured to generate the next MT test case. In order to verify the performance of this method, series of experiments on four programs with different numbers of inputs are introduced. The results show that MT-ART performs better than other ART algorithms not only in test effectiveness, but also in test efficiency and test coverage. Based on this article, the following conclusions can be drawn: first, considering the effectiveness of MRs and test cases in MT may lead to better results. In this way, most of the existing research can be improved by this method. This is the most important contribute of our research. Second, not only the source test cases, but also the follow-up test cases can improve the performance of MT. Therefore, they should be considered together during the process of the next test case generation. Third, the average distance performs better than the max distance and the minimum distance in metamorphic test case selection.

Applying Heuristics to Generate Test Cases for Automated Driving Safety Evaluation

Comprehensive safety evaluation methodologies for automated driving systems that account for the large complexity real traffic are currently being developed. This work adopts a scenario-based safety evaluation approach and aims at investigating an advanced methodology to generate test cases by applying heuristics to naturalistic driving data. The targeted requirements of the generated test cases are severity, exposure, and realism. The methodology starts with the extraction of scenarios from the data and their split in two subsets—containing the relatively more critical scenarios and, respectively, the normal driving scenarios. Each subset is analysed separately, in regard to the parameter value distributions and occurrence of dependencies. Subsequently, a heuristic search-based approach is applied to generate test cases. The resulting test cases clearly discriminate between safety critical and normal driving scenarios, with the latter covering a wider spectrum than the former. The verification of the generated test cases proves that the proposed methodology properly accounts for both severity and exposure in the test case generation process. Overall, the current study contributes to fill a gap concerning the specific applicable methodologies capable of accounting for both severity and exposure and calls for further research to prove its applicability in more complex environments and scenarios.

Automated Specification-Based Testing of REST APIs.

Nowadays, REpresentational State Transfer Application Programming Interfaces (REST APIs) are widely used in web applications, hence a plethora of test cases are developed to validate the APIs calls. We propose a solution that automates the generation of test cases for REST APIs based on their specifications. In our approach, apart from the automatic generation of test cases, we provide an option for the user to influence the test case generation process. By adding user interaction, we aim to augment the automatic generation of APIs test cases with human testing expertise and specific context. We use the latest version of OpenAPI 3.x and a wide range of coverage metrics to analyze the functionality and performance of the generated test cases, and non-functional metrics to analyze the performance of the APIs. The experiments proved the effectiveness and practicability of our method.

Bio Inspired Approach for Generating Test data from User Stories

In Agile model where the software prototypes are developed frequently and also rapidly, testing becomes more critical. Generating an effective Test case for complex system is a challenging task involved in software testing. The major research challenge in this area includes the test case generation with limited resources, identifying the essential functional requirement that plays a crucial role and automation of the test case generation process. To solve this issue, a hybridized bio inspired approach is proposed to generate test cases from the user stories which accepts the business requirements as input, processed using NLP and develop functional test cases from it. The proposed algorithm is compared with other existing algorithms and the experimental results proved that the proposed algorithm is more efficient in many cases.

DEO: A Dynamic Event Order Strategy for t-way Sequence Covering Array Test Data Generation

Sequence covering array (SCA) generation is an active research area in recent years. Unlike the sequence-less covering arrays (CA), the order of sequence varies in the test case generation process. This paper reviews the state-of-the-art of the SCA strategies, earlier works reported that finding a minimal size of a test suite is considered as an NP-Hard problem. In addition, most of the existing strategies for SCA generation have a high order of complexity due to the generation of all combinatorial interactions by adopting one-test-at-a-time fashion. Reducing the complexity by adopting one-parameter- at-a-time for SCA generation is a challenging process. In addition, this reduction facilitates the supporting for a higher strength of coverage. Motivated by such challenge, this paper proposes a novel SCA strategy called Dynamic Event Order (DEO), in which the test case generation is done using one-parameter-at-a-time fashion. The details of the DEO are presented with a step-by-step example to demonstrate the behavior and show the correctness of the proposed strategy. In addition, this paper makes a comparison with existing computational strategies. The practical results demonstrate that the proposed DEO strategy outperforms the existing strategies in term of minimal test size in most cases. Moreover, the significance of the DEO increases as the number of sequences increases and/ or the strength of coverage increases. Furthermore, the proposed DEO strategy succeeds to generate SCAs up to t=7. Finally, the DEO strategy succeeds to find new upper bounds for SCA. In fact, the proposed strategy can act as a research vehicle for variants future implementation.

Test Suit Generation for Object Oriented Programs: A Hybrid Firefly and Differential Evolution Approach

In model-based testing, the test suites are derived from design models of system specification documents instead of actual program codes to reduce cost and time of testing. In search-based software testing approach, the nature inspired meta-heuristic search algorithms are used for automating and optimizing the test suite generation process of software testing. This paper proposes a concrete model-based testing framework; using UML behavioral state chart model along with the hybrid version of the two most popular nature inspired algorithms, Firefly algorithm (FA) and Differential Algorithm (DE). The hybrid algorithm is adopted to generate optimized test suits for the benchmark triangle classification problem. Experimental results evidently show that the hybrid FA-DE search algorithm outperforms the individual model-based Firefly and Differential Evolution algorithm's performances in terms of time complexity, better exploration and exploitation as well as variations in test case generation process. The framework generates optimized test data for complete transition path coverage of the available feasible paths of the example problem.

Automated optimal test data generation for OCL specification using harmony search algorithm

Exploring software testing possibilities at an early software life cycle is increasingly necessary to avoid the propagation of defects to the subsequent phases. This requirement demands technique that can generate automated test cases at the initial phases of software development. Thus, we propose a novel framework for automated test data generation using formal specifications written in object constraint language (OCL). We also defined a novel fitness function named exit-predicate-wise branch coverage (EPWBC) to evaluate the generated test data. Another focus of the proposed approach is to optimise the test case generation process by applying, harmony search (HS) algorithm. The experimental results indicate that the proposed framework outperforms the other OCL-based test case generation techniques. Furthermore, it has been inferred that OCL based testing adopting HS algorithm forms an excellent combination to produce more test coverage and an optimal test suite thereby improving the quality of a system.

Test Case Generation Process using Soft Computing Techniques

Software testing is the SDLC's important and most expensive step. Software testing is difficult and time-consuming work requiring a great deal of money for software development. Testing is both an operation that is static and adaptive. Software testing process deals with the creation of test cases, checking and validating either passed or failed test cases. It is unidealistic to check only the discerning parts of the material as a whole at once. It is not possible to test the whole system once, so selected parts of the code are considered for analysis. Since the input space of the Product Under Test (PUT) can be very large, it is important to analyze a representative subset of test cases. During software testing, the most important task is to build appropriate test cases. An effective set of test cases can detect more errors. Software testing always requires high deficiencies. Test cases are constructed using the test data. In the automation of software testing, the important task is to generate test data according to a given level of competence. The improved test data are determined using the test case development methodology and the test data adequacy criterion being applied. For increase the level of automation and performance, these aspects of test case development need to be studied. This paper studies the various test case generation techniques using soft computing techniques like Genetic Algorithm, Artificial Bee colony methods. Further an evaluation criterion for the test case generation process, empirical study of Code Coverage and its importance is discussed.

FastSyzkaller: Improving Fuzz Efficiency for Linux Kernel Fuzzing

Linux is one of the most popular operating systems and enjoys a large number of subscribers on a world-wide scale. To ensure its stability and reliability, fuzzing is an effective method to trigger crashes hidden in the Linux kernel. In this paper, we apply N-Gram model to extract vulnerable program behaviours and dig out vulnerable patterns to guide the test case generation phase of the traditional fuzzing technique so as to improve the fuzzing efficiency from the Linux system call aspect. The experiment is implemented on the basis of a Google well-known open-source project Syzkaller, a fuzzer that targeted to the Linux kernel and other OS kernels as well. It is an unsupervised coverage-guided kernel fuzzer, set up with a manager, a fuzzer and an executor. This paper proposes FastSyzkaller which combines Syzkaller with N-Gram model in its fuzzer to optimize the test case generation process to improve the fuzzing efficiency. In 4 weeks, FastSyzkaller exposes 29 different crash types. However, Syzkaller only exposes 12 types. FastSyzkaller also produces crashes 3x faster than Syzkaller, also with 3x more unique crashes from the quantity perspective.

Increasing test efficiency by risk-driven model-based testing

Abstract We introduce an approach and a tool, RIMA , for adapting test models used for model-based testing to augment information regarding failure risk. We represent test models in the form of Markov chains. These models comprise a set of states and a set of state transitions that are annotated with probability values. These values steer the test case generation process, which aims at covering the most probable paths. RIMA refines these models in 3 steps. First, it updates transition probabilities based on a collected usage profile. Second, it updates the resulting models based on fault likelihood at each state, which is estimated based on static code analysis. Third, it performs updates based on error likelihood at each state, which is estimated with dynamic analysis. The approach is evaluated with two industrial case studies for testing digital TVs and smart phones. Results show that the approach increases test efficiency by revealing more faults in less testing time.

Automated Optimal Test Data Generation for OCL Specification Using Harmony Search Algorithm

Exploring software testing possibilities at an early software life cycle is increasingly necessary to avoid the propagation of defects to the subsequent phases. This requirement demands technique that can generate automated test cases at the initial phases of software development. Thus, we propose a novel framework for automated test data generation using formal specifications written in object constraint language (OCL). We also defined a novel fitness function named exit-predicate-wise branch coverage (EPWBC) to evaluate the generated test data. Another focus of the proposed approach is to optimise the test case generation process by applying, harmony search (HS) algorithm. The experimental results indicate that the proposed framework outperforms the other OCL-based test case generation techniques. Furthermore, it has been inferred that OCL based testing adopting HS algorithm forms an excellent combination to produce more test coverage and an optimal test suite thereby improving the quality of a system.

Comparative Evaluation of the State-of-art Requirements-based Test Case Generation Approaches

The overall aim of software testing is to deliver the error-free and high-quality software products to the end users. The testing process ensures that a software is aligned with the user specification and requirements. In software testing process, there are many challenging tasks however test case generation process is considered as the most challenging one. The quality of the generated test cases has a significant impact on efficiency and effectiveness of the testing process. In order to improve the quality of a developed software, the test cases should be able to achieve maximum adequacy in the testing and requirements' coverage. This paper presents a comparative evaluation of the prominent requirement-based test case generation approaches. Five evaluation criteria namely, inputs for test case generation, transformation techniques, coverage criteria, time and tool's support are defined to systematically compare the approaches. The results of the evaluation are used to identify the gap in the current approaches and research opportunities in requirement-based test case's generation.

QuickFuzz testing for fun and profit

Abstract Fuzzing is a popular technique to find flaws in programs using invalid or erroneous inputs but not without its drawbacks. At one hand, mutational fuzzers require a set of valid inputs as a starting point, in which modifications are then introduced. On the other hand, generational fuzzing allows to synthesize somehow valid inputs according to a specification. Unfortunately, this requires to have a deep knowledge of the file formats under test to write specifications of them to guide the test case generation process. In this paper we introduce an extended and improved version of QuickFuzz, a tool written in Haskell designed for testing unexpected inputs of common file formats on third-party software, taking advantage of off-the-self well known fuzzers. Unlike other generational fuzzers, QuickFuzz does not require to write specifications for the file formats in question since it relies on existing file-format-handling libraries available on the Haskell code repository. It supports almost 40 different complex file-types including images, documents, source code and digital certificates. In particular, we found QuickFuzz useful enough to discover many previously unknown vulnerabilities on real-world implementations of web browsers and image processing libraries among others.

GECOJAP: A novel source-code preprocessing technique to improve code coverage

Abstract Safety critical standards such as DO178B/DO178C/ RTCA (Radio Technical Commission for Aeronautics) mandates coverage based testing in Aerospace applications. These standards mandate Level A certification for Modified Condition/Decision Coverage (MC/DC). To perform exhaustive and rigorous testing, concolic testing is used in the testing phase of the software development life cycle. But, still some concolic testers need to improve their performance, so that they can achieve higher coverage. We present an automated Java code transformation technique that can be used as a front-end to concolic testing tool for achieving high coverage. We have developed our tool using four modules. The tool named GEaring COverage for JAva Program (GECOJAP) for implementation of our approach. The first module shows a source code preprocessing technique called JEX-NCT (Java Exclusive-NOR Code Transformer) that inserts dummy branches according to Modified Condition / Decision Coverage (MC/DC) criterion. The second module represents a concolic tester named jCUTE (an open source tool) we used to generate test cases. The third module presents computation of MC/DC% using the generated test cases and original program. The fourth module shows the speed calculator that measures speed of test case generation, GECOJAP is more powerful and efficient in comparison to the existing techniques in terms of code transformation. Using GECOJAP one can, achieve higher code coverage. Also, GECOJAP results in the time and speed of the test case generation process. Our experimentation on ten Java programs for thirty executions shows that our approach achieves higher Branch Coverage and MC/DC over traditional concolic testers by 13.79% and 19% respectively.

Automated and Optimized Software Test Suite Generation Technique for Structural Testing

Software testing consumes 50% of total software development cost. Test case design gains central importance in testing activity with respect to quality. The manual test suite generation is a time consuming and tedious task which needs automation. Unit testing is normally done in stringent time schedules by the developers or rarely by testers. In structural testing, it is not possible to check exhaustively all possible test data and the quality of test is dependent heavily on the performance of single developer or tester. Thus automation and optimization is required in generating test data to assist developer or tester with the selection of appropriate test data. A novel hybrid technique is developed to automate the test suite generation process for branch coverage criteria using evolutionary testing. The hybrid technique applies both Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) to automatically generate test data. This technique improves efficiency and effectiveness of test case generation process when compared to applying Genetic Algorithm or Particle Swarm Optimization alone. The performance of proposed technique is evaluated and is observed that hybrid technique reduces the number of iterations by 47% when compared to GA and PSO applied separately and it reduces the execution time by 52% than GA and 48% than PSO.