Test Case Generation Approach Research Articles

Multi‐objective optimization‐based and fault localization‐oriented test case generation for novice programs

SummaryOnline judgment (OJ) systems are capable of evaluating program results by automatically executing test cases, significantly improving the efficiency of traditional guidance approaches. Moreover, existing studies attempt to assist novices through automated fault localization techniques to provide feedback to novices, which can help them quickly find the location of faulty statements. Among them, spectrum‐based fault localization (SBFL) techniques have been widely used for their lightweight and efficiency, which only requires coverage information and test results of test cases to conduct fault localization. However, manually constructing high‐quality test cases for a large number of OJ questions is tough work to complete. To solve this problem, we propose the novice program‐oriented Multi‐Objective Optimization‐Based Fault Localization‐Oriented Test Case Generation (MFTCG) for automatically generating test inputs. Specifically, we use multi‐objective optimization algorithms to evolve the test case in terms of both fault localization and faulty code detection capability. We conduct experiments with 8911 programs from the well‐known public OJ platform AtCoder. The results show that our proposed approach MFTCG can achieve the best fault localization performance compared with existing automated test case generation approaches in most cases and can achieve the similar faulty code detection capability compared to manually designed test cases.

Finding Cross-Rule Optimization Bugs in Datalog Engines

Datalog is a popular and widely-used declarative logic programming language. Datalog engines apply many cross-rule optimizations; bugs in them can cause incorrect results. To detect such optimization bugs, we propose an automated testing approach called Incremental Rule Evaluation (IRE), which synergistically tackles the test oracle and test case generation problem. The core idea behind the test oracle is to compare the results of an optimized program and a program without cross-rule optimization; any difference indicates a bug in the Datalog engine. Our core insight is that, for an optimized, incrementally-generated Datalog program, we can evaluate all rules individually by constructing a reference program to disable the optimizations that are performed among multiple rules. Incrementally generating test cases not only allows us to apply the test oracle for every new rule generated—we also can ensure that every newly added rule generates a non-empty result with a given probability and eschew recomputing already-known facts. We implemented IRE as a tool named Deopt, and evaluated Deopt on four mature Datalog engines, namely Soufflé, CozoDB, μZ, and DDlog, and discovered a total of 30 bugs. Of these, 13 were logic bugs, while the remaining were crash and error bugs. Deopt can detect all bugs found by queryFuzz, a state-of-the-art approach. Out of the bugs identified by Deopt, queryFuzz might be unable to detect 5. Our incremental test case generation approach is efficient; for example, for test cases containing 60 rules, our incremental approach can produce 1.17× (for DDlog) to 31.02× (for Soufflé) as many valid test cases with non-empty results as the naive random method. We believe that the simplicity and the generality of the approach will lead to its wide adoption in practice.

Topological Data Analysis for Software Test Cases Generation

The escalating development of digital technologies over the last several decades has given rise to an accompanying surge in data analysis.The software based smart applications increase the importance of behaviour analysis. Software testing of expert systems such as electronic health records, health information system and software as a medical device (SaMD) refers to detect the difference between expected behaviour and actual outcome during healthcare expert systems development. Test cases are the core source of effective software testing. Test cases generation for expert systems are discover challenges of identifying the expected behaviour of the system, where the decision logic is obtained via a data –driven paradigm. In the traditional system software object oriented expected behaviour, provide the clear test cases does not change the flow such as in the healthcare system. Intelligent software test case generation approach required for smart health systems. In this research contributes key performance indicators from massive data sets, node –link diagrams from decision trees, test cases from adjacency matrix are elaborated. The experimental results of healthcare expert systems provide empirically evidence that topological data analysis are compact contribution for software requirement validation.

Software Test Case Generation Tools and Techniques: A Review

Software Industry is evolving at a very fast pace since last two decades. Many software developments, testing and test case generation approaches have evolved in last two decades to deliver quality products and services. Testing plays a vital role to ensure the quality and reliability of software products. In this paper authors attempted to conduct a systematic study of testing tools and techniques. Six most popular e-resources called IEEE, Springer, Association for Computing Machinery (ACM), Elsevier, Wiley and Google Scholar to download 738 manuscripts out of which 125 were selected to conduct the study. Out of 125 manuscripts selected, a good number approx. 79% are from reputed journals and around 21% are from good conference of repute. Testing tools discussed in this paper have broadly been divided into five different categories: open source, academic and research, commercial, academic and open source, and commercial & open source. The paper also discusses several benchmarked datasets viz. Evosuite 10, SF100 Corpus, Defects4J repository, Neo4j, JSON, Mocha JS, and Node JS to name a few. Aim of this paper is to make the researchers aware of the various test case generation tools and techniques introduced in the last 11 years with their salient features.

Generating Class-Level Integration Tests Using Call Site Information

Search-based approaches have been used in the literature to automate the process of creating unit test cases. However, related work has shown that generated unit-tests with high code coverage could be ineffective, i.e., they may not detect all faults or kill all injected mutants. In this paper, we propose CLING, an integration-level test case generation approach that exploits how a pair of classes, the caller and the callee, interact with each other through method calls. In particular, CLING generates integration-level test cases that maximize the Coupled Branches Criterion (CBC). Coupled branches are pairs of branches containing a branch of the caller and a branch of the callee such that an integration test that exercises the former also exercises the latter. CBC is a novel integration-level coverage criterion, measuring the degree to which a test suite exercises the interactions between a caller and its callee classes. We implemented CLING and evaluated the approach on 140 pairs of classes from five different open-source Java projects. Our results show that (1) CLING generates test suites with high CBC coverage, thanks to the definition of the test suite generation as a many-objectives problem where each couple of branches is an independent objective; (2) such generated suites trigger different class interactions and can kill on average 7.7% (with a maximum of 50%) of mutants that are not detected by tests generated at the unit level; (3) CLING can detect integration faults coming from wrong assumptions about the usage of the callee class (32 for our subject systems) that remain undetected when using automatically generated unit-level test suites.

An Approach of Test Case Generation with Software Requirement Ontology

An Approach of Test Case Generation with Software Requirement Ontology

Entropy Based Method for Malicious File Detection

Ransomware is by no means a recent invention, having existed as far back as 1989, yet it still poses a real threat in the 21st century. Given the increasing number of computer users in recent years, this threat will only continue to grow, affecting more victims as well as increasing the losses incurred towards the people and organizations impacted in a successful attack. In most cases, the only remaining courses of action open to victims of such attacks were the following: either pay the ransom or lose their data. One commonly shared behavior by all crypto ransomware strains is that there will be attempts to encrypt the victims’ files at a certain point during the ransomware execution. This paper demonstrates a technique that can identify when these encrypted files are being generated and is independent of the strain of the ransomware. Previous research has highlighted the difficulty in differentiating between compressed and encrypted files using Shannon entropy, as both file types exhibit similar values. Among the experiments described in this study, one showed a unique characteristic for the Shannon entropy of encrypted file header fragments, which was used to differentiate between encrypted files and other high entropy files such as archives. The Shannon entropy of encrypted file header fragments has a unique characteristic in one of the tests discussed in this study. This property was used to distinguish encrypted files from other files with high entropy, such as archives. To overcome this drawback, this study proposed an approach for test case generation by enhancing the entropy-based threat tree model, which would improve malicious file identification. The file identification was enhanced by combining three entropy algorithms, and the test case was generated based on the threat tree model. This approach was then evaluated using accuracy measurements: True Positive, True Negative, False Positive, False Negative. A promising result is expected. This method solves the challenge of leveraging file entropy to distinguish compressed and archived files from ransomware-encrypted files in a timely manner.

A Novel Mutation Analysis-Based Approach for Testing Parallel Behavioural Scenarios in Multi-Agent Systems

In this work, we propose a new test case generation approach that can cover behavioural scenarios individually in a multi-agent system. The purpose is to identify, in the case of the detection of an error, the scenario that caused the detected error, among the scenarios running in parallel. For this, the approach used, in the first stage, the technique of mutation analysis and parallel genetic algorithms to identify the situations in which the agents perform the interactions, presented in the sequence diagram, of the scenario under test only; these situations will be considered as inputs of the test case. In the second stage, the approach used the activities presented in the activity diagram to identify the outputs of the test case expected for its inputs. Subsequently, the generated test cases will be used for the detection of possible errors. The proposed approach is supported by a formal framework in order to automate its phases, and it is applied to a concrete case study to illustrate and demonstrate its usefulness.

Comprehensive Safety Evaluation of Highly Automated Vehicles at the Roundabout Scenario

A highly automated vehicle (HAV) is a safety-critical system. Therefore, a verification and validation (V&V) process that rigorously evaluates the safety of HAVs is necessary before their release to the market. In this paper, we propose an interaction-aware safety evaluation framework for the HAV and apply it to the roundabout entering scenario. Instead of assuming that the primary other vehicles (POVs) take predetermined maneuvers, we model the POVs as game-theoretic agents. To capture a wide variety of interactions between the POVs and the vehicle under test (VUT), we use level- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> game theory and social value orientation (SVO) to characterize the interactive behaviors and train a diverse library of POVs using reinforcement learning. The game-theoretic library, together with initial conditions, form a rich testing space for the two-POV roundabout scenario. On the other hand, we propose an adaptive test case generation scheme based on adaptive sampling, stochastic optimization and upper confidence bound (UCB) algorithm to efficiently generate customized challenging cases for the VUT from the testing space. In simulations, the proposed testing space design captured a wide range of interactive patterns at the roundabout scenario. The proposed test case generation scheme was found to cover the failure modes of the VUT more effectively compared to other test case generation approaches.

Automatic Cause–Effect Graph Tool with Informal Korean Requirement Specifications

In requirement engineering, it is a very important issue to generate test cases with natural language automatically. However, no test case tools deal with informal Korean requirement specifications. In the Korean military software system and airspace industrial area, it is strongly suggested to automatically make just 30% of all possible test cases with requirements. Unlike the previous approaches, we adapted Gary E. Mogyorodi’s cause-effect graphing approach and the model-driven architecture (MDA) approach for automatic test case generation with natural language. In order to generate test cases with informal Korean requirement specifications, we propose an automatic cause–effect tool as an intermediate model for (1) simplifying complicated requirements; (2) modeling the C3Tree (that is, condition and result); (3) identifying incomplete requirements; (4) constructing causes, effects, and relationships; and (5) integrating with two units (that is, similar causes or effects) to remove redundant requirements. We evaluated the accuracy of two generated cause–effect graphs in two ways. With our approach, we can also remove requirement redundancy.

An EFSM-Based Test Data Generation Approach in Model-Based Testing

Testing is an integral part of software development. Current fast-paced system developments have rendered traditional testing techniques obsolete. Therefore, automated testing techniques are needed to adapt to such system developments speed. Model-based testing (MBT) is a technique that uses system models to generate and execute test cases automatically. It was identified that the test data generation (TDG) in many existing model-based test case generation (MB-TCG) approaches were still manual. An automatic and effective TDG can further reduce testing cost while detecting more faults. This study proposes an automated TDG approach in MB-TCG using the extended finite state machine model (EFSM). The proposed approach integrates MBT with combinatorial testing. The information available in an EFSM model and the boundary value analysis strategy are used to automate the domain input classifications which were done manually by the existing approach. The results showed that the proposed approach was able to detect 6.62 percent more faults than the conventional MB-TCG but at the same time generated 43 more tests. The proposed approach effectively detects faults, but a further treatment to the generated tests such as test case prioritization should be done to increase the effectiveness and efficiency of testing.

Exploring Software Security Test Generation Techniques: Challenges and Opportunities

Ensuring the security of the software has raised concerns from the research community which triggered numerous approaches that tend to eliminate it. The process of ensuring the security of software includes the introduction of processes in the Software Development Life Cycle where one of them is testing after the software is developed. Manually testing software for security is a labor-intensive task. Therefore, it is required to automate the process of testing by generating test cases by automated techniques. In this paper, we review various software security test case generation approaches and techniques. We try to explore and classify the most eminent techniques for test case generation. The techniques are summarized and presented briefly to covers all researches work that has been done in the targeted classification. Moreover, this paper aims to depict the sound of security in the current state of the art of test case generation. The findings are summarized and discussed where the opportunities and challenges are revealed narratively. Although the paper intends to provide a comprehensive view of the research in test case generation, there was a noticeable lack in the test case generation from the security perspectives

An Enhanced Multiview Test Case Generation Technique for Object-oriented Software using Class and Activity Diagrams

Software testing is one of the vital steps in software development life cycle. Test case generation is the first process in software testing which takes a lot of time, cost and effort to build an effective product from the start. Automatic test case generation is the best way to address this issue and model-based test case generation approach would be suitable for this automation process. One way to generate test cases automatically is by generating test cases from Unified Modeling Language (UML) models. The challenge with the existing test case generation techniques using UML models is that they provide a single view, meaning that the techniques capture a single aspect of the system, such as structural or behavioral but not both. In this paper, we have successfully developed a technique that automatically generates test cases which capture both structural and behavioral views of the system. These test cases can help to discover software faults early in the software development cycle. Finally, we conducted an experiment by comparing our technique with a manual process. The results show that the proposed technique can produce same test cases as manually writing test cases of the same system model but this technique saves a lot of time, effort and cost as well.

Smart Test Case Quantifier Based on Mc/Dc Coverage Criterion with Uml Sequence Diagram

Software testing is an inevitable phase in the Software Development Life Cycle (SDLC). It plays a vital role in ensuring the quality of the product. Testing can be performed on requirement, design and code. Traditional test case generation approaches majorly focuses on designing and implementation models, as a huge percentage of errors in the software are caused due to the lack of understanding in the early phases. So, it’s advisory to follow testing in the initial phase in SDLC as most of the errors/faults can be eliminated and can be prevented without disseminating to the next phase. So, testing must not be secluded to a single phase alone in SDLC. In software testing, test case generation is the fundamental and challenging part. The major purpose to design test case is to create set of tests that are valuable in validation. Due to the increase in software size, it’s not quite advisable to have the choice as manual testing to test the code as its error prone, complex and time consuming. Automation of testing would help the tester to test effectively and timely. Applications of new techniques improve the test process and cut down the tester’s effort. The proposed work presents a sandwich of black box and white box testing. Design-based testing, a black box approach is implemented, that aids in fixing the errors at initial phase. Unified Modeling Language (UML) Use case diagram, Activity diagram and Sequence Diagram are considered for designing. The code-based generation of test cases, is a white box testing approach, test coverage criteria Modified Condition/Decision Condition (MC/DC) has been used to ensure the maximum coverage of the code during testing phase for generating the test cases. The objective of this paper is to automate the generation of minimal number of test cases required to test a system with maximum coverage by removing the redundant test cases using MC/DC criterion. This present an idea to the beginners of the testing about minimal test cases they necessitate for testing their system.

Application of particle swarm optimisation for coverage estimation in software testing

A Markov approach for test case generation and code coverage estimation using particle swarm optimisation technique is proposed. Initially, the DD-graph is taken from control flow graph of the software code by joining decision to decision. The DD-graph identifies the sequences of independent paths using c-uses and p-uses based on set theory approach and compared to cyclomatic complexity. Automatic test cases are generated and the nature of test cases are integer, float and Boolean variables. Using this initial test suite, the code coverage summary is generated using gcov code coverage analysis tool, the branch probability percentage is considered as TPM values with respect to each branch in the DD-graph. Path coverage is used as a fitness function which is the product of node coverage and TPM values. Iterate this algorithm until reaches 100% code coverage among each independent test path. The randomness of the proposed approach is compared to genetic algorithm.

Application of particle swarm optimisation for coverage estimation in software testing

A Markov approach for test case generation and code coverage estimation using particle swarm optimisation technique is proposed. Initially, the DD-graph is taken from control flow graph of the software code by joining decision to decision. The DD-graph identifies the sequences of independent paths using c-uses and p-uses based on set theory approach and compared to cyclomatic complexity. Automatic test cases are generated and the nature of test cases are integer, float and Boolean variables. Using this initial test suite, the code coverage summary is generated using gcov code coverage analysis tool, the branch probability percentage is considered as TPM values with respect to each branch in the DD-graph. Path coverage is used as a fitness function which is the product of node coverage and TPM values. Iterate this algorithm until reaches 100% code coverage among each independent test path. The randomness of the proposed approach is compared to genetic algorithm.

CPN-based Test Case Generation Approach for Testing BPEL-based Web Services Composition

CPN-based Test Case Generation Approach for Testing BPEL-based Web Services Composition

An Automated Approach to Generate Test Cases From Use Case Description Model

Test complexity and test adequacy are frequently raised by software developers and testing agents. However, there is little research works at this aspect on specification-based testing at the use case description level. Thus, this research proposes an automatic test cases generator approach to reduce the test complexity and to enhance the percentage of test coverage. First, to support the infrastructure for performing automatic, this proposed approach refines the use cases using use case describing template and save it in the text file. Then, the saved file is input to the Algorithm of Control Flow Diagram (ACFD) to convert use case details to a control flow diagram. After that, the Proposed Tool of Generating Test Paths (PTGTP) is used to generate test cases from the control flow diagram. Finally, the genetic algorithm associated with transition coverage is adapted to optimize and evaluate the adequacy of such test cases. A money withdrawal use case in the ATM system is used as an ongoing case study. Preliminary results show that the generated test cases achieve high coverage with an optimal test case. This automatic test case generation approach is effective and efficient. Therefore, it could promote to use other test case coverage criteria.

Tester versus Bug: A Generic Framework for Model-Based Testing via Games

We propose a generic game-based approach for test case generation. We set up a game between the tester and the System Under Test, in such a way that test cases correspond to game strategies, and the conformance relation ioco corresponds to alternating refinement. We show that different test assumptions from the literature can be easily incorporated, by slightly varying the moves in the games and their outcomes. In this way, our framework allows a wide plethora of game-theoretic techniques to be deployed for model based testing.

Automatic test cases generation from business process models

Traditional test case generation approaches focus on design and implementation models while a large percentage of software errors are caused by the lack of understanding in the early phases. One of the most important models in the early phases of software development is business process model which closely resembles the real world and captures the requirements precisely. The aim of this paper is to present a model-based approach to automatically generate test cases from business process models. We first model business processes and convert them to state graphs. Then, the graphs are traversed and transformed to the input format of the “Spec explorer” tool that generates the test cases. Furthermore, we conduct a study to evaluate the impact of process characterizations on the performance of the proposed method.