Test Coverage Criteria Research Articles

Hybridizing genetic algorithm and harmony search for higher-order mutation testing

Mutation testing can evaluate the quality of the test inputs, generate test data, and simulate any test coverage criterion. Genetic algorithms and harmony search have been applied to reduce the cost of generating test inputs. Although hybridizing search algorithms enhances the efficiency of searching the solution domain, there is a shortage of applying the hybrid search techniques in mutation testing. This paper merges the genetic and harmony search algorithms to effectively generate test data to kill higher-order mutants. In addition, the performance of the proposed technique will be evaluated and compared with a stand-alone genetic algorithm and a stand-alone harmony search algorithm through an empirical study using a set of benchmark programs. The experimental study shows that the proposed technique outperformed the compared algorithms, reaching a higher killing ratio, where the proposed approach kills 92.8% of higher-order mutants for all tested programs. In comparison, GA kills 88.7%, and HA kills 86.6%. Besides, the proposed algorithm overcame the compared algorithm in reaching a targeted killing ratio faster than the compared algorithms. HGA reduced the execution time for each program with a reduction ratio ranging from 58.9% to 89.8%.

Effective Test Cases Generation with Harmony Search and RBF Neural Network

Software testing is one of the integral activities during development of software products. Generation and selection of the test cases either in static or dynamic form play pivot role for ensuring the quality of software products. There are numerous approaches in the literature for automatic generation of test cases but coverage criteria and fault detection rate are prominent metrics for checking the effectiveness of the software products during testing phase of software development. In the present, a new Harmony Radial Testing (HRT) is proposed by combining the concepts of Harmony Search Algorithm (HSA) and Radial Basis Function-Neural Network (RBF-NN) approaches. The main objective of the proposed HRT method is to generate automatic test cases by considering the criteria of branch coverage with improvement in the Maximum branch Coverage (MaxC), Average Coverage (AC) and Average Percentage Fault Detection (APFD) rates. The proposed approach combined with the Radial Basis Function (RBF), denoted as a HRT approach. The proposed approach is used to optimize harmony search over the randomly selected sample test cases, training the RBF-NN to simulate the fitness function. Seven Python codes have been tested through proposed approach and computed results are compared with Primal-Dual Genetic Algorithm (PDGA), Simple Genetic Algorithm (SGA) and random methods. It is observed that the proposed HRT algorithm optimizes consistently yielded reliable results, which may be used in future for enriching the software testing process by the software industries.

Generalized Coverage Criteria for Combinatorial Sequence Testing

We present a new model-based approach for testing systems that use sequences of actions and assertions as test vectors. Our solution includes a method for quantifying testing quality, a tool for generating high-quality test suites based on the coverage criteria we propose, and a framework for assessing risks. For testing quality, we propose a method that specifies generalized coverage criteria over sequences of actions, which extends previous approaches. Our publicly available tool demonstrates how to extract effective test suites from test plans based on these criteria. We also present a Bayesian approach for measuring the probabilities of bugs or risks, and show how this quantification can help achieve an informed balance between exploitation and exploration in testing. Finally, we provide an empirical evaluation demonstrating the effectiveness of our tool in finding bugs, assessing risks, and achieving coverage.

Assessing business process models: a literature review on techniques for BPMN testing and formal verification

PurposeTesting business processes is crucial to assess the compliance of business process models with requirements. Automating this task optimizes testing efforts and reduces human error while also providing improvement insights for the business process modeling activity. The primary purposes of this paper are to conduct a literature review of Business Process Model and Notation (BPMN) testing and formal verification and to propose the Business Process Evaluation and Research Framework for Enhancement and Continuous Testing (bPERFECT) framework, which aims to guide business process testing (BPT) research and implementation. Secondary objectives include (1) eliciting the existing types of testing, (2) evaluating their impact on efficiency and (3) assessing the formal verification techniques that complement testing.Design/methodology/approachThe methodology used is based on Kitchenham's (2004) original procedures for conducting systematic literature reviews.FindingsResults of this study indicate that three distinct business process model testing types can be found in the literature: black/gray-box, regression and integration. Testing and verification approaches differ in aspects such as awareness of test data, coverage criteria and auxiliary representations used. However, most solutions pose notable hindrances, such as BPMN element limitations, that lead to limited practicality.Research limitations/implicationsThe databases selected in the review protocol may have excluded relevant studies on this topic. More databases and gray literature could also be considered for inclusion in this review.Originality/valueThree main originality aspects are identified in this study as follows: (1) the classification of process model testing types, (2) the future trends foreseen for BPMN model testing and verification and (3) the bPERFECT framework for testing business processes.

Specialized path-based technique to test Internet of Things system functionality under limited network connectivity

Contemporary Internet-of-Things (IoT) systems are hindered by several reliability-related issues, especially, the dynamic behavior of IoT systems caused by limited and often unstable network connectivity. Several intuitive ad-hoc approaches can be employed to test this behavior; however, the effectiveness of these approaches in detecting defects and their overall testing costs remain questionable. Therefore, we present a new specialized path-based technique to test the processes of an IoT system in scenarios wherein parts of these processes are influenced by limited or disrupted network connectivity. The proposed technique can be scaled using two levels of test coverage criteria to determine the strengths of the test cases. For this purpose, we propose two algorithms for generating test cases to implement the technique: an ant colony optimization-based search and a graph-traversal-based test case composition. We compared the efficiency of the proposed approach with possible solutions obtained using a standard path-based testing approach based on prime paths computed by a set-covering algorithm. We consider the total number of test case steps as the main proxy for test effort in experiments employing 150 problem models. For the less intensive of the two used test-coverage criteria, EachBorderOnce, an ant colony optimization-based algorithm, produced test sets with the same averaged number of steps as the graph traversal-based test-case composition; however, this algorithm performed with averaged number of steps 10% lower than a prime paths-based algorithm. For the more intensive test coverage criterion, AllBorderCombinations, these differences favoring the ant colony optimization-based algorithm were 18% and 25%, respectively. For these two types of defined test coverage criteria, the ant colony optimization-based search, graph-traversal-based algorithm, and standard path-based testing approach based on prime paths achieved the best results for 93 and 78, 14 and 24, and 13 and 17 models for AllBorderCombinations and EachBorderOnce criterion, respectively. Therefore, to guarantee the best test set, all compared algorithms are combined in a portfolio strategy that yields the best results based on the potential of the produced test sets to detect simulated defects caused by limited network connectivity. Additionally, this portfolio strategy also yields test sets, implying the lowest test effort for experimental problem instances.

COSMOS: A comprehensive framework for automatically generating domain-oriented test suite

Coverage criteria are satisfied by at least one examination of the test target, while many faults are not revealed by one execution. However, despite executing the faulty statement, the test result is correct in certain circumstances. Such coincidentally passing test cases execute the faulty statement but do not cause failures. This paper introduces the new concept of domain solver. Domain solvers attempt to detect the domain of inputs rather than a single input satisfying a path constraint. Domain coverage is a new metric to evaluate the relative accuracy of the detected domains. The promising point is that the proposed approach similarly treats nonlinear and linear constraints. Domain solver splits a path constraint into conjunctions of simple conditions comparing two expressions. Such a splitting simplifies the constraint-solving task to detect regions of the input space satisfying a comparison between two expressions. After finding a region, an improved version of an algorithm, update, is used to determine the domain of variables involved in the comparing expressions. Our proposed approach, COSMOS, is implemented using the Roslyn compiler. We compared COSMOS with well-known constraint solvers using various linear/nonlinear constraints. The results show that COSMOS improves the number of supported data types involved in a constraint and solves100% of the instances in which the other solvers fail. Besides, COSMOS achieves the best relative accuracy of 84% compared to the existing domain-oriented test suite generation approaches. Moreover, our experiment results illustrate that COSMOS improves the fault-finding capability of other existing test coverage criteria by detecting coincidentally correct test cases. Combining domain coverage and compiler as a service makes a powerful constraint satisfaction method outperforming the existing constraint solvers regarding the number of solved linear/nonlinear constraints. Augmenting other structural test coverage criteria with domain coverage reveals the coincidentally correct test cases.

Novel Strategy Generating Variable-Length State Machine Test Paths

Finite State Machine is a popular modeling notation for various systems, especially software and electronic. Test paths (TPs) can be automatically generated from the system model to test such systems using a suitable algorithm. This paper presents a strategy that generates TPs and allows to start and end TPs only in defined states of the finite state machine. The strategy also simultaneously supports generating TPs only of length in a given range. For this purpose, alternative system models, test coverage criteria, and a set of algorithms are developed. The strategy is compared with the best alternative based on the reduction of the test set generated by the established N-switch coverage approach on a mix of 171 industrial and artificially generated problem instances. The proposed strategy outperforms the compared variant in a smaller number of TP steps. The extent varies with the used test coverage criterion and preferred TP length range from none to two and half fold difference. Moreover, the proposed technique detected up to 30% more simple artificial defects inserted into experimental SUT models per one test step than the compared alternative technique. The proposed strategy is well applicable in situations where a possible TP starts and ends in a state machine needs to be reflected and, concurrently, the length of the TPs has to be in a defined range.

TORACLE 2021 Workshop Summary

Testing is an important activity in software engineering, especially with the growing adoption of software systems in safety-critical domains. Testing research, however, is mostly focused on deter- mining which test inputs to use (e.g., proposing and evaluating test coverage criteria or automatic test generation tools). Re- gardless of the used coverage criterion, we need to know whether a given program executes correctly on a given input. Indeed, a test execution for which we cannot discriminate between success or failure serves little purpose. This corresponds to the so-called øracle problem", the problem of knowing whether a program be- haves correctly for a speci c input. Although the importance of the oracle problem is well understood, only a few alternatives exist to manually deriving test oracles. This makes automated test or- acle generation one of the main bottlenecks for achieving full test automation. Therefore, novel approaches and tools are needed to address this important problem.

N-Switch and All-Path Test Coverage Criterion for Extended Finite State Machine

N-Switch and All-Path Test Coverage Criterion for Extended Finite State Machine

A Machine Learning Based Framework for Verification and Validation of Massive Scale Image Data

Big data validation and system verification are crucial for ensuring the quality of big data applications. However, a rigorous technique for such tasks is yet to emerge. During the past decade, we have developed a big data system called CMA for investigating the classification of biological cells based on cell morphology which is captured in diffraction images. CMA includes a collection of scientific software tools, machine learning algorithms, and a large-scale cell image repository. In order to ensure the quality of big data system CMA, we developed a framework for rigorously validating the massive scale image data as well as adequately verifying both the software tools and machine learning algorithms. The validation of big data is conducted by iteratively selecting the data using a machine learning approach. An experimental approach guided by a feature selection algorithm is introduced in the framework to select an optimal feature set for improving the machine learning performance. The verification of software and algorithms is developed on the iterative metamorphic testing approach due to the non-testable property of the software and algorithms. A machine learning approach is introduced for developing test oracles iteratively to ensure the adequacy of the test coverage criteria. Performance of the machine learning algorithm is evaluated with a stratified N-fold cross validation and confusion matrix. We describe the design of the proposed big data verification and validation framework with CMA as the case study, and demonstrate its effectiveness through verifying and validating the dataset, the software and the algorithms in CMA.

Testing coverage criteria for optimized deep belief network with search and rescue

A new data-driven programming model is defined by the deep learning (DL) that makes the internal structure of a created neuron system over a fixed of training data. DL testing structure only depends on the data labeling and manual group. Nowadays, a lot of coverage criteria have been developed, but these criteria basically count the neurons' quantity whose activation during the implementation of a DL structure fulfilled certain properties. Also, existing criteria are not adequately fine-grained to capture delicate behaviors. This paper develops an optimized deep belief network (DBN) with a search and rescue (SAR) algorithm for testing coverage criteria. For an optimal selection of DBN structure, the SAR algorithm is introduced. The main objective is to test the DL structure using different criteria to enhance the coverage accuracy. The different coverage criteria such as KMNC, NBC, SNAC, TKNC, and TKNP are used for the testing of DBN. Using the generated test inputs, the criteria is validated and the developed criteria are capable to capture undesired behaviors in the DBN structure. The developed approach is implemented by Python platform using three standard datasets like MNIST, CIFAR-10, and ImageNet. For analysis, the developed approach is compared with the three LeNet models like LeNet-1, LeNet-4 and LeNet-5 for the MNIST dataset, the VGG-16, and ResNet-20 models for the CIFAR-10 dataset, and the VGG-19 and ResNet-50 models for the ImageNet dataset. These models are tested on the four adversarial test input generation approaches like BIM, JSMA, FGSM, and CW, and one DL testing method like DeepGauge to validate the efficiency of the suggested approach. The simulation results proved that the proposed approach obtained high coverage accuracy for each criterion on four adversarial test inputs and one DL testing method as compared to other models.

Model-Based Ideal Testing of GUI Programs–Approach and Case Studies

Traditionally, software testing is aimed at showing the presence of faults. This paper proposes a novel approach to testing graphical user interfaces (GUI) for showing both the presence and absence of faults in the sense of ideal testing. The approach uses a positive testing concept to show that the GUI under consideration (GUC) does what the user expects; to the contrary, the negative testing concept shows that the GUC does not do anything that the user does not expect, building a holistic view. The first step of the approach models the GUC by a finite state machine (FSM) that enables the model-based generation of test cases. This is always possible as the GUIs are considered as strictly sequential processes. The next step converts the FSM to an equivalent regular expression (RE) that will be analyzed first to construct test selection criteria for excluding redundant test cases and construct test coverage criteria for terminating the positive test process. Both criteria enable us to assess the adequacy and efficiency of the positive tests performed. The negative tests will be realized by systematically mutating the FSM to model faults, the absence of which are to be shown. Those mutant FSMs will be handled and assessed in the same way as in positive testing. Two case studies illustrate and validate the approach; the experiments' results will be analyzed to discuss the pros and cons of the techniques introduced.

Research on the Effectiveness Design of Information System Software Test Case Based on Clustering Algorithms

With the rapid development of software industry, the scale of software becomes larger and larger, and the complexity of software becomes higher and higher. The test coverage criterion traverses to get the corresponding test path and deletes the redundant path according to the constraints, then analyses and eliminates the contradictions in each test scenario to get the consistent requirement scenario, and finally generates the test case set of interlocking software. Test cases are the basis of test execution. Effective writing of test cases plays an important role in the whole testing process of information system. There are constraints (such as dependencies and mutual exclusions) between input items or parameter values, and the constraints certainly exist. From the initial frequent pattern mining to closed pattern mining, maximum pattern mining, incremental mining, topic interest measurement, privacy protection, data flow and other types of data mining association rules.

A Test Case Prioritization Genetic Algorithm Guided by the Hypervolume Indicator

Regression testing is performed during maintenance activities to assess whether the unchanged parts of a software behave as intended. To reduce its cost, test case prioritization techniques can be used to schedule the execution of the available test cases to increase their ability to reveal regression faults earlier. Optimal test ordering can be determined using various techniques, such as greedy algorithms and meta-heuristics, and optimizing multiple fitness functions, such as the average percentage of statement and branch coverage. These fitness functions condense the cumulative coverage scores achieved when incrementally running test cases in a given ordering using Area Under Curve (AUC) metrics. In this paper, we notice that AUC metrics represent a bi-dimensional (simplified) version of the hypervolume metric, which is widely used in many-objective optimization. Thus, we propose a Hypervolume-based Genetic Algorithm, namely HGA, to solve the Test Case Prioritization problem when using multiple test coverage criteria. An empirical study conducted with respect to five state-of-the-art techniques shows that (i) HGA is more cost-effective, (ii) HGA improves the efficiency of Test Case Prioritization, (iii) HGA has a stronger selective pressure when dealing with more than three criteria.

Smart Test Case Quantifier Based on Mc/Dc Coverage Criterion with Uml Sequence Diagram

Software testing is an inevitable phase in the Software Development Life Cycle (SDLC). It plays a vital role in ensuring the quality of the product. Testing can be performed on requirement, design and code. Traditional test case generation approaches majorly focuses on designing and implementation models, as a huge percentage of errors in the software are caused due to the lack of understanding in the early phases. So, it’s advisory to follow testing in the initial phase in SDLC as most of the errors/faults can be eliminated and can be prevented without disseminating to the next phase. So, testing must not be secluded to a single phase alone in SDLC. In software testing, test case generation is the fundamental and challenging part. The major purpose to design test case is to create set of tests that are valuable in validation. Due to the increase in software size, it’s not quite advisable to have the choice as manual testing to test the code as its error prone, complex and time consuming. Automation of testing would help the tester to test effectively and timely. Applications of new techniques improve the test process and cut down the tester’s effort. The proposed work presents a sandwich of black box and white box testing. Design-based testing, a black box approach is implemented, that aids in fixing the errors at initial phase. Unified Modeling Language (UML) Use case diagram, Activity diagram and Sequence Diagram are considered for designing. The code-based generation of test cases, is a white box testing approach, test coverage criteria Modified Condition/Decision Condition (MC/DC) has been used to ensure the maximum coverage of the code during testing phase for generating the test cases. The objective of this paper is to automate the generation of minimal number of test cases required to test a system with maximum coverage by removing the redundant test cases using MC/DC criterion. This present an idea to the beginners of the testing about minimal test cases they necessitate for testing their system.

Test coverage criteria for software product line testing: Systematic literature review

Abstract Context In software product line testing (SPLT), test coverage criterion is an important concept, as it provides a means of measuring the extent to which domain testing has been performed and redundant application testing can be avoided based on the test coverage level achieved in domain testing. However, no previous literature reviews on SPLT have addressed test coverage criterion in SPLT. Objective The objectives of this paper are as follows: (1) to clarify the notions of test basis and test coverage criterion for SPLT; (2) to identify the test coverage criteria currently used for SPLT; (3) to investigate how various SPLT aspects, such as the SPLT method, variability implementation mechanism, and variability management approach, affect the choice of test coverage criterion for SPLT; and (4) to analyze the limitations of test coverage criteria currently used for SPLT. Method This paper conducts a systematic review of test coverage criteria in SPLT with 78 selected studies. Results We have several findings that can guide the future research on SPLT. One important finding is that choice of test coverage criterion in SPLT is independent from variability implementation mechanism, variability management, SPL approach, and binding time but is dependent on the variability representation used in development artifacts. Another that is easily overlooked is that SPL test coverage criteria with the same test coverage criterion names of single system testing neither adequately convey what should be covered by the test methods applying them, nor can they be more generally regarded as extensions or generalizations for SPLT of their corresponding test coverage criteria of single system testing. Conclusion This study showed that SPL test coverage criteria should be defined or redefined so that they can clearly deliver the target properties to be satisfied by SPLT.

Task-based Automated Test Case Generation for Human-machine Interaction

Testing is an effective approach for finding discrepancies between intended and actual system behavior. However, the complexity of modern system can make it difficult for analysts to anticipate all the interactions that need to be tested. This is particularly true for human-interactive systems where humans may do things that were not anticipated by analysts. We address this by introducing a novel approach to automated test case generation for human-machine interaction. We do this by combining formal models of human-machine interfaces with formal models of human task behavior. We then use the robust search capabilities of model checking to generate test sequences guaranteed to satisfy test coverage criteria. We demonstrate the capabilities of our approach with of a pod-based coffee machine. Results and future research are discussed.

Structural Test Coverage Criteria for Deep Neural Networks

Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we propose a family of four novel test coverage criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that test inputs that are generated with guidance by our proposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods, we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples and correlation with functional coverage) and the computational cost of test input generation. Our experiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and ImageNet.

An end‐user‐centric test generation methodology for performance evaluation of mobile networked applications

SummaryWe propose a model‐based test generation methodology to evaluate the impact of the interaction of the wireless network and application configurations on the performance of mobile networked applications. We consider waiting time delay to model wireless network quality. We classify mobile applications into two groups. Group I represents applications where end‐user experience is mainly affected by waiting time delay during service consumption, while group II represents applications where end‐user experience is affected by waiting time delay before service consumption. Test generation is formulated as an inversion problem. However, for group I applications, solving the inversion problem is expensive. Therefore, we utilize metamorphic testing to mitigate the cost of test oracles. We formulate metamorphic test generation as maximization of the distance between seed and follow‐up test cases. Two test coverage criteria are proposed: user experience and user‐experience‐and‐input interaction. Network models are developed for a mobile device that has network access through a WiFi hot spot and uses either transmission control protocol or user datagram protocol. Two mobile applications are used to demonstrate the methodology: multimedia streaming and web browsing. Application of the methodology when user actions are taken into consideration is also addressed. The effectiveness of the methodology is evaluated using two metrics: the incurred time cost and redundancy in the generated test suite. The obtained results show the advantage of casting test generation as an inversion problem, compared with random testing. For apps with intensive performance models, combining metamorphic testing with the methodology has tremendously reduced the cost of test oracles.

The world changes

This issue is the first issue of STVR since Jeff Offutt stood down as joint Editor-in-Chief. Jeff was Editor-in-Chief and then joint Editor-in-Chief from 2007 to 2019. This has been an interesting and challenging period for academic journals, with many changes to how journals operate. During this period, Jeff significantly strengthened the journal and has been at the forefront of developments within the field. As Jeff mentioned in the last editorial, there are still many challenges, but Jeff has left STVR in an excellent position. I also hope that we will soon start receiving his papers again! On a personal note, I was honoured when Jeff invited me to be joint EiC, and I welcomed the opportunity to work with him. I accepted immediately and have never regretted the decision. It has been a pleasure to work with Jeff: not only is he a major figure in the field, he is also charming and humble. Jeff also genuinely cares about the discipline and those working within it. As an example, he has spent a significant amount of time providing invaluable advice to junior researchers and students. Although I was sad to see Jeff step down, I am really pleased to be able to welcome Professor Tao Xie as joint Editor-in-Chief of STVR. I first met Tao at a conference (some years ago), and I was immediately impressed by him as a person but also by the work of his that we discussed. This work addressed a real problem in a practical manner but was also based on sound intuition and theory. He has since become a leading figure in the field. Jeff and I were convinced that Tao was the right person to take over and were delighted when he accepted our invitation. Tao has already shown great energy and enthusiasm, and in his upcoming editorial, he will say a little more about some of the ideas of his that we will be implementing. This issue contains two papers. Our first paper is “An Improved Software Reliability Prediction Model by Using High Precision Error Iterative Analysis Method” by Gul Jabeen, Ping Luo and Wasif Afzal (recommended by Min Xie). This concerns a classical problem, which is that of trying to model how the reliability of a piece of software will change during development. Techniques that achieve this can, for example, be used to help plan the software release. The authors use neural networks to help model the error and so make a model more accurate. One of the benefits of the approach is that it can potentially be used with any parametric software reliability prediction model. The second paper is “An End-user-centric Test Generation Methodology for Performance Evaluation of Mobile Networked Applications” by Mustafa Al-tekreeti, Atef Abdrabou and Kshirasagar Naik (recommended by Jane Hayes). The paper concerns the testing of mobile network applications, with a focus on performance (waiting time delay) and its impact on user experience. The authors introduce new test coverage criteria associated with two types of delays and also use metamorphic testing to address the oracle problem.