High Fault Detection Capability Research Articles

Identifying metamorphic relations: A data mutation directed approach

SummaryMetamorphic testing (MT) is an effective technique to alleviate the test oracle problem. The principle of MT is to detect failures by checking whether some necessary properties, commonly known as metamorphic relations (MRs), of software under test (SUT) hold among multiple executions of source and follow‐up test cases. Since both the generation of follow‐up test cases and test result verification depend on MRs, the identification of MRs plays a key role in MT, which is an important yet difficult task requiring deep domain knowledge of the SUT. Accordingly, techniques that can direct a tester to identify MRs effectively are desirable. In this paper, we propose MT, a data mutation directed approach to identifying MRs. MT guides a tester to identify MRs by providing a set of data mutation operators and template‐style mapping rules, which not only alleviates the difficulties faced in the process of MR identification but also improves the identification effectiveness. We have further developed a tool to implement the proposed approach and conducted an empirical study to evaluate the MR identification effectiveness of MT and the performance of MRs identified by MT with respect to fault detection capability and statement coverage. The empirical results show that MT is able to identify MRs for numeric programs effectively, and the identified MRs have high fault detection capability and statement coverage. The work presented in this paper advances the field of MT by providing a simple yet practical approach to the MR identification problem.

Studying test-driven development and its retainment over a six-month time span

In this paper, we investigate the effect of TDD, as compared to a non-TDD approach, as well as its retainment (or retention) over a time span of (about) six months. To pursue these objectives, we conducted a (quantitative) longitudinal cohort study with 30 novice developers (i.e., third-year undergraduate students in Computer Science). We observed that TDD affects neither the external quality of software products nor developers’ productivity. However, we observed that the participants applying TDD produced significantly more tests, with a higher fault-detection capability, than those using a non-TDD approach. As for the retainment of TDD, we found that TDD is retained by novice developers for at least six months.

Regression Test Case Prioritization Based on Fixed Size Candidate Set ART Algorithm

Regression testing is a very time-consuming and expensive testing activity. Many test case prioritization techniques have been proposed to speed up regression testing. Previous studies show that no one technique is always best. Random strategy, as the simplest strategy, is not always so bad. Particularly, when a test suite has higher fault detection capability, the strategy can generate a better result. Nevertheless, due to the randomness, the strategy is not always as satisfactory as expected. In this context, we present a test case prioritization approach using fixed size candidate set adaptive random testing algorithm to reduce the effect of randomness and improve fault detection effectiveness. The distance between pair-wise test cases is assessed by exclusive OR. We designed and conducted empirical studies on eight C programs to validate the effectiveness of the proposed approach. The experimental results, confirmed by a statistical analysis, indicate that the approach we proposed is more effective than random and the total greedy prioritization techniques in terms of fault detection effectiveness. Although the presented approach has comparable fault detection effectiveness to ART-based and the additional greedy techniques, the time cost is much lower. Consequently, the proposed approach is much more cost-effective.

Summary of the 14th edition of the IEEE/ACM Workshop on Automation of Software Test (AST)

Effective and efficient testing with reduced costs and a high fault detection capability is the desirable goal in industry which can be achieved only through automation of all parts of the testing process. In the past decades, a great amount of research effort has been spent on automating all various parts of the testing process such as test case derivation, test selection, test oracle construction, test execution, and others. In addition, there has been a rapid growth in automated software testing tools which is stimulated in part through the shift towards agile development practices in industry that demands a high level of automation. Work on this topic has long been published as an important part of software engineering. In recent years, testing has been consistently among the top-most popular topics in submissions to software engineering conferences. The practice of software test automation (TA) has also moved forward significantly in the past few years. However, progress in TA is still required. Software systems have become more and more complicated through the integration of components developed by different vendors and using different techniques in different programming languages running on different platforms. The advent of cloud computing, mobile computing and the Internet of Things has imposed grave new challenges to TA. Those systems become increasingly reactive to changes in their environment, requiring equally adaptive TA approaches. Few software testing tools can currently handle the needed requirements to test such systems.

Employing rule mining and multi-objective search for dynamic test case prioritization

Test case prioritization (TP) is widely used in regression testing for optimal reordering of test cases to achieve specific criteria (e.g., higher fault detection capability) as early as possible. In our earlier work, we proposed an approach for black-box dynamic TP using rule mining and multi-objective search (named as REMAP) by defining two objectives (fault detection capability and test case reliance score) and considering test case execution results at runtime. In this paper, we conduct an extensive empirical evaluation of REMAP by employing three different rule mining algorithms and three different multi-objective search algorithms, and we also evaluate REMAP with one additional objective (estimated execution time) for a total of 18 different configurations (i.e., 3 rule mining algorithms × 3 search algorithms × 2 different set of objectives) of REMAP. Specifically, we empirically evaluated the 18 variants of REMAP with 1) two variants of random search while using two objectives and three objectives, 2) three variants of greedy algorithm based on one objective, two objectives, and three objectives, 3) 18 variants of static search-based prioritization approaches, and 4) six variants of rule-based prioritization approaches using two industrial and three open source case studies. Results showed that the two best variants of REMAP with two objectives and three objectives significantly outperformed the best variants of competing approaches by 84.4% and 88.9%, and managed to achieve on average 14.2% and 18.8% higher Average Percentage of Faults Detected per Cost (APFDc) scores.

RAMBUTANS: automatic AOP-specific test generation tool

Aspect-oriented programming (AOP) is a programmatic methodology to handle better modularized code by separating crosscutting concerns from the traditional abstraction boundaries. Automated testing, as one of the most demanding needs of the software development to reduce both human effort and costs, is a delicate issue in testing aspect-oriented programs. Prior studies in the automated test generation for aspect-oriented programs have been very limited with respect to the need for both adequate tool support and capability concerning effectiveness and efficiency. This paper describes a new AOP-specific tool for testing aspect-oriented programs, called RAMBUTANS. The RAMBUTANS tool uses a directed random testing technique that is especially well suited for generating tests for aspectual features in AspectJ. The directed random aspect of the tool is parameterized by associating weights to aspects, advice, methods, and classes by controlling object and joint point creations during the test generation process. We present a comprehensive empirical evaluation of our tool against the current AOP test generation approaches on three industrial aspect-oriented projects. The results of the experimental and statistical tests showed that RAMBUTANS tool produces test suites that have higher fault-detection capability and efficiency for AspectJ-like programs.

Testing access control policy through Change Rule and Swap Rule Algorithm (CRSR)

We propose an algorithm for testing policies based on XACML Context Schema, known as Change Rule and Swap Rule Algorithm (CRSR). Compared to other testing techniques and tools for testing access control policies, where policy set or policy is evaluated first, this algorithm focuses on the rule and target of a policy set or policy. Our approach represents policy as a vector of bits. A Boolean variable 1 is used to represent the applicability of a policy to a request and a Boolean variable 0 is used to represent the non-applicability of a policy to a request. We assume that a correct policy evaluates to 1, indicating that all the elements, attributes ID and their values are correctly enforced. First of all we identify and extract the rule and target from the policy and then generate mutant policies and requests by applying the proposed algorithm. The rule and target are then evaluated using the original policies, mutant policies and requests generated. We compare our results against mutants generated using mutant operators, the Targen tool and the simple combinatorial approach to see the fault detection capability. The proposed algorithm is found to be effective, efficient and of high fault detection capability in testing access control policies, as it ensures a total coverage of the various elements of a policy based on the XACML Context Schema.

Empirical study of the effects of different profiles on regression test case reduction

Determining how to select a subset of test cases with high-fault detection capability becomes a key issue in code-level regression testing. Cluster analysis has been proposed to deal with this issue. It partitions test cases into clusters based on the similarity of execution profiles. In previous studies, execution profiles were represented as binary or numeric vectors. The vector model only considers the number of times that a function or statement is executed. However, it ignores sequential, the relations and structural information between function calls. Therefore vector-based methods do not always generate satisfying results. In this study, the authors presented cluster analysis of three different types of structural profiles, that is, function execution sequence, function call sequence (FCS) and function call tree. They designed and conducted empirical studies on five medium-sized programs to validate the effects of different profiles on regression test case reduction. Experimental results illustrate that sequential, call relations and structural information can aid to further improve fault detection effectiveness. In view of cost-effectiveness, FCS is regarded as to be the optimal profile. Furthermore, cluster analysis of FCSs is comparable to the additional branch coverage reduction technique with respect to fault detection effectiveness.

Regression Testing-Based Requirement Prioritization of Mobile Applications

Mobile software application development process must be matured enough to handle the challenges (especially market related) associated with the development of high quality mobile software development. Ever increasing number of both mobile users and mobile applications had presented software engineers with the challenge of satisfying billions of users with high quality software applications to be delivered within deadline and budgets. Always there had been a lot of pressure to develop complex software categorized by thousands of requirements, under resource constrained environment. Requirement prioritization is one of the activities undertaken by software engineer to deliver partial software product to its customers such that most important requirements are implemented in the earliest releases. During next releases some changed and pending requirements are implemented, an activity that generates ripple effects. Such ripple effects need to be tested by executing modified source code against test cases of previous releases (regression testing). Regression testing is a very effortful activity that requires a software tester to select test cases that have high fault detection capability, execute the modified code against selected test cases and performing debugging. This regression testing activity can be lowered to the maximum extend by considering dependencies between requirements during the time of requirement prioritization. Thus requirement prioritization will be carried out not only against aspects like cost, time, risks, business values etc but against dependencies also. The aim is to implement almost all dependent highest priority requirements in current release so that implementation of new requirements is unlikely to have ripple effects. Changes in requirements might not be related to variable usage and definition and might not involve a change in functionality. In such cases there is no need to select already executed test cases of previous versions. Module dependencies can lead to test case selections of previous versions if changes of requirement lead to ripple effects. This paper aims to implement highest priority requirements such that regression testing is performed to minimum thereby improving development process of mobile applications. The proposed technique had been successfully evaluated on Android based notification software application that meets the specification of Aakash tablets.

Systematic Structural Testing of Firewall Policies

Firewalls are the mainstay of enterprise security and the most widely adopted technology for protecting private networks. As the quality of protection provided by a firewall directly depends on the quality of its policy (i.e., configuration), ensuring the correctness of firewall policies is important and yet difficult. To help ensure the correctness, we propose a systematic structural testing approach for firewall policies. We define structural coverage (based on coverage criteria of rules, predicates, and clauses) on the firewall policy under test. To achieve high structural coverage effectively, we have developed four automated packet generation techniques: the random packet generation, the one based on local constraint solving (considering individual rules locally in a policy), the one based on global constraint solving (considering multiple rules globally in a policy), and the one based on boundary values. We have conducted an experiment on a set of real policies and a set of faulty policies to detect faults with generated packet sets. Generally, our experimental results show that a packet set with higher structural coverage has higher fault-detection capability (i.e., detecting more injected faults). Our experimental results show that a reduced packet set (maintaining the same level of structural coverage with the corresponding original packet set) maintains similar fault-detection capability with the original set.

New after‐laying test methods for XLPE cable lines

AbstractAn after‐laying test is required to demonstrate the soundness of XLPE underground cable lines with joints. AC voltage application is desirable for the test; however, the equipment for ac application must be very large because it must supply large capacitive current for cables. DC voltage is normally applied, which can be supplied with small equipment; however, the effectiveness of the dc application is questioned because the electrical stress distribution is quite different from that with ac application and, moreover, dc application could damage insulation properties of cables.The authors targeted the Oscillating Wave (OSW) method and the Very Low‐Frequency (VLF) method as alternative after‐laying tests. They first developed modeling methods of various defects on underground cable lines and applied OSW and VLF to these defect models. They found that OSW and VLF methods have higher fault detection capability than dc voltage application. They also demonstrated that the defects which can be easily detected with the OSW method are difficult to detect with the VLF method and vice versa. They showed that serial application of both methods highly improve the fault detection ability for XLPE underground cable lines.