Metamorphic Relations Research Articles

Test cases generation for multiple paths based on PSO algorithm with metamorphic relations

The generation of multiple-path test cases can greatly enhance the efficiency of path-wise testing. Various methods adopting meta-heuristic algorithm to generate multiple-path test cases have been proposed, but existing methods focus on improving the meta-heuristic algorithm to get better test case generation efficiency, and test cases covering each path needs to be generated by meta-heuristic algorithm searching. To improve efficiency, a test case generation method for multiple-path coverage is proposed in this study, which combines a particle swarm optimisation (PSO) algorithm with metamorphic relations (MRs). The method first generates a test case using the PSO algorithm, and then generates new test cases by repeatedly using MRs between test cases. This method reduces evolving numbers of PSO algorithm. The experimental results show that the proposed method can significantly enhance the efficiency in terms of fitness evaluations and time consumption.

Testing bidirectional model transformation using metamorphic testing

ContextIn model-based software development, bidirectional model transformation (BX) provides a fundamental solution to model synchronization that can retain the consistency among models. Similar to conventional programs, a BX program may also contain bugs. Accordingly, a BX program must be tested prior to being used in practice. A challenging problem of testing BX programs is to construct test oracles (e.g., assertions and expected output models), which are usually difficult and/or expensive to manually specify. ObjectiveIn we paper, we investigate how to alleviate the oracle problem in BX testing via reducing the costs of developing test oracles. MethodWe propose a metamorphic testing approach for BX. First, we identify three generic metamorphic relations for BX. Afterwards, we define a metamodel MT4MT to establish metamorphic test groups and test scripts. We also propose a testing framework to support metamorphic testing based on MT4MT. ResultsWe conducted an experimental study of mutation analysis and a case study on three ATL-based ad-hoc BXs. The results of the experimental study and the case study showed that our approach killed 79.38% mutants and enabled us to test real-world ATL-based ad-hoc BXs. We also demonstrated that MT4MT can be used to test the semantics properties of BXs. ConclusionOur approach is an effective and practical approach with lower costs of developing test oracles.

Performance metamorphic testing: A Proof of concept

ContextPerformance testing is a challenging task mainly due to the lack of test oracles, i.e. mechanisms to decide whether the performance of a program is acceptable or not because of a bug. Metamorphic testing enables the generation of test cases in the absence of an oracle by exploiting the so–called metamorphic relations between the inputs and outputs of multiple executions of the program under test. In the last two decades, metamorphic testing has been successfully used to detect functional faults in different domains. However, its applicability to performance testing remains unexplored. ObjectiveWe propose the application of metamorphic testing to reveal performance failures. MethodWe define Performance Metamorphic Relations (PMRs) as expected relations between performance measurements of multiple executions of the program under test. These relations can be turned into assertions for the automated detection of performance bugs, removing the need for complex benchmarks and domain experts guidance. As a further benefit, PMRs can be turned into fitness functions to guide search–based techniques on the generation of test data. ResultsThe feasibility of the approach is illustrated through an experimental proof of concept in the context of the automated analysis of feature models. ConclusionThe results confirm the potential of metamorphic testing, in combination with search-based techniques, to automate the detection of performance bugs.

Metamorphic Testing

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research.

Automated inference of likely metamorphic relations for model transformations

Approach to automatically infer likely metamorphic relations in model transformations.A catalogue of 24 domain-independent likely metamorphic relations has been proposed.Five domain-independent patterns in the transformations traces have been identified.Specific metamorphic relations are automatically inferred in each transformation.Precision of 96.4% with 7 case studies and mutation score of 93.3% in one of them. Model transformations play a cornerstone role in Model-Driven Engineering (MDE) as they provide the essential mechanisms for manipulating and transforming models. Checking whether the output of a model transformation is correct is a manual and error-prone task, referred to as the oracle problem. Metamorphic testing alleviates the oracle problem by exploiting the relations among different inputs and outputs of the program under test, so-called metamorphic relations (MRs). One of the main challenges in metamorphic testing is the automated inference of likely MRs.This paper proposes an approach to automatically infer likely MRs for ATL model transformations, where the tester does not need to have any knowledge of the transformation. The inferred MRs aim at detecting faults in model transformations in three application scenarios, namely regression testing, incremental transformations and migrations among transformation languages. In the experiments performed, the inferred likely MRs have proved to be quite accurate, with a precision of 96.4% from a total of 4101 true positives out of 4254 MRs inferred. Furthermore, they have been useful for identifying mutants in regression testing scenarios, with a mutation score of 93.3%. Finally, our approach can be used in conjunction with current approaches for the automatic generation of test cases.

Structural architecture and low‐grade metamorphism of the Mikabu‐Northern Chichibu accretionary wedge, SW Japan

ABSTRACTTo understand tectono‐metamorphic processes within or close to the brittle–ductile transition of quartz‐rich crustal rocks in an accretionary wedge, an integrated field, petrological, geochronological and Raman spectroscopic study was conducted on the Mikabu‐Northern Chichibu belt in SW Japan. Field mapping in central Shikoku reveals that the Northern Chichibu belt is comprised of a pile of four tectono‐stratigraphic units, referred to as A, B, C and D units. The A unit (dominated by pelagic sedimentary rocks) represents the structurally lowest and youngest accretionary complex that forms a composite unit with the Mikabu ophiolitic suite. The B unit (consisting of chert‐clastic rock sequences) overlies the A unit and is overlain by the C and D units (mudstone‐matrix mélange units). Raman spectroscopy of carbonaceous material constrains the peak temperature of each unit to be ~290°C for the A unit, 270–290°C for the B unit, 230–250°C for the C unit and ~220°C for the D unit. Ductile deformation and pervasive metamorphism are limited to rocks in the Mikabu, A and B units. Alkali pyroxene and sodic amphibole occur in metabasite from the Mikabu, A and B units, and the widespread occurrence of prograde veins containing lawsonite+quartz pseudomorphs after laumontite was newly recognized from the C unit. Phase petrological data constrain the peak pressure of each unit to be ~0.65 GPa for the Mikabu‐A unit (aragonite stable), ~0.45–0.6 GPa for the B unit (jadeite+albite stable in the structurally lower part), and ~0.35 GPa for the C unit (prehnite+lawsonite stable). The peak metamorphic pressure increases towards structurally lower and younger accretionary complexes, but the thickness of the preserved strata is insufficient to account for the inferred pressure range. The structural–metamorphic relations imply thickening of the accretionary wedge by underplating was followed by a significant phase of thinning by both ductile and brittle processes.

RBF-MLMR: A Multi-Label Metamorphic Relation Prediction Approach Using RBF Neural Network

Metamorphic testing has been successfully used in many different fields to solve the test oracle problem. However, how to find a set of appropriate metamorphic relations for metamorphic testing remains a complicated and tedious task. Recently some machine learning approaches have been proposed to predict metamorphic relations. These approaches predicting single label metamorphic relation can alleviate this problem to some extent. However, many applications involve multi-group metamorphic relations, and these approaches are clearly inefficient. To address this problem, in this paper we propose a M ulti- L abel M etamorphic R elations prediction approach based on an improved radial basis function (RBF) neural network named RBF-MLMR. First, RBF-MLMR uses state-of-the-art soot analysis tool to generate control flow graph and corresponds labels from the source codes of programs. Second, the extracted nodes and the path properties constitute multi-label data sets for the control flow graph. Finally, a multi-label RBF neural network prediction model is established to predict whether the program satisfies multiple metamorphic relations. In order to improve the prediction results, affinity propagation and $k$ -means clustering algorithms are used to optimize the RBF neural network structure of RBF-MLMR. A set of dedicated experiments based on public programs is conducted to validate RBF-MLMR. The experimental results show that RBF-MLMR can achieve accuracy of around 80% for predicting two and three metamorphic relations.

Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program

One of the grand challenges for adequately testing complex software is due to the oracle problem. Metamorphic Testing (MT) is a promising technique to alleviate the oracle problem through using one or multiple Metamorphic Relations (MRs) as test oracles. MT checks the satisfaction of every MR among the outputs of the MR-related tests instead of the correctness of individual test outputs. In practice, it is fairly easy to find MRs for testing any program, but it is very difficult to develop “good” MRs and evaluate their adequacy. A systematic approach for developing MRs and evaluating their adequacy in MT remains to be developed. In this paper, we propose a framework for evaluating MT and iteratively developing adequate MRs monitored by MT adequacy evaluation. The MT adequacy is measured by program coverages, mutation testing, and testing MRs with mutation tests. The MT evaluation results are used for guiding the iterative development of MRs, generating tests, and analyzing test outputs. We explain the framework through a testing example on an image processing program that is used for building the 3-dimensional structure of a biology cell based on its confocal image sections. In order to demonstrate the effectiveness of the proposed framework, we reported a case study of testing a complex scientific program: a Monte Carlo modeling program that simulates photon propagations in turbid tissue phantoms for accurate and efficient generation of reflectance images from biological tissues. The case study has shown the effectiveness of proposed MT framework for testing scientific software in general and the necessity of the MT enhancement in the development of adequate MRs. The case study results can be easily adapted for testing other software.

A runtime-testing method for integer overflow detection based on metamorphic relations

A runtime-testing method for integer overflow detection based on metamorphic relations

A Survey on Metamorphic Testing

A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output “morphs” into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.

Novel Approach for Whole Test Suite Generation using Metamorphic Relations

Background: The software or an individual program will not get crash by the minute bugs in the code and always manual method for testing the code was not feasible, most of the cases the tester will adds the test oracles to the test cases using the manual method but it is not optimal solution for the large programs and software's and this method can targets only covering the one goal at a time. There is a problem with this coverage goals due to these goals are not independent. Methodology: To get out from these problems we propose a unique approach, in this approach we are generating the test cases automatically and developed an integrated method for program correctness, testing and debugging. Findings: We developed an unique approach in order to solve the oracle problem by using the metamorphic testing this approach also address the automatic debugging this testing uses the synergy algorithm, it does not attempt to traverse the execution tree , instead it attempts to cover all abstract states. Applications/Improvement: With metamorphic relations the system is ideal for medium and large scale applications and this approach also uses the fuzzy logic to provide the result whether the test case pass or fail.

Metamorphic Relations Based Test Oracles for Image Processing Applications

Evaluation of output images carrying visual semantics is a challenging task which is carried out by domain experts through visual inspection. Automatic test oracle is required to augment the test oracle problem and to eliminate the manual efforts. Metamorphic testing is an effective technique to alleviate these problems. In this paper, the authors have demonstrated that how inherent properties of implementation under test can be used to generate an automatic test oracle for image processing applications. Metamorphic testing is a general technique in which follow-up test cases are generated using a transformation function and the anticipated output is evaluated. They have used some general metamorphic relations and also designed some algorithm specific metamorphic relations for morphological image operations. Selection of metamorphic relations is the most important step and the authors have analyzed relative effectiveness of different metamorphic relations using mutation analysis. The results show metamorphic testing is a very effective technique to automate output images evaluation and to alleviate oracle problem.

MT4WS: an automated metamorphic testing system for web services

The use of web services has been growing significantly, with increasingly large numbers of applications being implemented through the web. A difficulty associated with this development is the quality assurance of these services, specifically the challenges encountered when testing the applications - amongst other things, testers may not have access to the source code, and the correctness of the output may not be easily ascertained known as the oracle problem. Metamorphic testing MT has been introduced as a technique to alleviate the oracle problem. MT makes use of properties of the software under test, known as metamorphic relations, and checks whether or not these relations are violated. Since MT does not require source code to generate the metamorphic relations, it is suitable for testing web services-based applications. We have designed an XML-based language representation to facilitate the formalisation of metamorphic relations, the generation of follow-up test cases, and the verification of the test results. Based on this, we have also developed a tool to support the automation of MT for web service applications. This tool has been used in an experiment to test web services, the evaluation of which is reported in this paper.

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

SummaryComprehensive, automated software testing requires an oracle to check whether the output produced by a test case matches the expected behaviour of the programme. But the challenges in creating suitable oracles limit the ability to perform automated testing in some programmes, and especially in scientific software. Metamorphic testing is a method for automating the testing process for programmes without test oracles. This technique operates by checking whether the programme behaves according to properties called metamorphic relations. A metamorphic relation describes the change in output when the input is changed in a prescribed way. Unfortunately, finding the metamorphic relations satisfied by a programme or function remains a labour‐intensive task, which is generally performed by a domain expert or a programmer. In this work, we propose a machine learning approach for predicting metamorphic relations that uses a graph‐based representation of a programme to represent control flow and data dependency information. In earlier work, we found that simple features derived from such graphs provide good performance. An analysis of the features used in this earlier work led us to explore the effectiveness of several representations of those graphs using the machine learning framework of graph kernels, which provide various ways of measuring similarity between graphs. Our results show that a graph kernel that evaluates the contribution of all paths in the graph has the best accuracy and that control flow information is more useful than data dependency information. The data used in this study are available for download at http://www.cs.colostate.edu/saxs/MRpred/functions.tar.gz to help researchers in further development of metamorphic relation prediction methods. Copyright © 2015 John Wiley & Sons, Ltd.

METRIC: METamorphic Relation Identification based on the Category-choice framework

Metamorphic testing is a promising technique for testing software systems when the oracle problem exists, and has been successfully applied to various application domains and paradigms. An important and essential task in metamorphic testing is the identification of metamorphic relations, which, due to the absence of a systematic and specification-based methodology, has often been done in an ad hoc manner—something which has hindered the applicability and effectiveness of metamorphic testing. To address this, a systematic methodology for identifying metamorphic relations based on the category-choice framework, called metric, is introduced in this paper. A tool implementing this methodology has been developed and examined in an experiment to determine the viability and effectiveness of metric, with the results of the experiment confirming that metric is both effective and efficient at identifying metamorphic relations.

Automated metamorphic testing of variability analysis tools

SummaryVariability determines the capability of software applications to be configured and customized. A common need during the development of variability‐intensive systems is the automated analysis of their underlying variability models, for example, detecting contradictory configuration options. The analysis operations that are performed on variability models are often very complex, which hinders the testing of the corresponding analysis tools and makes difficult, often infeasible, to determine the correctness of their outputs, that is, the well‐known oracle problem in software testing. In this article, we present a generic approach for the automated detection of faults in variability analysis tools overcoming the oracle problem. Our work enables the generation of random variability models together with the exact set of valid configurations represented by these models. These test data are generated from scratch using stepwise transformations and assuring that certain constraints (a.k.a. metamorphic relations) hold at each step. To show the feasibility and generalizability of our approach, it has been used to automatically test several analysis tools in three variability domains: feature models, common upgradeability description format documents and Boolean formulas. Among other results, we detected 19 real bugs in 7 out of the 15 tools under test. Copyright © 2015 John Wiley & Sons, Ltd.

A property-based testing framework for encryption programs

In recent years, a variety of encryption algorithms were proposed to enhance the security of software and systems. Validating whether encryption algorithms are correctly implemented is a challenging issue. Software testing delivers an effective and practical solution, but it also faces the oracle problem (that is, under many practical situations, it is impossible or too computationally expensive to know whether the output for any given input is correct). In this paper, we propose a property-based approach to testing encryption programs in the absence of oracles. Our approach makes use of the so-called metamorphic properties of encryption algorithms to generate test cases and verify test results. Two case studies were conducted to illustrate the proposed approach and validate its effectiveness. Experimental results show that even without oracles, the proposed approach can detect nearly 50% inserted faults with at most three metamorphic relations (MRs) and fifty test cases.

A Cost-driven Approach for Metamorphic Testing

Metamorphic testing has been applied in various systems from different domains. Many studies showed that the selection of metamorphic relations greatly affected the effectiveness of metamorphic testing. However, these studies mainly focused on the fault-detection effectiveness. They did not consider the cost that metamorphic relations involved, such as the number of test inputs. Good metamorphic relations should have high fault-detection effectiveness with a low cost. In this paper, we propose a cost-driven approach for metamorphic testing. The key idea is to design metamorphic relations sharing the same test inputs to reduce the testing cost. We conduct a case study on a bank system and compare the cost-effectiveness of metamorphic relations derived from this approach and those constructed by the conventional approach. The experimental results show that metamorphic relations derived from our approach are more cost-effective. We also find that this approach not only reduces the cost of metamorphic testing, but also helps to construct different metamorphic relations to detect different types of faults.

How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?

In software testing, something which can verify the correctness of test case execution results is called an oracle. The oracle problem occurs when either an oracle does not exist, or exists but is too expensive to be used. Metamorphic testing is a testing approach which uses metamorphic relations, properties of the software under test represented in the form of relations among inputs and outputs of multiple executions, to help verify the correctness of a program. This paper presents new empirical evidence to support this approach, which has been used to alleviate the oracle problem in various applications and to enhance several software analysis and testing techniques. It has been observed that identification of a sufficient number of appropriate metamorphic relations for testing, even by inexperienced testers, was possible with a very small amount of training. Furthermore, the cost-effectiveness of the approach could be enhanced through the use of more diverse metamorphic relations. The empirical studies presented in this paper clearly show that a small number of diverse metamorphic relations, even those identified in an ad hoc manner, had a similar fault-detection capability to a test oracle, and could thus effectively help alleviate the oracle problem.

Salinia revisited: a crystalline nappe sequence lying above the Nacimiento fault and dispersed along the San Andreas fault system, central California

Salinia revisited: a crystalline nappe sequence lying above the Nacimiento fault and dispersed along the San Andreas fault system, central California