MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing

Image contrast enhancement uses the object intensity transformation function to maximize the amount of information to enhance an image. In this paper, the image enhancement problem is regarded as an optimization problem, and the particle swarm algorithm is used to obtain the optimal solution. First, an improved particle swarm optimization algorithm is proposed. In this algorithm, individual optimization, local optimization, and global optimization are used to adjust the particle's flight direction. In local optimization, the topology is used to induce comparison and communication between particles. The sparse penalty term in speed update formula is added to adjust the sparsity of the algorithm and the size of the solution space. Second, the three channels of the color images R, G, and B are represented by a quaternion matrix, and an improved particle swarm algorithm is used to optimize the transformation parameters. Finally, contrast and brightness elements are added to the fitness function. The fitness function is used to guide the particle swarm optimization algorithm to optimize the parameters in the transformation function. This paper verifies via two experiments. First, improved particle swarm algorithm is simulated and tested. By comparing the average values of the four algorithms under the three types of 6 test functions, the average value is increased by at least 15 times in the single-peak 2 test functions: in the multi-peak and multi-peak fixed-dimension 4 test functions, this paper can always search for the global optimal solution, and the average value is either the same or at least 1.3 times higher. Second, the proposed algorithm is compared with other evolutionary algorithms to optimize contrast enhancement, select images in two different data sets, and calculate various evaluation indicators of different algorithms under different images. The optimal value is the algorithm in this paper, and the performance indicators are at least a 5% increase and a minimum 15% increase in algorithm running time. Final results show that the effects the proposed algorithm have obvious advantages in both subjective and qualitative aspects.

Automated testing of complex software systems can be a challenging task, and today there are a large number of methods for its implementation. One such method is metamorphic testing, which effectively solves the problems of usual methods and is gaining popularity. However, performing metamorphic tests can take a long time, so the question arises of their distributed running, including in the cloud. Thus, the authors of this study considered the designing of a cloud serverless architecture of software for metamorphic testing. The serverless architecture for metamorphic testing is proposed, which is based on the composition of the entire system from 5 individual components: models, data generator, software artifact under test, metamorphic relations, and serverless functions. For each of the main possible types of software artifacts, the possibility of using a serverless architecture for metamorphic testing is considered. The developed architecture is presented in the form of component, deployment, and sequence diagrams. The use of the proposed architecture in practice is shown by the example of testing two software artifacts – a class library and a web application. Performance measurements have shown that despite the additional network delay when running one test, the performance of all tests in general in the case of a serverless architecture is closer to local startup and will be faster with increasing complexity and number of tests.

With the wide application of machine translation, the testing of Machine Translation Systems (MTSs) has attracted much attention. Recent works apply Metamorphic Testing (MT) to address the oracle problem in MTS testing. Existing MT methods for MTS generally follow the workflow of input transformation and output relation comparison, which generates a follow-up input sentence by mutating the source input and compares the source and follow-up output translations to detect translation errors, respectively. These methods use various input transformations to generate the test case pairs and have successfully triggered numerous translation errors. However, they have limitations in performing fine-grained and rigorous output relation comparison and thus may report many false alarms and miss many true errors. In this article, we propose a word closure-based output comparison method to address the limitations of the existing MTS MT methods. We first propose word closure as a new comparison unit, where each closure includes a group of correlated input and output words in the test case pair. Word closures suggest the linkages between the appropriate fragment in the source output translation and its counterpart in the follow-up output for comparison. Next, we compare the semantics on the level of word closure to identify the translation errors. In this way, we perform a fine-grained and rigorous semantic comparison for the outputs and thus realize more effective violation identification. We evaluate our method with the test cases generated by five existing input transformations and the translation outputs from three popular MTSs. Results show that our method significantly outperforms the existing works in violation identification by improving the precision and recall and achieving an average increase of 29.9% in F1 score. It also helps to increase the F1 score of translation error localization by 35.9%.

This research work focuses on investigating the propagation of ultrasonic waves, which propagate mechanical vibrations of molecules or particles inside materials. Ultrasound imaging is extensively used and deeply rooted in the medical field. The key technologies that form the basis for many different uses in the area include transducers, contrast agents, pulse compression, beam shaping, tissue harmonic imaging, techniques for measuring blood flow and tissue motion, and three-dimensional imaging. The third-order non-linear β\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\beta$$\\end{document}-fractional Westervelt model has been used as a governing model in the imaging process for securing the different wave structures. The exact solutions of different types, including mixed, dark, singular, bright-dark, bright, complex and combined solitons are extracted. These solutions are obtained by using two newly introduced techniques, namely modified generalized Riccati equation mapping method and modified generalized exponential rational function method. Moreover, breather, lump and other waves are extracted by the assistance of logarithmic transformation and different test functions. The used methodologies are extremely effective and possess substantial computing capacity to effectively address the different solutions with a high level of accuracy in these systems. The techniques used are well-known for being effective, simple, and flexible enough to integrate multiple soliton systems into a unified framework. In addition, we provide 2D and 3D graphs that explain the behavior of the solution at various parameter values, under the influence of β\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\beta$$\\end{document}-fractional derivatives. The results offered in this study may improve the comprehension of the nonlinear dynamic behavior of the specific system and confirm the efficacy of the approaches used. We expect that our approaches will be beneficial for a wide range of nonlinear models and other problems in the related fields.

Applications in the fields of scientific computing, simulation, optimization, machine learning, etc. are sometimes said to be "non-testable programs" because there is no reliable test oracle to indicate what the correct output should be for arbitrary input. In some cases, it may be impossible to know the program's correct output a priori; in other cases, the creation of an oracle may simply be too hard. These applications typically fall into a category of software that Weyuker describes as "Programs which were written in order to determine the answer in the first place. There would be no need to write such programs, if the correct answer were known." The absence of a test oracle clearly presents a challenge when it comes to detecting subtle errors, faults, defects or anomalies in software in these domains. Without a test oracle, it is impossible to know in general what the expected output should be for a given input, but it may be possible to predict how changes to the input should effect changes in the output, and thus identify expected relations among a set of inputs and among the set of their respective outputs. This approach, introduced by Chen et al., is known as "metamorphic testing". In metamorphic testing, if test case input x produces an output f(x), the function's so-called "metamorphic properties" can then be used to guide the creation of a transformation function t, which can then be applied to the input to produce t(x); this transformation then allows us to predict the expected output f(t(x)), based on the (already known) value of f(x). If the new output is as expected, it is not necessarily right, but any violation of the property indicates a defect. That is, though it may not be possible to know whether an output is correct, we can at least tell whether an output is incorrect. This thesis investigates three hypotheses. First, I claim that an automated approach to metamorphic testing will advance the state of the art in detecting defects in programs without test oracles, particularly in the domains of machine learning, simulation, and optimization. To demonstrate this, I describe a tool for test automation, and present the results of new empirical studies comparing the effectiveness of metamorphic testing to that of other techniques for testing applications that do not have an oracle. Second, I suggest that conducting function-level metamorphic testing in the context of a running application will reveal defects not found by metamorphic testing using system-level properties alone, and introduce and evaluate a new testing technique called Metamorphic Runtime Checking. Third, I hypothesize that it is feasible to continue this type of testing in the deployment environment (i.e., after the software is released), with minimal impact on the user, and describe a generalized approach called In Vivo Testing. Additionally, this thesis presents guidelines for identifying metamorphic properties, explains how metamorphic testing fits into the software development process, and discusses suggestions for both practitioners and researchers who need to test software without the help of a test oracle.

Evaluation of output images carrying visual semantics is a challenging task which is carried out by domain experts through visual inspection. Automatic test oracle is required to augment the test oracle problem and to eliminate the manual efforts. Metamorphic testing is an effective technique to alleviate these problems. In this paper, the authors have demonstrated that how inherent properties of implementation under test can be used to generate an automatic test oracle for image processing applications. Metamorphic testing is a general technique in which follow-up test cases are generated using a transformation function and the anticipated output is evaluated. They have used some general metamorphic relations and also designed some algorithm specific metamorphic relations for morphological image operations. Selection of metamorphic relations is the most important step and the authors have analyzed relative effectiveness of different metamorphic relations using mutation analysis. The results show metamorphic testing is a very effective technique to automate output images evaluation and to alleviate oracle problem.

Evaluation of output images carrying visual semantics is a challenging task which is carried out by domain experts through visual inspection. Automatic test oracle is required to augment the test oracle problem and to eliminate the manual efforts. Metamorphic testing is an effective technique to alleviate these problems. In this paper, the authors have demonstrated that how inherent properties of implementation under test can be used to generate an automatic test oracle for image processing applications. Metamorphic testing is a general technique in which follow-up test cases are generated using a transformation function and the anticipated output is evaluated. They have used some general metamorphic relations and also designed some algorithm specific metamorphic relations for morphological image operations. Selection of metamorphic relations is the most important step and the authors have analyzed relative effectiveness of different metamorphic relations using mutation analysis. The results show metamorphic testing is a very effective technique to automate output images evaluation and to alleviate oracle problem.

Deep Neural Networks (DNNs) are increasingly applied to solve path planning problems in recent years. However, unexpected or incorrect behaviors of DNNs greatly threaten the reliability of DNN-based path planning algorithms. Therefore, the reliability should be evaluated through the software testing process. The quality of the training dataset is of great importance to the pre-trained DNN models. The pretrained model may still lack generality by using a randomly generated and insufficient training dataset. And DNN-based system testing is faced with Oracle problems. Because Metamorphic Testing (MT) has been shown considerable effectiveness in alleviating the absence of oracle problems. To increase the reliability of DNN-based path planning algorithms, in this paper, we present a test technique specialized for DNN-based path planning algorithms based on metamorphic testing. We present a framework for systematically designing sixteen metamorphic relations (MRs) by combining input transformations and output relations. And experiments are carried out on an actually released business software system, which demonstrates that our method is effective. The results show that our approach can effectively improve the diversity of test data, the accuracy of the DNN model, and the reliability of the software.

It is challenging to test machine learning (ML) applications, which are intended to learn properties of data sets where the correct answers are not already known. In the absence of a test oracle, one approach to testing these applications is to use metamorphic testing, in which properties of the application are exploited to define transformation functions on the input, such that the new output will be unchanged or can easily be predicted based on the original output; if the output is not as expected, then a defect must exist in the application. Here, we seek to enumerate and classify the metamorphic properties of some machine learning algorithms, and demonstrate how these can be applied to reveal defects in the applications of interest. In addition to the results of our testing, we present a set of properties that can be used to define these metamorphic relationships so that metamorphic testing can be used as a general approach to testing machine learning applications.

When improving their code, developers often turn to interactive debuggers. The correctness of these tools is crucial, because bugs in the debugger itself may mislead a developer, e.g., to believe that executed code is never reached or that a variable has another value than in the actual execution. Yet, debuggers are difficult to test because their input consists of both source code and a sequence of debugging actions, such as setting breakpoints or stepping through code. This paper presents the first metamorphic testing approach for debuggers. The key idea is to transform both the debugged code and the debugging actions in such a way that the behavior of the original and the transformed inputs should differ only in specific ways. For example, adding a breakpoint should not change the control flow of the debugged program. To support the interactive nature of debuggers, we introduce interactive metamorphic testing. It differs from traditional metamorphic testing by determining the input transformation and the expected behavioral change it causes while the program under test is running. Our evaluation applies the approach to the widely used debugger in the Chromium browser, where it finds eight previously unknown bugs with a true positive rate of 51%. All bugs have been confirmed by the developers, and one bug has even been marked as release-blocking.

The Critical Assessment of protein Function Annotation algorithms (CAFA) is a large-scale experiment for assessing the computational models for automated function prediction (AFP). The models presented in CAFA have shown excellent promise in terms of prediction accuracy, but quality assurance has been paid relatively less attention. The main challenge associated with conducting systematic testing on AFP software is the lack of a test oracle, which determines passing or failing of a test case; unfortunately, the exact expected outcomes are not well defined for the AFP task. Thus, AFP tools face the oracle problem. Metamorphic testing (MT) is a technique used to test programs that face the oracle problem using metamorphic relations (MRs). A MR determines whether a test has passed or failed by specifying how the output should change according to a specific change made to the input. In this work, we use MT to test nine CAFA2 AFP tools by defining a set of MRs that apply input transformations at the protein-level. According to our initial testing, we observe that several tools fail all the test cases and two tools pass all the test cases on different GO ontologies.

Simple transformation functions for finding better minima

The Urban Seismic Risk index (USRi) published in a previous article (Carreno et al., Nat Hazards 40:137–172, 2007) is a composite indicator that measures risk from an integrated perspective and guides decision-making for identifying the main interdisciplinary factors of vulnerability to be reduced or intervened. The first step of the method is the evaluation of the potential physical damage (hard approach) as a result of the convolution of the seismic hazard with the physical vulnerability of buildings and infrastructure. Subsequently, a set of social context conditions that aggravate the physical effects is also considered (soft approach). According to this procedure, the physical risk index is evaluated for each unit of analysis from existing loss scenarios, whereas the total risk index is obtained by multiplying the former index by an impact factor using an aggravating coefficient, based on variables associated with the socio-economic conditions of each unit of analysis. The USRi has been developed using the underlying holistic and multi-hazard approach of the Urban Risk Index framework proposed for the evaluation of disaster risk in different megacities worldwide. This article presents the sensitivity analysis of the index to different parameters such as input data, weights and transformation functions used for the scaling or normalization of variables. This analysis has been performed using the Monte Carlo simulation to validate the robustness of this composite indicator, understanding as robustness how the cities maintain the ranking as well as predefined risk level ranges, when compared with the deterministic results of risk. Results are shown for different cities of the world.

Proteins are the workhorses of life and gaining insight on their functions is of paramount importance for applications such as drug design. However, the experimental validation of functions of proteins is highly-resource consuming. Therefore, recently, automated protein function prediction (AFP) using machine learning has gained significant interest. Many of these AFP tools are based on supervised learning models trained using existing gold-standard functional annotations, which are known to be incomplete. The main challenge associated with conducting systematic testing on AFP software is the lack of a test oracle, which determines passing or failing of a test case; unfortunately, due to the incompleteness of gold-standard data, the exact expected outcomes are not well defined for the AFP task. Thus, AFP tools face the oracle problem. In this work, we use metamorphic testing (MT) to test nine state-of-the-art AFP tools by defining a set of metamorphic relations (MRs) that apply input transformations to protein sequences. According to our results, we observe that several AFP tools fail all the test cases causing concerns over the quality of their predictions.

As machine learning (ML) applications become prevalent in various aspects of everyday life, their dependability takes on increasing importance. It is challenging to test such applications, however, because they are intended to learn properties of data sets where the correct answers are not already known. Our work is not concerned with testing how well an ML algorithm learns, but rather seeks to ensure that an application using the algorithm implements the specification correctly and fulfills the users’ expectations. These are critical to ensuring the application’s dependability. This paper presents three approaches to testing these types of applications. In the first, we create a set of limited test cases for which it is, in fact, possible to predict what the correct output should be. In the second approach, we use random testing to generate large data sets according to parameterization based on the application’s equivalence classes. Our third approach is based on metamorphic testing, in which properties of the application are exploited to define transformation functions on the input, such that the new output can easily be predicted based on the original output. Here we discuss these approaches, and our findings from testing the dependability of three real-world ML applications.